llama-server : implement universal assisted decoding #12635
+168
−62
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This looks really interesting! It's surprising how much crossover there is between many models' tokenisers. |
There was a problem hiding this comment.
Pull Request Overview
This PR implements universal assisted decoding in llama-server, enabling speculative decoding with draft models that have incompatible tokenizers with the main model. The implementation translates tokens between the two models by detokenizing and retokenizing text.
Key changes:
- Removes strict vocabulary compatibility checks between draft and target models
- Adds token translation mechanism through text conversion when vocabularies are incompatible
- Introduces string replacement functionality for handling tokenizer differences
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/server/server.cpp | Updates server to handle incompatible vocabularies and configure token replacements |
| examples/speculative/speculative.cpp | Removes vocabulary compatibility validation checks |
| examples/speculative-simple/speculative-simple.cpp | Removes compatibility checks and updates speculative initialization |
| common/speculative.h | Updates function signatures to accept both target and draft contexts |
| common/speculative.cpp | Implements core universal assisted decoding logic with token translation |
| common/common.h | Adds replacements field to speculative parameters |
| common/arg.cpp | Adds command-line argument parsing for token replacements |
Comments suppressed due to low confidence (1)
common/speculative.cpp:30
- The comment 'ctx_main' doesn't match the struct member name 'ctx_tgt'. This should be '/* .ctx_tgt = */' to be consistent with the actual member name.
/* .ctx_main = */ ctx_tgt,
CISC
approved these changes
Jul 29, 2025
ggerganov
reviewed
Jul 30, 2025
| struct common_speculative * spec, | ||
| const std::string& input) { | ||
| std::string result = input; | ||
| for (const auto& pair : spec->tgt_dft_replacements) { |
There was a problem hiding this comment.
Suggested change
| for (const auto& pair : spec->tgt_dft_replacements) { | |
| for (const auto & pair : spec->tgt_dft_replacements) { |
Comment on lines
+212
to
+213
| const auto *model_tgt = llama_get_model(ctx_tgt); | ||
| const auto *vocab_tgt = llama_model_get_vocab(model_tgt); |
There was a problem hiding this comment.
Fix the whitespace in a few more places:
Suggested change
| const auto *model_tgt = llama_get_model(ctx_tgt); | |
| const auto *vocab_tgt = llama_model_get_vocab(model_tgt); | |
| const auto * model_tgt = llama_get_model(ctx_tgt); | |
| const auto * vocab_tgt = llama_model_get_vocab(model_tgt); |
| std::string text; | ||
| text = common_detokenize(ctx_tgt, prompt_tgt_main_model, true); | ||
| text = replace_to_dft(spec, text); | ||
| LOG_DBG("main->draft detokenized string: '%s'\n", text.c_str()); |
There was a problem hiding this comment.
Follow the log pattern:
Suggested change
| LOG_DBG("main->draft detokenized string: '%s'\n", text.c_str()); | |
| LOG_DBG("%s: main->draft detokenized string: '%s'\n", __func__, text.c_str()); |
Comment on lines
+3256
to
+3262
| add_opt(common_arg( | ||
| {"--spec-replace"}, "TARGET", "DRAFT", | ||
| "translate the string in TARGET into DRAFT if the draft model and main model are not compatible", | ||
| [](common_params & params, const std::string & tgt, const std::string & dft) { | ||
| params.speculative.replacements.push_back({ tgt, dft }); | ||
| } | ||
| ).set_examples({LLAMA_EXAMPLE_SPECULATIVE, LLAMA_EXAMPLE_SERVER})); |
There was a problem hiding this comment.
In which cases do we need to use replacements?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request implements universal assisted decoding in llama-server. This is a method for performing speculative decoding given a draft model whose tokenizer is incompatible with the main model, by decoding and reencoding the generated text between the two models.
It currently works, but some improvements can be made.