DeepSeek DSpark: 400% Acceleration of LLMs Unveiled

DeepSeek DSpark: 400% Acceleration of LLMs Unveiled

What is Speculative Decoding?

LLM generation is slow because each token requires a complete pass through the model. Speculative decoding speeds up this process using a smaller draft model that predicts multiple future tokens at once, which the target model then verifies in a single pass.

If the draft model makes good predictions, multiple tokens can be produced from a single pass through the target model. Conversely, if it makes poor predictions, it reverts to its normal pace. Output quality is maintained as the target model checks the predictions against its own probability distribution.

The main challenge is developing an appropriate draft model:

• When it is sequential and accurate over long predictions, it cannot keep up with the target model and fails to produce multiple tokens before the target model finishes.
• In this case, latency increases based on the number of blocks processed.

By making the draft model faster and parallel rather than sequential, predictions become less accurate in the latter part of the block. DSpark demonstrates a solution that addresses both factors simultaneously.

The Main Idea: Semi-Autoregressive Drafting

Here’s a prediction model: in an autoregressive context (i.e., Eagle3), each generated token is conditioned on all previously generated tokens. While this is representative of traditional machine learning training, it is inefficient, as the model experiences a linear increase in latency as the number of generated tokens increases.

In a parallel context (i.e., DFlash), the model generates an entire block of tokens in a single pass. This produces very fast output. However, each token is estimated in isolation from the other tokens in the block, which can create an incoherent mix of words.

DSpark combines a largely parallel structure for speed (many independent processing paths) with a small sequential structure that adds local dependencies between tokens. Together, it is a primarily parallel approach with a thin layer of autoregression to correct inconsistencies across the sequence.

The paper presents two sequencing structures:

• A Markov head uses only the previous token plus a low-rank matrix, achieving almost no overhead.
• An RNN head maintains minimal recurrent state across the block, giving it more context than the Markov head.

DeepSeek found that the Markov head essentially offers all the benefits at a much lower complexity, which is why they have put it into production.

Getting Started with DeepSpec

DeepSeek has open-sourced the training and evaluation code for their draft models under the name DeepSpec. This is a comprehensive repository for training any type of draft models, not just for DSpark, but also for DFlash and Eagle3. You can reproduce their comparisons of these models using this repository.

To install the dependencies and clone this repository, refer to the included README files in the repository.

git clone https://github.com/deepseek-ai/DeepSpec.git
python -m pip install -r requirements.txt

This covers the installation for training and evaluating models with DeepSpec. However, you will need to prepare your data separately using a mechanism to infer the outputs from the target model. For more information on this, refer to the scripts/data/README.md file in this repository.

Practice: Training and Evaluating a Draft Model

There are three steps in a DeepSpec workflow: data preparation, training your draft model, and evaluation. The output of one step becomes the input for the next.

Step 1: Choose a Configuration

You can find configurations in the config/ folder (there is a file for each pair of algorithms and target models).

ls config/dspark/
# dspark_qwen3_4b.py  dspark_qwen3_8b.py  dspark_gemma4_12b.py

Each configuration file specifies the target model, block size, and which sequential head should be used. If you want your configuration to match the smallest reference described in the article, you will want to use the dspark_qwen3_4b.py configuration file.

Step 2: Train the Model

To start training, you will use the following command:

bash scripts/train/train.sh --opts config_path=config/dspark/dspark_qwen3_4b.py

The script can create a worker for each GPU in your system. Checkpoint files will be saved in ~/checkpoints///step_*. If you are using a single node for training, you will need to set the CUDA_VISIBLE_DEVICES variable to match the number of GPUs you have.

During the training process, we optimize three types of losses simultaneously:

• a cross-entropy term (to correctly predict the next token),
• a distribution matching term (which directly relates to the "acceptance rate" of the generated content),
• a "confidence loss."

The latter is important as it allows us to implement the scheduling trick described in the next section.

Step 3: Evaluation

bash scripts/eval/eval.sh \
--target_name_or_path Qwen/Qwen3-4B \
--draft_name_or_path ~/checkpoints/deepspec/dspark_block8_qwen3_4b/step_latest

Verification is done in a single pass, so it is necessary to measure how many tokens are accepted across three types of tasks: math, code, and chat. More accepted tokens means fewer unnecessary passes to the target model.

Experimental Results

The figures presented by DeepSeek were indeed remarkable. DSpark exceeded the accepted length of Eagle3 by about 27-31%. The output of DSpark surpassed that of DFlash by 16-18%. These improvements remained consistent across all target models Qwen3-4B, 8B, and 14B. Additionally, they also achieved similar results on the Gemma4-12B model, indicating that there is also something with the results of Gemma and not just a peculiarity of Qwen.

The inter-family result helps clarify why DeepSeek's post had the titles of Gemma and Qwen listed. This should be considered a better indication than comparing to just a single model. Architecture-specific tricks generally have limitations when tested on another model division.

Key Takeaways and Pitfalls

Here are some very important insights, regardless of how they are presented:

• Discussion verifies differently from code: Discussion has more valid subsequent tokens (meaning it has a lower confidence rate) than code, so confidence decreases more rapidly and scheduling will be more aggressive.

• Static thresholds are not dynamic scheduling: A static threshold is last year's technology, and the threshold does not account for how busy your system is; DSpark will recalculate a dynamic threshold with each batch.

• Causality is non-negotiable: Because you cannot see into the future, the scheduler cannot verify a token before ensuring that the token has been validated. This is often managed offline using the two-step confidence prediction process that was ongoing at the end of V2.

• The extremes of nominal percentages are very misleading: For example, the 661% multiplier for MTP-1@V4-Flash is under artificial conditions; the metric does not reflect the actual output of a manufacturer, so do not use this multiplication as an expected value, but rather use the 60-85% corresponding throughput.

• You cannot recover drafting costs: Even if your request is not accepted, you still pay full drafting fees at the time of the request, even if the system eliminates verification after scheduling.

DSpark is a solid reminder that inference accelerations can come from many places. Not all gains require a larger model or better hardware; sometimes, it comes from admitting that drafts can be inaccurate and allowing the scheduler to work intelligently around that admission.

If you are running speculative decoding under variable request loads, the idea applies even if your architecture is not of the DeepSeek type. The principle is simple: only verify what has a positive expected value.

And if you are wondering how the Markov head compares to full attention for the draft block, that is the next topic to explore. You can test it yourself, since the...