DSpark in llama.cpp: +31.5% Generation on Qwen3-8B

Key Takeaways

1DSpark in llama.cpp increases the generation of Qwen3‑8B from 95.0 to 124.9 tokens/s at constant parameters

2DSpark combines parallel draft generation and sequential components, with confidence filtering

3Support for DSpark models remains limited and integration into llama.cpp is recent

4DeepSeek announces a 60–85% gain on DeepSeek‑V4 compared to MTP‑1, without generalization to small local models

Why it matters : DSpark provides measurable acceleration of local inference without the addition of a GPU, but its use is still conditioned by model compatibility and the maturity of the integration.

A local test under llama.cpp shows an increase in generation throughput from 95.0 to 124.9 tokens/s with DSpark on Qwen3-8B, at identical parameters. The DeepSeek technique remains simple to activate, but its model support is still limited and its integration into llama.cpp is recent.

DSpark support remains limited and still young in llama.cpp

The support for models compatible with a DSpark draft is restricted, which limits experimentation to a small number of configurations. The integration into llama.cpp is recent and may expose users to bugs or instabilities depending on the model and version used. During measurements, the prompt processing speed is lower with DSpark than in standard execution. For long outputs, the main interest lies in the token generation throughput, which can have a more significant impact on overall inference time.

DSpark combines parallel and sequential drafts with confidence filtering

DSpark combines parallel draft production with a lightweight sequential component. This organization allows later draft positions to take into account previous predictions while maintaining the speed of parallel generation. Strictly parallel approaches often see accuracy decrease on subsequent tokens of the block; DSpark aims to address this. The system can estimate the probability of retaining tokens during verification and remove segments deemed low-confidence. In llama.cpp, an optional confidence threshold allows this behavior to be exposed.

Constant testing protocol to isolate the effect of DSpark

The test bench relies on Qwen3-8B under llama.cpp, first running a reference without speculative decoding, then the same configuration with DSpark. The prompt remains identical, requesting a complete Python implementation of merge sort with explanations and complexities. Decoding is deterministic with a temperature of 0 and a fixed top-k of 1. GPU offloading options are enabled, with -ngl all for the main model and Flash Attention activated.

Le brief IA que lisent les pros

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

Results: from 95.0 to 124.9 tokens/s, approximately 1.31×

Without DSpark, the benchmark summary of llama.cpp shows a generation rate of 95.0 tokens/s, a value retained as a reference. With DSpark, the draft model is loaded via -md, the mode --spec-type draft-dspark is activated, and --spec-draft-n-max is set to 3, with draft offloading to the GPU. The benchmark summary then indicates a generation rate of 124.9 tokens/s. The comparison shows an increase from 95.0 to 124.9 tokens/s, yielding a gain of approximately 1.31×, or about 31.5% faster, while the prompt phase is measured to be slower with DSpark.

Setup: CUDA compilation and Qwen3-8B models in GGUF

The setup involves building the latest version of llama.cpp with CUDA acceleration, after installing git, cmake, and build-essential. The official repository ggml-org/llama.cpp is cloned, and the binary targets including llama-cli are built. GGUF files are stored in /workspace/models and downloaded via the Hugging Face CLI, with authentication possible through the HF_TOKEN variable. The main model Qwen3-8B Q4_K_M is retrieved from Qwen/Qwen3-8B-GGUF, and the corresponding DSpark draft dspark-Qwen3-8B-Q8_0.gguf from ggml-org/Qwen3-8B-GGUF. The observed sizes are around 4.7G for the target model and 1.2G for the draft.

Context: More widespread MTP, DSpark promises better drafts

Several optimization levers exist beyond adding GPUs, including quantization, optimized kernels, and improved inference engines. On the speculative decoding side, traditional approaches use a smaller draft model, MTP anticipates multiple tokens, and variants like Medusa, EAGLE, or DFlash modulate how candidate blocks are generated. MTP is often presented as the most practical option for local inference as it is simpler and available on more models. However, DSpark can gain an advantage when the quality of the draft increases the share of accepted speculative tokens. DeepSeek reports gains of 60 to 85% compared to its MTP-1 baseline with DeepSeek-V4, figures that are not set as an expectation for a small local model.