Nemotron-Labs: Revolutionizing Diffusion Models

⚡

Key Takeaways

1Nemotron-Labs introduces diffusion models that generate multiple tokens simultaneously, optimizing the use of modern GPUs.

2The Diffusion 8B model from Nemotron-Labs offers inference speeds up to 6.4 times faster than autoregressive models.

3Developers can easily integrate these models thanks to compatibility with autoregressive and diffusion modes.

💡Why it matters — This innovation enables faster and more flexible text generation, which is essential for latency-sensitive applications.

The Limits of Autoregressive Models

Large language models (LLMs) have become essential tools for code generation, mathematical problem-solving, and document synthesis. However, most of these models operate autoregressively, generating text one token at a time, with each token depending on the previous ones. While this method has proven effective, it has notable limitations. Autoregressive models consume their own outputs, which can lead to errors that propagate during generation. Additionally, token-by-token generation can leave performance on the table, as most of the GPU time is spent on memory operations rather than computation.

Once a token is generated by an autoregressive model, it is final, and these models do not inherently have the ability to revise previous tokens. This can pose challenges for developers building latency-sensitive applications or running smaller batch sizes. For these reasons, the autoregressive approach creates a strict limit: each new token requires a full pass through the model, and each weight must be loaded from memory before computation can begin.

The Innovation of Nemotron-Labs

Nemotron-Labs offers an innovative approach with its diffusion language models (DLMs). These models generate multiple tokens in parallel and iteratively refine them, allowing for better utilization of modern GPUs. This method also provides the ability to revise generated tokens, thereby reducing potential errors. The generation and refinement property of diffusion models offers an integrated way to control the inference budget, reducing the number of refinement steps to decrease real-time computational requirements.

Flexible Generation Modes

The Nemotron-Labs Diffusion model incorporates three generation modes:

Autoregressive Mode: Functions like a standard model, ensuring compatibility with existing processes.
Diffusion Mode: Generates tokens in blocks, thus improving generation speed. The diffusion model (FastDiffuser) fills a block of 32 tokens at a time by iteratively denoising it, and a confidence threshold decides which tokens are 'good enough' at each step.
Auto-Speculation Mode: Combines the speed of diffusion with the reliability of autoregressive verification. The auto-speculation model pushes this even further to 6 times for linear auto-speculation and 6.4 times for quadratic auto-speculation, with comparable accuracy on evaluated tasks.

This flexibility allows developers to choose the mode best suited to their needs without modifying their applications. Selecting the desired inference mode requires almost no changes at the application level, as it is a deployment parameter.

Performance and Training

The Diffusion 8B model from Nemotron-Labs shows an improved accuracy of 1.2% compared to Qwen3 8B. In terms of speed, the diffusion mode achieves 2.6 times more tokens per forward pass (TPF) than autoregressive models, and auto-speculation is up to 6.4 times faster. The model has been pre-trained on 1.3 trillion tokens and fine-tuned with an additional 45 billion tokens.

Efficient-DLM has demonstrated that pre-trained AR models can be converted into diffusion language models through continuous pre-training and by modifying the attention mechanism to a block-based approach. This design helps preserve the capabilities of AR models while allowing for parallel decoding compatible with KV caching.

Deployment via SGLang

The Nemotron-Labs Diffusion models will soon be available in the main branch of SGLang. Support for inference is available via an issue follow-up request on GitHub. Developers can choose between autoregressive, diffusion, or auto-speculation modes by simply modifying the algorithm configuration.

The integration allows serving the same checkpoint in three different ways, chosen by a single line in your algorithm configuration:

Simple Autoregressive: Set ar_mode=true and the model behaves like any other causal LM. Useful as a correction reference, or if you simply want validity control against pure AR output.
Diffusion Mode (FastDiffuser): The model fills a block of 32 tokens at a time by iteratively denoising it, and a confidence threshold decides which tokens are "good enough" at each step.
Auto-Speculation (LinearSpec): The same model writes a block bidirectionally, then verifies it causally; any matching prefix is validated. The output is lossless compared to AR at temperature 0, but we achieved around 865 tokens/s on B200 on the speedbench dataset—about 4 times the autoregressive reference on the same hardware.

Conclusion

With Nemotron-Labs Diffusion, developers now have a powerful and flexible solution for text generation. This advancement promises to significantly enhance applications requiring fast and accurate generation. To get started, explore the Nemotron-Labs Diffusion model family, read the technical report, and try the available training recipe.