Nemotron 3 Nano 4B: Compact and High-Performance Local AI

The Nemotron 3 Nano 4B represents a significant advancement in the field of lightweight language models, combining efficiency and accuracy through its hybrid Mamba-Transformer architecture. This model is designed to operate on all platforms compatible with NVIDIA GPUs, offering cutting-edge instruction tracking and optimized tool usage while minimizing VRAM footprint.

With only 4 billion parameters, the Nemotron 3 Nano 4B is compact enough to run at the edge on platforms such as NVIDIA Jetson (Jetson Thor/Jetson Orin Nano) as well as on NVIDIA DGX Spark and NVIDIA RTX GPUs. This compactness allows for faster response times, better data privacy, and flexible deployment, all while keeping inference costs low.

The Nemotron 3 Nano 4B is our first model specifically optimized for on-device deployment, designed to power local conversational agents and personas across the use cases of GeForce RTX, Jetson, and Spark customers. It achieves cutting-edge accuracy and efficiency across several key dimensions for production use at the edge.

Performance and Benchmarks

The model excels in several benchmarks, including instruction tracking (IFBench, IFEval) and agility in game intelligence (Orak), while maintaining a low VRAM footprint. It also offers reduced latency, making it ideal for edge use. Efficiency benchmarks were measured on an RTX 4070 using Llama.cpp with quantized Q4_K_M versions of both models.

In addition to its tool usage performance, the Nemotron 3 Nano 4B is highly competitive in terms of hallucination avoidance. These capabilities demonstrate the model's strong suitability for edge use cases.

Compression and Distillation

The Nemotron 3 Nano 4B has been pruned and distilled from the Nemotron Nano 9B v2 using the Nemotron Elastic framework, allowing it to inherit strong reasoning capabilities as a hybrid reasoning model. Rather than training a 4B model from scratch, Nemotron Elastic employs structured pruning guided by a router, which is jointly trained with the model using an auxiliary loss addressing the student model size plus the original knowledge distillation loss.

Nemotron Elastic introduces an end-to-end trained router that performs neural architecture search across multiple compression axes, as well as knowledge distillation passage. For the Nano 4B, the framework was used in a single-budget configuration—targeting only the 4B parameter count—where the router's role is to determine which axes to prune and by how much to reach the target budget.

The router converged on the following pruning decisions: for the Nemotron Nano 9B v2, there were 56 layers (27 Mamba, 4 attention, 25 MLP), while for the Nemotron 3 Nano 4B, there are 42 layers (21 Mamba, 4 Attention, 17 MLP).

Two-Step Distillation

After the router determined the pruned architecture, the compressed model is retrained using knowledge distillation from the frozen parent 9B using pre-training and post-training data from Nano v2. This precision recovery process occurs in two steps:

• Step 1 — Short Context Distillation (8K sequence length): The 4B model is trained on 63B tokens using an 8K context window with a data mix composed of approximately 70% post-training data and 30% pre-training data from the parent Nano v2. This step is crucial for the initial recovery of the model's accuracy after compression.

• Step 2 — Long Context Extension (49K sequence length): To restore performance on more challenging tasks requiring extended reasoning chains, the context is extended to 49K tokens. In this step, the model is trained on 150B tokens.

Supervised Fine-Tuning and Reinforcement Learning

We conducted two supervised fine-tuning (SFT) steps with relevant subsets of the Nemotron-Post-Training-v3 collection using Megatron-LM. The first SFT step trains the model on a mix of reasoning and non-reasoning data covering various domains such as mathematics, programming, science, chat, instruction tracking, and agentic tasks. The second step is a smaller-scale training, targeted to reinforce safety behaviors.

Once the model is primed with SFT, we move to a three-step reinforcement learning pipeline using NeMo-RL to target our areas of interest, instruction tracking and tool calling/agentic behavior. In the first step, we use single-turn instruction tracking data. In the second step, we use NeMo-Gym environments for single-turn and multi-turn instruction tracking as well as for structured outputs (JSON, XML). Finally, in the third step, we use a preliminary version of Nemotron-RL-Agentic-Conversational-Tool-Use-Pivot-v1 for multi-turn conversational tool calling. A balanced 50-50 ratio of reasoning and non-reasoning data was used throughout the three RLVR steps, with a progressively increased KL penalty at each step.

Efficiency Improvement with Quantization

For edge devices, it is essential to further reduce model size through quantization to enhance efficiency and reduce VRAM usage. The Nemotron 3 Nano 4B is released in FP8 and Q4_K_M GGUF to be efficient on-device at the edge.

To preserve accuracy while improving efficiency, a selective quantization strategy was applied rather than quantizing the entire network. The comparison of a set of quantization configurations showed that maintaining the self-attention layers (4 out of 42 layers) and the 4 Mamba layers preceding the self-attention layers in BF16 provided an optimal balance for accuracy recovery and efficiency improvement trade-offs. The model weights, activations, and KV-Cache are quantized in FP8. The Conv1D in all Mamba layers are kept in BF16. The FP8 model achieved 100% median accuracy recovery compared to target benchmarks compared to the BF16 model. The quantized FP8 version offers up to 1.8X latency and throughput improvement over the original BF16 version on DGX Spark and Jetson Thor.

For Llama.cpp support, we use the widely adopted GGUF quantization method Q4_K_M, a 4-bit scheme that offers an excellent balance between efficiency and accuracy. The GGUF Q4_K_M version achieved 100% median accuracy recovery compared to target benchmarks compared to the BF16 model.

This GGUF version is also well-suited for Jetson deployments. On the Jetson Orin Nano 8GB designed for small embedded devices, the Q4_K_M checkpoint running with Llama.cpp delivers 18 tokens/s, up to 2× more throughput than the Nemotron Nano 9B v2, highlighting the efficiency of the Nemotron 3 Nano 4B for edge inference in embedded AI and robotics use cases.

The Nemotron 3 Nano 4B is available on a variety of inference engines, including Transformers, vLLM, TRT-LLM, and Llama.cpp, enabling support for a wide range of edge deployment scenarios. To get started, visit the Hugging Face repositories below to download the model checkpoints. Usage examples for Hugging Face Transformers, vLLM, TRT-LLM, and Llama.cpp are available in the model card.