NVIDIA Nemotron 3 Nano Omni: A Multimodal Revolution for AI

Introduction to NVIDIA Nemotron 3 Nano Omni

NVIDIA recently unveiled the Nemotron 3 Nano Omni, an omni-modal understanding model that promises to transform document analysis, audio-video interpretation, and general reasoning. This model expands the Nemotron multimodal range, integrating advanced capabilities in text, image, video, and audio. It is designed to analyze real documents, reason across multiple images, automatically recognize speech, understand long-term audio-video, utilize agent computers, and perform general reasoning.

The Nemotron 3 Nano Omni stands out for its exceptional accuracy on complex benchmarks such as MMlongbench-Doc and OCRBenchV2. It also excels in video and audio benchmarks like WorldSense and DailyOmni, achieving peak performance on VoiceBench for audio comprehension. Its cost-effectiveness in video understanding is demonstrated on MediaPerf.

An Innovative and High-Performance Architecture

The model is based on the hybrid architecture Mamba-Transformer Mixture-of-Experts, combining a vision encoder C-RADIOv4-H and an audio encoder Parakeet-TDT-0.6B-v2. This architecture is designed to preserve visual details, add native audio understanding, and adapt to very long multimodal contexts. It allows for the processing of dense images, documents, videos, and mixed-modal reasoning.

The training recipe employs stepwise multimodal alignment and context extension, followed by preference optimization and multimodal reinforcement learning. Nemotron 3 Nano Omni offers up to 9x higher throughput and 2.9x single-stream reasoning speed compared to alternatives, thanks to this sophisticated method.

The BF16, FP8, and NVFP4 checkpoints can be downloaded from HuggingFace, and for more information on the model architecture, training recipe, data pipelines, and benchmarks, the full report on Nemotron 3 Nano Omni is available.

Benchmark Highlights

Building on Nemotron Nano V2 VL, Nemotron 3 Nano Omni provides substantial visual gains and adds new audio and video + audio capabilities, while outperforming another open-weight omni model, Qwen3-Omni, in many areas. The relevant domains include document understanding, CharXiv reasoning, video comprehension, video + audio understanding, voice interaction, and HF Open ASR where less is more.

Efficiency Highlights

Compared to other open omni models with similar interactivity, Nemotron 3 Nano Omni offers 7.4x more system efficiency for multi-document use cases and 9.2x more system efficiency for video use cases.

Objectives of Nemotron 3 Nano Omni

At a high level, Nemotron 3 Nano Omni is intended for five classes of workloads:

1. Real Document Analysis: This is not just about OCR. The model is designed for long, messy, high-value documents where understanding depends on layout, tables, figures, formulas, section structure, and cross-page references. Think of contracts, technical papers, reports, manuals, multi-page forms, or compliance packages. The model can handle documents over 100 pages.

2. Automatic Speech Recognition: Nemotron 3 Nano Omni includes strong speech comprehension capabilities that allow for high-quality transcription in various audio conditions. It manages long audio with diverse speakers, accents, and background noise. These capabilities can be integrated into broader workflows, enabling transcription, analysis, and combining spoken content with other modalities for tasks like synthesis, question answering, and intermodal reasoning.

3. Long Audio-Video Comprehension: Many enterprise and developer workflows rely on mixed audio and visual elements: screen recordings with narration, training videos, meetings with slides, tutorials, product demonstrations, customer support captures, and long video archives. Nemotron 3 Nano Omni is designed to reason about these inputs jointly.

4. Use of Agent Computers: The Nemotron 3 Nano Omni model is specifically trained for the use of agent computers, allowing it to assist in tasks within graphical user interface (GUI) environments. Its capabilities include interpreting screenshots, monitoring the state of the user interface, anchoring its reasoning in on-screen visuals, and aiding in action selection or workflow automation.

5. General Multimodal Reasoning: The model is designed for more than perception. It excels in reasoning-intensive tasks that require synthesizing information across long context windows, multiple modalities, and structured or semi-structured evidence. It can perform multi-step reasoning, carry out calculations, and connect signals from text, images, tables, and other inputs to arrive at coherent and well-supported answers.

Model Architecture and Key Innovations

Nemotron 3 Nano Omni employs a unified encoder-projector-decoder design. The language architecture is Nemotron 3 Nano 30B-A3B, paired with the vision encoder C-RADIOv4-H and the audio encoder Parakeet-TDT-0.6B-v2. The modality-specific encoders connect to the LLM architecture via lightweight projectors.

A Hybrid Mamba-Transformer-MoE Architecture for Long Multimodal Contexts

The model's architecture intertwines three key components:

• 23 Mamba layers of selective state processing for efficient long context handling
• 23 MoE layers with 128 experts, top-6 routing, and a shared expert for conditional capacity
• 6 layers of grouped query attention to preserve strong global interaction and expressiveness

Nemotron 3 Nano Omni combines spatial state models, attention, and MoE in a unified design that maintains strong reasoning performance while remaining practical for long multimodal contexts.

Dynamic Resolution for Dense Documents, Graphs, and Screens

On the vision side, Nemotron 3 Nano Omni replaces the tiling strategy used in the v2 model with dynamic resolution processing at the native aspect ratio. Each image can be represented by a variable number of 16 x 16 patches, with a minimum of 1,024 to a maximum of 13,312 visual patches per image. For square images, this equates to 512 x 512 and 1840 x 1840, respectively.

This flexibility is crucial for handling complex, high-resolution visual inputs such as OCR-rich documents, financial tables, slides, research figures, screenshots, and GUI layouts—especially when fine details and overall structure must be understood together.

Temporal Compression Conv3D for Video

For video, Nemotron 3 Nano Omni utilizes a dedicated Conv3D tubelet embedding path. Instead of embedding each image independently, each pair of consecutive images is merged into a single "tubelet" before the ViT, halving the number of visual tokens that the language model must process. This allows either to double the number of images with the same token budget or to halve the number of tokens with the same number of images.

EVS — Efficient Video Sampling

EVS is an important feature, activated during inference time, which eliminates redundant video tokens after the vision encoder. This reduces latency and improves throughput while maintaining accuracy. The first frame of the video is kept in its entirety, and then for each subsequent frame, EVS retains the "dynamic" tokens where the video changes and eliminates the "static" tokens where nothing has changed compared to the previous frame. We combine this with Conv3D to allow for superior compression: Conv3D merges tokens from pairs of images into one, and then EVS removes redundant static information.

Native Audio Input, Not Just Text Transcriptions

The audio side is powered by Parakeet-TDT-0.6B-v2, connected to the architecture via its own 2-layer MLP projector. Audio is sampled at 16 kHz, and the model is trained with inputs of up to 1,200 seconds (20 minutes), while the maximum context length of the LLM supports over 5 hours.

This represents a shift from traditional VLM pipelines by allowing native audio processing within a shared multimodal sequence, enabling audio, visual, and textual tokens to be modeled together. This is crucial for scenarios like narrated screen recordings, video question-answering where speech alters visual meaning, long training or meeting content, and tasks requiring multimodal reasoning anchored in time.

Lightweight Modality Projectors and Unified Token Interleaving

Each encoder is connected to the LLM with a lightweight 2-layer MLP projector that maps the encoder features into the shared embedding space. Once projected, the visual, audio, and textual tokens are interleaved and processed together.

This design keeps the overall system modular while enabling true intermodal reasoning within the architecture itself.

Training Data, Infrastructure, and System History

The SFT stages are trained on NVIDIA H100, scaling from 32 to 128 nodes depending on the stage. The stack uses Megatron-LM, Transformer Engine, and Megatron Energon, with tensor parallelism, expert parallelism, sequence parallelism, context parallelism for long context stages, online sequence packing, and selective activation recomputation.

Post-SFT reinforcement learning uses NeMo-RL and NeMo Gym with a Megatron backend. The RL infrastructure employs a distributed configuration based on Ray across B200 and H100 clusters, plus multimodal deduplication, so that repeated rollouts do not multiply the memory for image, video, and audio.

NVIDIA open-sources substantial parts of its training code, facilitating access to this revolutionary technology.