NVIDIA NeMo Leads with Its Agentic Recovery Pipeline

The NVIDIA NeMo Retriever team has recently propelled its agentic retrieval pipeline to the top of the ViDoRe v3 leaderboard, achieving first place with an NDCG@10 score of 69.22. This same pipeline also secured second place on the BRIGHT leaderboard, with a score of 50.90, a benchmark recognized for its rigor in reasoning.

An agentic pipeline for smarter retrieval

In the ever-evolving field of artificial intelligence information retrieval, many solutions are often specialized, designed to excel in specific and narrow tasks. However, real-world enterprise applications require systems capable of adapting to a variety of challenges, ranging from analyzing complex visual layouts to executing deep logical reasoning. This is why NVIDIA has focused on generalization in the design of its pipeline, which dynamically adjusts its search and reasoning strategy based on the available data, thus delivering top-tier performance across varied benchmarks without requiring underlying architectural changes.

The motivation: why semantic similarity is not enough

For years, dense retrieval based on semantic similarity has been the norm for finding information. However, as retrieval applications evolve, the search for relevant documents goes beyond mere semantic similarity. A complex document search requires reasoning skills, an understanding of real-world systems, and iterative exploration. LLMs excel at thinking and reasoning but cannot process millions of documents at once. Conversely, retrievers can easily sift through millions of documents but have limited reasoning capabilities. Agentic retrieval bridges this gap by creating an active and iterative loop between the LLM and the retriever.

How it works: the agentic loop

NVIDIA's agentic retrieval pipeline relies on a ReACT architecture. Rather than a single "one-off" query, the agent iteratively searches, evaluates, and refines its approach. The agent uses built-in tools like think to plan its approach and final_results to output the exact documents needed for a given query, in addition to a retrieve (query, top_k) tool to explore the corpus. Through this loop, we have observed the natural emergence of successful search patterns:

• Generating better queries: The agent dynamically adjusts its search queries based on new information discovered.
• Persistent reformulation: It continuously reformulates queries until useful information is found.
• Decomposing complexity: It translates complex queries into several simpler queries with clear objectives.

Finally, to synthesize the iterative discoveries, the agent calls a final_results tool to output the most relevant documents, ranked by their relevance to a given query. As a safety net, for example, when the agent reaches the maximum number of steps or the context length limit, the pipeline reverts to Rank Reciprocal Fusion (RRF), which evaluates documents based on their ranks across all retrieval attempts in the agent's trajectory.

Engineering for speed and scale

Agentic workflows are notoriously slow and resource-intensive. To make this pipeline viable for leaderboard-scale evaluation, NVIDIA had to rethink the communication between the LLM agent and the retriever. Initially, the retriever was exposed to the agent via a Model Context Protocol (MCP) server—a natural choice, since the MCP is designed precisely to give LLMs access to external tools. However, in practice, this architecture imposed a cumulative cost on the speed of experiments. Each execution required starting a separate MCP server, loading the correct data corpus into GPU memory, and orchestrating the lifecycle of the client and server. Each of these steps was an opportunity for silent misconfiguration or server blocking under high-volume requests. Network round-trips added latency to each retrieval call, and the overall cognitive burden of managing a two-process configuration made it much more difficult for other teams to adopt and iterate on the pipeline.

To address this issue, NVIDIA replaced the MCP server with a thread-safe singleton retriever that resides within the process. The singleton loads models and corpus embeddings only once, protects all access with a reentrant lock, and exposes the same retrieve() interface to an arbitrary number of concurrent agent tasks—achieving the main advantage of an MCP server (shared and secure access to a GPU-resident retriever from multiple threads) without incurring network serialization overhead or requiring a separate server process. This unique architectural change eliminated an entire class of deployment errors and significantly improved both GPU utilization and experimental throughput.

Generalization vs. specialization across benchmarks

A common observation in modern retrieval evaluation is that highly optimized solutions for a specific type of task often experience a performance gap when applied to a completely different domain. NVIDIA secured second place on the BRIGHT leaderboard with an NDCG@10 of 50.90. The number one solution on this leaderboard, INF-X-Retriever, achieves an impressive 63.40. However, to test cross-domain adaptability, NVIDIA evaluated the INF-X pipeline (coupled with the same nemotron-colembed-vl-8b-v2 embedding model used in their agentic pipeline) on ViDoRe v3, a dataset focused on visually rich and diverse enterprise documents. On this different task, its performance was established at an NDCG@10 of 62.31, lower than the dense retrieval score of 64.36. In other words, INF-Query-Aligner does not outperform the dense retrieval baseline on ViDoRe v3.

In contrast, the same agentic pipeline from NVIDIA (pairing Opus 4.5 with nemotron-colembed-vl-8b-v2) achieved first place on ViDoRe v3 with a score of 69.22. This highlights a fundamental strength of their approach: generalization. Rather than relying on dataset-specific heuristics or a query rewriter/alignment, NVIDIA's agentic loop naturally adapts its strategy to the dataset in question, whether it requires multi-step logical reasoning or deciphering complex visual layouts.

Ablation studies: closed models vs. open models

NVIDIA conducted extensive ablation studies to understand the trade-off between state-of-the-art closed models and open-weight alternatives:

• Model choice: On ViDoRe v3, replacing Opus 4.5 with the open model gpt-oss-120b resulted in a slight drop in accuracy (from 69.22 to 66.38 NDCG@10) and required significantly fewer retrieval calls. On BRIGHT, the gap was wider, indicating that deeper reasoning tasks still heavily benefit from state-of-the-art models like Opus 4.5.

• Embeddings: Pairing the agent with specialized embeddings (nemotron-colembed-vl-8b-v2 for ViDoRe and llama-embed-nemotron-reasoning-3b for BRIGHT) yielded the best results, proving that a solid base retriever offers a higher ceiling for the agent to reach.

It is also noteworthy that the agent can reduce the gap between stronger and weaker embedding models. For example, on ViDoRe, the gap between the strongest nemotron-colembed-vl-8b-v2 and the weakest llama-nemotron-embed-vl-1b-v2 is about 8.5 in dense retrieval, but when paired with the agent gpt-oss-120b, the gap shrinks to about 4. Similarly, llama-embed-nemotron-reasoning-3b is about 19 points better than llama-nemotron-embed-vl-1b-v2 on BRIGHT, but the lead reduces to about 7.5 when coupled with the agent gpt-oss-120b.

The cost of autonomy and next steps

There is no free lunch. Agentic retrieval is more costly and slower than standard dense retrieval. Looking at our results on ViDoRe v3, the agent takes an average of 136 seconds per query and consumes about 760k input tokens and 6.3k output tokens per query. (Note: sequential latency is measured on a single A100 GPU with a single concurrent Claude API call—that is, without searching multiple queries at the same time—to reflect the actual search time in real-world use cases).

However, we believe that agentic retrieval is a highly viable approach for complex and high-stakes queries. Our immediate next steps focus on cost reduction: we are actively seeking ways to distill these agentic reasoning patterns into smaller, specialized open-weight agents. By fine-tuning smaller models to orchestrate the reflection and retrieval loop natively, we aim to deliver Opus-level accuracy at a fraction of the latency and cost.

Build your own agentic pipeline

While our leaderboard-topping executions explored combinations like Claude Opus and gpt-oss alongside our search embedding models, the true strength of this architecture lies in its modularity. For production-ready deployments, we strongly encourage you to try pairing your agent of choice with our robust commercial embedding model llama-nemotron-embed-vl-1b-v2. To explore these models, dive into the tools and start building your own highly generalizable retrieval workflows.