AI Agents Fail in Research, Too Dependent on Memory

AI Agents and Memory Dependence

A recent study revealed that AI research agents, designed to explore the web for answers, often settle for confirming what they already know. Advanced models such as GPT-5.4, Gemini 3.1 Pro, Claude Sonnet 4.6, DeepSeek-V4-Pro, and Kimi-K2.6 score highly on the BrowseComp benchmark, which poses complex questions requiring multi-step navigation and the gathering of information from various web sources.

However, researchers from the Harbin Institute of Technology and Xiaohongshu demonstrated that these results reflect less the actual search capabilities of the agents than previously assumed. They introduced the concept of intrinsic knowledge dependence (IKD), which refers to the reliance on internal knowledge that models have absorbed during their training.

With static benchmarks, the necessary knowledge integrates into the parameter memory over generations of models, making tasks easier to solve over time. To counter this phenomenon, LiveBrowseComp was developed, posing time-sensitive questions to assess the models' ability to search for current information.

The Limits of Search Capabilities

The researchers tested a total of eleven models, first by removing all search and navigation tools. Surprisingly, even without internet access, the models achieved high scores. For instance, MiniMax M2.5 solved 44.5% of the BrowseComp tasks solely from its memory, while Kimi K2.6 reached 62% on the Chinese variant BrowseComp-ZH. This indicates that the benchmark performance largely stems from the models' memory, even before any search is conducted.

The second test revealed even more significant results. The researchers kept the search interface in place but removed all documents supporting the answers from the search index. Each tested model then performed worse than without access to any tools. MiniMax M2.5 dropped from 44.5% to 8.0%, and Kimi-K2.6 fell from 25.5% to 2.3%. The search seems to divert agents from their correct intuitive answers as soon as no confirming results appear.

As the search progresses, agents increasingly seek to confirm their own hypotheses rather than discover new facts. When they find supporting sources, they use them less than a third of the time. An analysis of search paths explains why: more than half of all queries stem from the model's own reasoning rather than from previously found results. Even when relevant evidence appears in search results, agents integrate it into their reasoning less than a third of the time. The loop is driven by the model, not by the evidence.

A Benchmark Beyond the Knowledge Frontier

To measure true search behavior, the authors constructed LiveBrowseComp. The benchmark contains 335 human-written questions, each dependent on at least one fact from the 90 days preceding its creation and impossible to answer without this current information.

The underlying events come from constantly updated sources such as movie databases, game directories, security vulnerability registries, and earthquake catalogs. Globally known events are deliberately filtered out, leaving obscure but publicly verifiable facts that had little chance of infiltrating the model's parameters during training.

The pipeline filters only facts from the last 90 days, discards unstable answers, and each question is verified by experts for relevance, difficulty, and clarity.

Human testers require about the same time for LiveBrowseComp as for BrowseComp and solve a similar number of tasks. The drop in performance among the models is thus due to the loss of the memory shortcut, not because the questions are more difficult.

Scoreboard Rankings Collapse

On LiveBrowseComp, all models in the closed-book test fall below two percent accuracy. With tools enabled, scores are about 25 to 40 points lower than those same models on BrowseComp.

Without tools, models solve up to 44.5% of BrowseComp questions from memory. On LiveBrowseComp, this figure drops below two percent overall, confirming the temporal blockage against parameter knowledge.

This alters the rankings. GLM 5.1 is clearly at the top among open-source models on BrowseComp but falls to the middle of the ranking on LiveBrowseComp. DeepSeek v3.2, which was at the bottom on BrowseComp, then climbed to the top on LiveBrowseComp, surpassing several models that had previously outperformed it. This shows that a model's position on a static leaderboard primarily reflects how much it already knows, not how effectively it searches.

Agents Need More Steps When They Can't Rely on Memory

On BrowseComp, agents solve many questions in very few steps, a sign of quick memory confirmation. On LiveBrowseComp, this pattern disappears. The number of steps increases significantly, suggesting that agents are conducting real searches instead of recalling stored knowledge.

The authors argue that dynamic and time-sensitive benchmarks should become the norm for evaluating AI agents. They also seek training signals that reward evidence-based searching rather than the typical guess-and-check approach.

Other studies have reported similar issues. A benchmark from Peking University found that top models often produce the correct answer when analyzing documents but cite the wrong source, a phenomenon researchers call attribution hallucination. A tool called CiteAudit recently discovered that fabricated references had already been accepted in papers at major AI conferences. The reason: commercial models do not reliably detect invented citations.