AI Coding Agents: Experts in Files, Not in Code Lines

An Innovative Repository Reveals the Limits of AI Coding Agents

In the ever-evolving world of software development, AI-based coding agents are often judged by their ability to fix bugs. However, a new study sheds light on an unexpected weakness of these agents: while they can identify the correct file where the problem lies, they often fail to pinpoint the specific lines of code that require correction. This finding comes from an international research team, including Jiao Tong University in Shanghai, which developed a repository named SWE-Explore. This repository focuses on the first step of the correction process, namely locating the relevant sections of code from a bug description.

Traditionally, repositories evaluate AI coding agents based on their ability to repair a bug, without considering whether the agent has actually consulted the relevant code. SWE-Explore stands out by isolating this initial research phase, allowing for an assessment of the agents' ability to identify crucial code sections even before attempting a fix.

Successes as a Benchmark for Evaluation

To establish a solid benchmark, the researchers analyzed a set of 848 problems for which at least two successful solutions were found by advanced models such as GPT-5.4, Gemini 3 Pro, Claude Sonnet 4.6, and Kimi K2.6. By examining the files and lines of code that these models actually consulted, the team was able to identify passages that converge towards a solution, thus serving as a useful contextual signal.

Rather than manually determining the necessary code sections, the team derived its benchmark from the reading traces of the successes. A second test was then conducted to verify whether better search scores translated into more effective repairs.

Keyword Search: An Outdated Method

The study compared traditional search methods to five general-purpose coding agents, such as Claude Code, Codex, OpenHands, Mini-SWE-Agent, and AweAgent, as well as four search systems specifically designed for code. The results show that keyword search, a classic method, is hardly more effective than random chance. For instance, terms like "RuntimeWarning on Overflow" frequently appear in project documents but rarely in the source code, misleading the agents.

AI agents, on the other hand, excel in their ability to explore the project sequentially, rather than merely sorting results all at once, which gives them a distinct advantage.

Failing Precision at the Line Level

Although coding agents are proficient at identifying the correct source file, their effectiveness drops dramatically when it comes to targeting individual lines of code. General-purpose agents manage to cover only 14 to 19% of the actually relevant lines.

To evaluate this performance, the results ranked by an agent are compared to the key regions identified from the successes. While agents succeed in locating the correct file, their coverage of essential lines remains uneven.

Even the use of more powerful language models does not resolve this issue. The team tested the same agent with several models from different companies, including OpenAI, Anthropic, Google, Moonshot, and Zhipu. Although the GPT family stands out slightly, the pattern remains the same: file success rates are consistently higher than line coverage.

CoSIL: A Notable Exception

Among the various agent architectures, the search system CoSIL stands out. By analyzing code as a network of interconnected blocks, CoSIL achieves significantly better line coverage. Among specialized systems, AutoCodeRover is precise but cautious, while OrcaLoca minimizes noise but misses relevant points.

The Importance of Sufficient Context for Repairs

In a controlled ablation experiment, the team artificially varied the visible context for the repair model. The results show that a clear threshold effect appears: as long as less than half of the key regions are visible, repairs primarily fail.

The success rate only improves when coverage reaches between 50 and 75%. For more challenging tasks, even better context does not provide significant benefits if the problem already exceeds the model's capabilities.

Once critical points are available, additional irrelevant code has little effect on performance. An agent that reads too little performs worse than one that reads too much. The conclusion is clear: to improve future performance, less filtering and more reading are necessary. The code and data from this study are available on GitHub and Hugging Face.

About two years ago, the SWE-bench repository was created to test AI coding agents against real GitHub issue reports. Since then, several variants have emerged, covering more languages and more complex professional tasks. However, the underlying success metric is increasingly being called into question. A study by the METR organization revealed that many solutions accepted by automated reviewers would be rejected by project managers due to basic functional errors.