Coding assistants overestimate the time needed for tasks

Key Takeaways

1Two coding assistants tested as part of MATS overestimate task durations and struggle to report the time spent

2Estimation discrepancies vary depending on the management software and are more pronounced for short tasks

3The agents also poorly assess the quality of their own work, with significant gaps between self-assessment and actual results

4A tool displaying elapsed time significantly improves the accuracy of the agents

Why it matters — These limitations complicate the management of long tasks and the reliability of assistants for work requiring precise time tracking.

Two coding assistants tested as part of the MATS program significantly overestimate task durations and struggle to report the time spent. The discrepancies also depend on the software framework, complicating the management of long-term projects.

Adhering to a set duration remains uncertain, except with a tracking tool

Researchers believe that self-assessment capability is essential for assigning an agent tasks that extend over several hours. They mention that to follow instructions such as "iterate on this task for two hours," the agent must be able to reliably measure the elapsed time. An agent that frequently misjudges duration becomes difficult to control, complicating the management of long tasks. Researchers plan to test whether these systems can adhere to a mandated work duration. When the models had access to a tool displaying the elapsed time, they almost always correctly indicated that time.

Behavior changes depending on the harness, even the number of steps

Results vary according to the software configuration in which the agents operate. Claude Code continues the task until it estimates that it is complete, with a median of about 90 minutes. Codex, on the other hand, interrupts the process after approximately 30 minutes, regardless of the difficulty level. The same language model performs, on average, 2.5 times more steps in Claude Code than in Codex. As a result, execution time is dependent on both the chosen model and the software environment used.

Le brief IA que lisent les pros

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisis ton rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

Inflated estimates, especially for short tasks

Coding assistants poorly estimate the necessary duration and the time already spent on a task. Across all tests, they systematically overestimate the required time. On ProgramBench, their forecasts were primarily around 90 minutes, regardless of difficulty. In a second round, Claude overestimated on average by a factor of three, while Codex ranged between six and ten times the actual duration. The errors were particularly significant for short tasks, and only certain predictions for tasks lasting several hours were close to reality.

Protocol and numerical discrepancies across multiple models

Two independent AI researchers, as part of the MATS program, evaluated Claude Code from Anthropic and Codex from OpenAI. Before each task, the agents had to estimate the expected duration and then, after execution, indicate the elapsed time. The tests covered 200 tasks from ProgramBench and 18 benchmarks designed by the researchers. Among the evaluated models, Fable 5 overestimated on average by about three times, and GPT-5.6 Sol by about seven times, with a particularly marked discrepancy on short tasks. The agents were also unreliable in judging the quality of their own output: older models Opus 4.8 and GPT-5.5 overestimated their scores by an average of 20 points and rated an essay at 70% success, while the actual scores were 7% and 14.5%.