METR Expands AI Testing to Judgment-Intensive Tasks

METR has updated its assessment of AI capabilities by adding longer tasks and others that require more judgment. The results indicate that while some abilities continue to progress exponentially, models perform worse on judgment exercises than on programming tasks.
Estimated Durations and Challenging Assessments
The durations assigned to the new tasks are primarily based on estimates. Systematically measuring human time would require compensating individuals for hundreds of hours on certain exercises or mobilizing the few experts capable of completing them. Both the cost and the time involved make this approach prohibitive.
Measuring judgment presents another vulnerability: for the exercise focused on philosophical arguments, a single person evaluated most of the arguments. Thus, agents receive a score based on their agreement with this evaluator. Consulting more experts could change the results but would require additional time. Even within METR, employees do not all share the same opinion on the relevance of this exercise for automating AI R&D.
The difficulty in constructing and then scoring these exercises also explains their low number, which limits possible forecasts. Matt Putz, a technical staff member at METR, believes that judgment-dependent activities are particularly challenging to study and quantify. He does not expect to derive trends as solid and precise as those for tasks with easily defined criteria.
Model Proposals Remain Unusable at METR
To test skills that may be relevant in AI research, METR asks models to invent assessments that its teams could use. Its employees then review the proposals. Although the most advanced systems outperform nearly all humans in software engineering, none of the assessments produced by AI have been deemed useful for METR's research by its staff. In the other exercise, which involves evaluating philosophical reasoning, the models also remain below human level.
However, the results do show signs of progress. Recent systems, including Opus 5.5, tend to reach the "above the bar limit" category for assessment design, although they still do not produce tools usable by the organization. A year ago, all were classified as "clearly below the bar."
OpenAI reported a similar gap in its own usage on Wednesday. The company indicated that its models perform worse on real tasks assigned by its employees than on simpler, verifiable exercises from METR's initial battery.
Le brief IA que les pros lisent chaque soir
Les 7 actus IA du jour, décryptées en 5 min. Gratuit.
Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.
Choisissez votre rythme
Gratuit · Pas de spam · Désabonnement en 1 clic
The Graph Measures Human Work Duration
In the initial setup, humans and models performed the same exercises: the time taken by humans was used to define the difficulty, and then to position the successes of the models. This complex measurement method contributed to the misinterpretations of a widely circulated graph. It showed that the human duration for tasks successfully completed by the best systems doubled every four months. Progress on verifiable tasks does not seem to be slowing down.
The expanded battery includes more exercises, some of which would require much more time from humans, as well as activities with less easily resolvable outcomes. These were absent from the original setup precisely because their evaluation was difficult. Choosing a new research direction, for example, does not provide the same signal of success as a mathematical solution or a program whose operation can be automatically verified.
This difference also comes into play during training. Reinforcement learning involves multiplying attempts, then favoring paths associated with successes and weakening those linked to failures. Automatic verification of responses has favored performance in mathematics and programming. This method is less easily applicable when success remains ambiguous or can only be assessed after a long delay.
Researchers Diverge on Automated Research
The ability of models to transfer their progress on verifiable problems to open-ended activities remains uncertain. The latter would be essential for any potential automation of research and development. Earlier this month, METR estimated, in another assessment from which many passages had been removed, that AI could double the pace of AI advancements at Anthropic. This perspective has fueled concerns within part of the AI community about complete recursive self-improvement: systems would then design their own successors without human intervention.
Sayash Kapoor, a computer scientist and future professor at UC Berkeley, believes that knowing how to select the right questions, hypotheses, and experiments will be a major obstacle to this automation. Current failures on open exercises do not surprise him: in August, he and his collaborators indicated that models still do not know how to independently replicate human research in computer science. However, he reports a competing interpretation, suggesting that training on verifiable problems will eventually lead to successes in open tasks as well.
Kapoor welcomes more creative assessments and considers METR's expansion promising, while believing that considerable work remains necessary. Najoung Kim, a linguist and computer scientist at Boston University, sees the observed improvements as information about current capabilities but is uncertain about their extrapolation. The consulted experts believe it is still premature to provide a definitive answer on the possibility of automated AI research or its timeline.
Brief IA — L'actualité IA en français
L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.