Brief IA
›

METR Warns About the Limits of AI Assessments

Models & LLM·Tom Levy·

METR Warns About the Limits of AI Assessments

METR Warns About the Limits of AI Assessments
Key Takeaways
1METR warns that advancements in models complicate their evaluation.
2Standard tests may overestimate performance in real-world usage, according to the organization.
3A naive extrapolation projects a 50% success rate on 200-hour tasks by July 2027.
Why it matters : METR believes that an acceleration in research could undermine humanity's ability to assess the catastrophic risks of new systems.

The AI research organization METR warns that the rapid advancements in models within artificial intelligence research complicate their evaluation. According to them, public data does not allow for the exclusion of a drastic acceleration in this research in the coming years.

Tests May Overestimate Actual Capabilities

According to METR, standard evaluations, which are easy to implement, risk presenting an overly favorable picture of performance in real-world conditions: developers use training environments that are very close to the tests. However, models still struggle with open-ended problems that lack immediate feedback and require judgment. They also face difficulties in situations that training can hardly replicate.

La lettre du soirGratuit · 19 h

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

METR Projects 200-Hour Tasks

METR observes a continuous progression of major models, up to Opus 5.5 and GPT-6 Astra. In well-defined programming or calculation exercises with direct feedback, they often outperform experienced professionals, according to the organization. However, they still lag behind large groups composed of top experts.

For long tasks that require human completion, METR measures a doubling every 3.6 months of the duration of tasks accessible to models. A naive extrapolation leads to a 50% success rate for the longest tasks, those of 200 hours, by July 2027. The organization warns that an acceleration in research could deprive humans of the ability to understand the catastrophic risks of new systems, to evaluate them, and to respond meaningfully.

Brief IA — L'actualité IA en français

L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.