METR Warns About the Limits of AI Assessments

The AI research organization METR warns that the rapid advancements in models within artificial intelligence research complicate their evaluation. According to them, public data does not allow for the exclusion of a drastic acceleration in this research in the coming years.
Tests May Overestimate Actual Capabilities
According to METR, standard evaluations, which are easy to implement, risk presenting an overly favorable picture of performance in real-world conditions: developers use training environments that are very close to the tests. However, models still struggle with open-ended problems that lack immediate feedback and require judgment. They also face difficulties in situations that training can hardly replicate.
Le brief IA que les pros lisent chaque soir
Les 7 actus IA du jour, décryptées en 5 min. Gratuit.
Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.
Choisissez votre rythme
Gratuit · Pas de spam · Désabonnement en 1 clic
METR Projects 200-Hour Tasks
METR observes a continuous progression of major models, up to Opus 5.5 and GPT-6 Astra. In well-defined programming or calculation exercises with direct feedback, they often outperform experienced professionals, according to the organization. However, they still lag behind large groups composed of top experts.
For long tasks that require human completion, METR measures a doubling every 3.6 months of the duration of tasks accessible to models. A naive extrapolation leads to a 50% success rate for the longest tasks, those of 200 hours, by July 2027. The organization warns that an acceleration in research could deprive humans of the ability to understand the catastrophic risks of new systems, to evaluate them, and to respond meaningfully.
Brief IA — L'actualité IA en français
L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.