Anthropic: METR Highlights a Leap and Areas of Uncertainty

Key Takeaways

1METR observes at Anthropic an acceleration in the development of models, about twice as fast as a year ago

2Part of this progress relies on unpublished internal models, including three recently trained ones

3Anthropic was able to censor information from the report, published after 11 weeks of investigation

4The report raises concerns about the safety and speed of AI improvement

Why it matters : The pace of progress observed at Anthropic raises questions about the ability of safety measures to keep up with the rapid evolution of AI.

METR describes a significant acceleration in the performance of Anthropic's models, driven by unpublished internal systems, after 11 weeks of observation at the company. Anthropic was able to censor parts of the report as the company prepares for an IPO, while its CEO has publicly called for a slowdown.

Censorship of the Report and Unpublished Internal Models

Anthropic had the contractual ability to remove information from METR's final report and extensively exercised this right. METR indicates that it had access to non-public data, including confidential research and performance metrics of the models. Over the past six months, Anthropic has completed the training of three unpublished models, and its researchers are using more advanced systems than those offered to the public. A significant portion of the productivity gains is attributed to these internal models. These developments come as Anthropic prepares to go public, while its CEO, Dario Amodei, has publicly called for a slowdown in AI development.

Accelerated Progress Measured by METR

According to METR, Anthropic's models are progressing about twice as fast as last year, with benchmark scores indicating the equivalent of two years of historical progress in just one year. METR researchers believe that this acceleration cannot be solely explained by an increase in computing power. No signs of a slowdown have been observed. The report, published on a Thursday, is based on an 11-week investigation conducted within Anthropic's R&D by this organization specializing in AI safety.

Le brief IA que lisent les pros

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

Reactions and Debate on Safety

The publication of the report has sparked reactions among AI safety researchers and commentators, who note a contradiction between Anthropic's stated commitment to safety and its contribution to the acceleration of a technology that some deem risky. The findings suggest that recursive improvement, where AI systems autonomously design more powerful successors, could be closer than expected. Doubts remain about the current safety measures' ability to keep pace, and it is suggested that even the most ambitious projections regarding AI progress could be underestimated. The results also raise questions about how quickly systems could improve with little or no human intervention.

Scope and Focus of METR's Investigation

METR's investigation is distinct from a separate program that allowed security researchers access to Anthropic's systems following highly publicized hacking incidents that occurred during the summer. METR considers these findings to be among the clearest indications of an acceleration in AI development. They illustrate the speed at which AI could contribute to its own progress and are seen as potentially prompting some skeptics to reassess their perception of the pace of advancement, even advancing their estimates regarding the emergence of systems capable of fully automating tasks such as software engineering.