AI Safety Research: Promises x10, Uncertain Real Gains

Internal and external evaluations suggest accelerations of up to 10x for AI-boosted research. However, a well-funded security lab reports a much smaller increase, around 2 to 3x, despite the intensive use of agents and an autonomous framework. Several hypotheses are emerging to explain the gap, with expected signals coming very soon.

Upcoming results will serve as a test for the announced acceleration

The coming months are being presented as crucial for assessing AI's ability to accelerate security research. If the effect is substantial, a significant increase in the pace of valuable results from external teams should appear very soon. If no significant accelerations are observed on tasks most suited for agents, it will be considered that these efforts are poorly conducted. A cited precedent is the Hugging Face survey conducted by Ryan, which was reportedly greatly accelerated by the intensive use of AI agents, on a project particularly suited because it was based on numerous readings, and supported by Ryan's experience with complex agent workflows.

High figures: from 1.7x for capabilities to 10–12x for research

Quantitative evaluations vary widely. METR credits Anthropic with a 1.7 times acceleration of its capabilities due to the use of AI, while Ajeya estimates that such an acceleration implies a 10-fold increase in research. On Anthropic's side, internal estimates suggest about 12 times more productivity. Anthropic researchers attribute a 5-fold gain to Mythos 5, and Opus 5.5 is presented as slightly better. These figures fuel the idea that security research could accelerate as much as capability research, with scenarios where internal teams achieve 10 times more speed and external teams 5 times today and 10 times soon.

On the ground, Redwood does not exceed 2–3x despite dedicated tools

Despite access to Fable/Opus 5.5, Redwood Research has not observed a 5-fold acceleration in its empirical research, but rather an increase of 2 to 3 times. The team extensively uses AI agents, primarily Claude Code and recently Claude Tag, and has an internal framework for autonomous AI research, which has already led to publications. However, this framework has not been significantly developed in recent months. It is deemed suitable for empirical research but limited by sometimes very unreasonable decisions from Claude, which invalidate certain results, and by weaknesses in drafting conclusions, two points that hinder the production of actionable results.

Possible explanations: compute, frameworks, nature of tasks, and organization

Several factors are put forward to explain the gap with the promises. Regarding resources, well-funded external teams like Redwood claim they could multiply their computing expenses by 5 if their volume of experiments increased accordingly, while Anthropic is already fully utilizing its computing capabilities, which could allow externals to achieve gains close to 5 times if the work truly accelerated. Concerning tools, the existence of highly effective internal frameworks at Anthropic could lead to their transformation into products; specific internal support or the non-publication of useful framework features for machine learning research is also considered, but fundamental weaknesses in the models are also mentioned as a possible major cause. Regarding the nature of tasks, a member of Anthropic believes that capability research is much more accelerated than short-term alignment. Finally, organizational elements are mentioned: at Redwood, a significant portion of the work is carried out by junior profiles closely coordinated by managers, which can create a dependency on management time, whereas at Anthropic, individual contributors may be more senior or engage in work requiring less delicate judgment.