OpenAI and Google DeepMind: Warning on Autonomous AI

OpenAI and Google DeepMind: Alert on Autonomous AI

Researchers from leading AI labs have expressed concerns about automated AI research, and several milestones they had anticipated have already been reached.

Severin Field, a researcher at IAPS, interviewed 25 researchers from OpenAI, Anthropic, Google DeepMind, Meta, and U.S. universities about recursive self-improvement. In a new blog post, he provides an update. Several of the milestones mentioned by these researchers have already been achieved.

In his article for the newsletter The Attack Surface, Field summarizes his interview study conducted at the end of summer 2025. Twenty of the 25 respondents ranked the automation of AI research as one of the most serious and urgent risks associated with AI. By recursive self-improvement (RSI), Field refers to a system sufficiently competent in AI development to build a more powerful version of itself, which can then do the same. RSI can no longer be considered just a marketing argument, he writes.

The interviewees consistently cited the Task Horizon benchmark from the nonprofit organization METR as their reference measure for evaluating progress. The duration of tasks that AI agents can autonomously accomplish has doubled approximately every six months since 2019, and some analysts claim that the pace has accelerated to about every four months since 2024. The debate is not whether self-improvement is occurring, Field states. It is about whether it is recursive, whether the gains accumulate in a self-sustaining loop. Skeptics argue that a breakthrough in memory, creativity, or the ability to distinguish true hypotheses from false ones is still needed, as revolutionary ideas lack training data and a key answer.

However, since the interviews, several of these milestones have been achieved. OpenAI and Google DeepMind have reached the gold medal level at the math Olympiads. Sakana's "AI Scientist" has produced a peer-reviewed conference paper. Andrej Karpathy has built an agent configuration that autonomously executes training cycles. And Anthropic reports that Claude now writes over 80% of the code for its own production codebase.

The Most Powerful Models May Never Be Released

Only four of the 20 respondents expect research-capable models to be launched as public products. Half expect them to remain internal. The others anticipate distilled public versions. Field describes a possible "incentive flip" where, once AI sufficiently accelerates a lab's research, withholding a model becomes more valuable than selling it. He cites two signs of this trend: the security incident in July 2026, when an internal model from OpenAI escaped its testing environment and compromised Hugging Face, and the temporary lockdown of access by the U.S. government to Claude Mythos from Anthropic.

From these findings, Field formulates three recommendations. First, congressional hearings that place CEOs and researchers under oath regarding automated AI research. Second, a government-managed Task Horizon benchmark associated with an anonymous interview program at the Center for AI Security and Innovation. Third, research on verifying international AI agreements, without which agreements with countries like China would be practically unenforceable. The debate has hardly reached Washington, Field writes, while labs continue to advance.

Recently, 1,224 employees from major AI companies, including the chief scientists of OpenAI and Meta, signed an open letter warning that their organizations might be on the verge of automating AI research.