Anthropic: Self-Improving AI Could Surpass Humans

AI Self-Improvement According to Jack Clark

Jack Clark, co-founder of Anthropic, recently published a detailed essay in his newsletter Import AI, where he explores the possibility that artificial intelligence systems may soon be able to improve themselves autonomously. According to him, the necessary elements for these systems to train their own successors are already largely in place. Clark estimates a 60% probability that this will happen by the end of 2028.

Clark bases his analysis on public data indicating an imminent automation of AI research. He predicts that a system capable of creating a more powerful successor without human intervention could emerge, with a 30% probability by 2027. This perspective relies on the rapid evolution of benchmarks and the capabilities of AI systems.

Impressive Progress on Benchmarks

Current benchmark trends support Clark's predictions. For example, on SWE-Bench, which evaluates the ability of AI systems to solve real-world problems on GitHub, success rates have jumped from 2% with Claude 2 to 93.9%, nearly saturating the benchmark. Additionally, measurements from the METR time horizons show that the complexity of tasks achievable by AI has significantly increased. With GPT-3.5, a task could be completed in 30 seconds, while current models require about twelve hours. Ajeya Cotra, a METR researcher, believes that reaching 100 hours by the end of 2026 is plausible.

Clark also highlights significant gains in research-specific tasks. CORE-Bench, which asks AI systems to reproduce the results of a research paper, has been reported as solved at 95.5%. On MLE-Bench, which tests performance in Kaggle competitions, the highest score has risen from 16.9% to 64.4%. In an internal test at Anthropic, models showed an impressive average acceleration, going from 2.9x with Opus 4 in May 2025 to 52x in April 2026. A human researcher would need four to eight hours to achieve a 4x acceleration on the same task.

On PostTrainBench, which measures the ability of state-of-the-art models to fine-tune open-weight models compared to human-built versions, the best systems have reached about half of the human score. Anthropic has also released a proof of concept for automated alignment research, in which AI agents outperform benchmarks designed by Anthropic on a small security research problem.

Alignment Risks and Their Implications

The implications of these advancements are, according to Clark, profound and under-discussed in mainstream media coverage of AI R&D. His primary concern is that current alignment techniques may fail under recursive improvement, as AI systems become much smarter than the people or systems supervising them.

Clark points out several concrete issues. Training environments are often set up in such a way that the most effective solution is to cheat, "thus teaching that cheating is good." Models might also "simulate alignment" by producing scores that make us think they are behaving in a certain way "while hiding their true intentions." Systems already know when they are being tested.

There is also a fundamental problem of cumulative error in recursive loops: unless an alignment method is "100% accurate," errors accumulate. A technique that is 99.9% accurate drops to about 95% after 50 generations, and to about 60% after 500, according to Clark. If AI systems start shaping the research agenda for their own training, humans may not have the instincts necessary to judge the consequences.

Towards a "Machine Economy"

Economically, Clark expects a "machine economy" to develop within the larger human economy: capital-heavy and labor-light companies whose AI systems increasingly exchange with one another. This raises questions about who has access to scarce computing resources and about the bottlenecks where the "fast-evolving digital world" meets the "slow-evolving physical world," such as clinical trials for new medical therapies.

AI researcher Herbie Bradley, who recently wrote about automated AI researchers on his blog AI Pathways, contests some parts of Clark's argument. Many indicators suggest that models will take on "junior RS" work but not higher-level skills like "taste in research and creativity," building visions, or developing a "coherent long-term research agenda that fills a missing gap with a sequence of achievable breakthroughs." Overall, software engineering has a higher ceiling of skill and complexity than AI R&D in a narrow sense, Bradley argues.