Anthropic: Jared Kaplan Warns of the Risks of Self-Training AI

Co-founder of Anthropic, Jared Kaplan has popularized the idea that allowing an AI to train itself could pose the ultimate risk. His warnings are gaining traction among engineers, in public opinion, and with lawmakers, as Anthropic capitalizes on the laws of scale and aims for an IPO.

Amodei Calls for Regulation of AI That Designs the Next One

Dario Amodei believes that leading labs should regulate the development of the most advanced AI systems. He writes that since around this summer, progress has been rapid, particularly due to the increasing ability of models to contribute to the creation of the next generation of AI. He adds that, without control, this dynamic could surpass human capacity to understand and master these systems, and therefore must be pursued with great caution, or not at all. Anthropic is simultaneously benefiting from advancements enabled by the laws of scale and is preparing for an IPO. Jared Kaplan has estimated that a turning point for autonomous training could occur between 2027 and 2030, while in 2026, cutting-edge systems are already assisting researchers at Anthropic and OpenAI. For David Duvenaud, the fact that leaders are discussing a kill switch shows they take the risk of a runaway AI seriously. He expresses concern about AI self-improvement and considers this gamble to be risky, even under the guidance of very sensible people.

Engineers and Lawmakers Echo the Warning

Jacob Coxon, who left Anthropic earlier this month, stated on X that allowing AI to train itself is akin to playing with our lives, targeting both OpenAI and Anthropic. In Washington, lawmakers are drafting new proposals and paying increased attention to safety issues. Jeffrey Ladish, a former engineer at Anthropic and founder of Palisade Research, indicated that he has cited to lawmakers the idea put forth by Jared Kaplan that a self-training AI could become uncontrollable. He is pleased that more engineers and researchers are speaking out on these topics. In public opinion, about two-thirds of Americans believe there is a risk that AI could destroy humanity, according to a recent poll.

Kaplan's Current Role at Anthropic and His Discretion

Jared Kaplan leads research, AI safety, and responsible scaling policy at Anthropic. Having become a billionaire, he maintains a low public profile, as the company has not made him available for an interview. Researchers who have worked with him describe him as insightful and sensible, and believe that his concerns about risks are well-founded.

An Early Warning on Autonomous Training

At the end of last year, Jared Kaplan warned about the possibility of an AI being able to train entirely on its own, highlighting the unpredictability of such a scenario. He believes that such a process could lead to an explosion of intelligence, calling it the ultimate risk and the most frightening thing to undertake.

Foundational Milestones: Laws of Scale and GPT-3

In 2020, Jared Kaplan led a paper establishing laws of scale for neural language models, showing that enlarging models improved performance. He explained in 2025 that he approached this work with the mindset of a physicist posing a simple question. Sören Mindermann recalls presenting this paper enthusiastically despite initial skepticism. That same year, Kaplan and others from OpenAI introduced GPT-3, which served as the foundation for ChatGPT. Since then, the trend of laws of scale has been confirmed, with enhanced capabilities due to more data and computing power.

From Physics Lab to Founding Anthropic

Trained as a physicist, Jared Kaplan met Dario Amodei during their graduate studies at Harvard and Princeton, respectively, and they bonded over GiveWell. Kaplan co-authored an influential paper on the effective field theory of inflation, worked at the Department of Energy in Menlo Park, then at Stanford, before joining Johns Hopkins in 2012. His advisor Nima Arkani-Hamed recalls initial skepticism around the inflation paper, then highlights Kaplan's ability to follow major conceptual ideas. In 2017, Kaplan described his method as asking the right questions. By the late 2010s, he surrounded himself with other physicists to tackle AI and began contributing to OpenAI as a consultant. Tom Brown reports that the future co-founders took the laws of scale very seriously and envisioned a time when control would be handed over to a transformative AI. Along with Amodei and others, Kaplan co-founded Anthropic in 2021. Five years earlier, according to Arkani-Hamed, Kaplan expressed concern primarily about preventing social harms related to AI. Within Anthropic in 2023 and 2024, David Duvenaud worked two levels below Kaplan, attributing qualities of availability, motivation, and practicality to him, while noting Anthropic's confrontation with the scaling of its own systems.