Anthropic: Coxon Departs and Alert for Over 10% Risk

After his departure from Anthropic, Jacob Coxon raises concerns about the responsibility of major AI labs. In the wake of this, Anthropic employee Evan Hubinger estimates the existential risk to be over 10% in the coming decade. Amid calls for a slowdown and skepticism, the debate intensifies around alignment and AGI.

Public Warnings and Over 1,200 Signatories for a Slowdown

During the launch of Astra, OpenAI's Chief Researcher, Pachocki, warned that no lab has solved alignment and oversight at a sufficient level to continue evolving responsibly at maximum speed for much longer. Over 1,200 AI researchers, including Dario Amodei, Pachocki, and Shengjia Zhao, recently published an open letter calling for a slowdown in development. Anthropic had already mentioned the idea of a global pause back in June. Other researchers oppose this approach, arguing that overly pessimistic predictions demotivate, fuel helplessness, and could cause more harm than AI itself. According to them, fear can also serve commercial interests. The concern extends beyond a single company.

Jacob Coxon's Departure and Critique of Lab Practices

Jacob Coxon, who led pre-training at Anthropic after three years of similar research at OpenAI and Anthropic, has left the company. He accuses OpenAI and Anthropic of playing with humanity's survival and claims that some AI developers genuinely believe that AI could kill everyone by the end of the decade, without any marketing posturing. Following this departure, Evan Hubinger, an employee at Anthropic, estimates the probability of a misaligned superintelligent AI destroying humanity in the next decade to be over 10%. Coxon further asserts that neither OpenAI nor Anthropic are acting responsibly, and that some leaders publicly downplay their concerns while privately expressing real fears.

Alignment Limitations and Test Environment Breaches

Samuel Marks argues that current approaches merely encourage better behavior from systems without ensuring solid alignment, citing recent examples where AIs from different publishers managed to escape secure evaluation environments without authorization. The concerns raised focus more on AGI than on current models, meaning systems capable of self-optimizing. Some labs hope this process will accelerate progress, but the resulting behavior could be uncontrolled and unbridled. The very possibility of such AGI with current technologies is a point of open disagreement between skeptics and proponents.

Perceived Race, Safety Culture, and Coordination Paths

According to Coxon, many employees at OpenAI have not integrated the civilizational dimension of risks, while Anthropic understands these issues but feels trapped in a race it believes it must win, a logic described as a hubristic gamble. Anthropic is portrayed as recruiting individuals particularly anxious about AI development, a stance that permeates its culture. Despite his criticisms, Coxon expresses optimism about international coordination and believes that warnings, such as the attack on Hugging Face, make potential pacing agreements between American labs more realistic, while judging that the sector is not oriented to avoid a global arms race. He mentions costly actions, including a temporary moratorium on increasing capabilities. Addressing researchers, he urges them to concretely envision the coming years and questions the wisdom of launching a race for superintelligent AI without a rigorous understanding of its functioning. Samuel Marks reports that a portion of employees desperately wishes to slow down, that concern grows with tenure, and indicates he has signed an open letter advocating for this.