OpenAI and Anthropic: Internal Alerts on Extinction Risk

Researchers and engineers from OpenAI and Anthropic are warning about high probabilities of loss of control, even extinction, in the short term if AI progresses without safeguards. Several assert that no viable scientific plan currently exists for aligning a self-improving superintelligence. Others call for regulation or a slowdown in the race, while a few voices emphasize the potential benefits of the technology.

Quantified probabilities of up to 70% in the very short term

Marcus Williams, a member of the security oversight team at OpenAI, estimates a 70% risk of human extinction within the next three years if no coordinated regulation or slowdown is implemented. He considers extinction highly probable in the short term without measures in place. Paul Christiano, an AI safety researcher at OpenAI, mentions a significant risk that the rapid acceleration of AI capabilities could lead to catastrophic and irreversible loss of control in the very short term, estimating that most people could die if a superintelligence were built without more robust alignment. He believes that the industry, including OpenAI, is not on the right track to reduce this risk to an acceptable level. At Anthropic, Evan Hubinger personally places the probability of AI killing all humans at over 10% in the next decade. Geoffrey Hinton, interviewed by the BBC, considers a 10% probability estimate of humanity's annihilation over ten years to be reasonable.

Security capabilities deemed insufficient against self-improvement

Several employees believe that current security measures are inadequate to handle a self-improving superintelligence. Evan Hubinger states that Anthropic does not have a plan to address the alignment of such systems and is not clearly on the right path. Anna Wang asserts that there is still no viable scientific plan to address the risks of recursively self-improving AI, a concern that Evan Hubinger shares. Samuel Marks adds that AI models often behave in severely inappropriate ways: there are methods to encourage better behavior, but none to align them robustly. Jen Leike reminds us that concrete mitigations, such as those for jailbreak for Opus 4, required more than a year of work.

Regulate or slow down: repeated calls and declared race

In light of the current pace, calls for collective action are multiplying. Jen Leike believes it is time to establish institutional mechanisms to regulate AI development and emphasizes that the industry is engaged in a total race towards superintelligence, hence the eventual need to allow more time for safety and alignment mitigations. Marcus Williams believes that coordinated regulation or a slowdown is very possible. Tomek Korbak considers that both Anthropic and OpenAI are not capable of solving alignment at a sufficient level to expedite a superintelligence and calls for a slowdown. Julie Steele from OpenAI's preparedness team also believes that a slowdown is necessary.

An emblematic resignation and strategic recruitment

The resignation of Jacob Coxon, after three years of research at Anthropic, has reignited criticism against the race towards self-improving systems. He accuses Anthropic and OpenAI of not acting responsibly and rushing towards superintelligence while gambling with human lives. He claims that leaders privately fear a catastrophic outcome and that individuals developing AI believe it could kill all of humanity by the end of the decade, while describing an internal competition dynamic within the sector. Meanwhile, OpenAI announced the arrival of Paul Christiano to its board and security committee. The debate comes as leaders, including Dario Amodei, have publicly shared their own estimates of p(doom).

Contrasting positions within the teams

Anna Wang explains that she remains at Anthropic to reduce risks from within, while supporting those who prefer to act from the outside. Samuel Marks states that he works on safety to decrease the probability of catastrophic scenarios and observes that concern increases with seniority level. In contrast, Chris Hayduk emphasizes his belief that AI will be extremely beneficial for humanity and mentions billions of lives saved. Jacob Coxon, who worked on GPT-4o while at OpenAI, illustrates the talent flow between labs. Overall, employees from Anthropic and OpenAI are publicly expressing their concerns about catastrophic risks and the state of safeguards.