Anthropic: No Plan to Secure AI Despite >10% Risk

Key Takeaways

1Evan Hubinger, security lead at Anthropic, estimates that there is over a 10% risk that AI could kill all humans by the end of the decade

2According to Hubinger, Anthropic does not yet have a plan to ensure the safety of advanced AI

3Jacob Coxon has resigned from Anthropic, denouncing a race towards uncontrollable superhuman systems

4The industry continues to pursue the goal of AI self-improvement, which has not yet been achieved

Why it matters : These public statements illustrate the seriousness of internal concerns regarding the safety of advanced AI and the speed of its development.

Evan Hubinger, a security executive at Anthropic, claims that there is currently no plan within the company to ensure the safety of advanced AI. At the same time, he estimates the probability that AI could kill all humans by the end of the decade at over 10%, while a colleague has just resigned, denouncing a reckless race.

Anthropic acknowledges the lack of a safety plan for advanced AI

Evan Hubinger states that Anthropic does not yet have a strategy to ensure that advanced AI remains safe and aligned with human values, adding that the company is not evidently on track to develop one. Jacob Coxon asserts that companies are engaged in a competition to be the first to develop advanced systems, despite the risks. This context comes as companies in the sector prepare for anticipated IPOs and the pace of system development accelerates. Furthermore, companies must manage the fallout from several incidents involving out-of-control agents and highly publicized warnings regarding the monitorability of cutting-edge models.

A security executive estimates extinction risk at over 10%

Evan Hubinger, who leads a security team at Anthropic, expresses his concern about self-improving AI and believes that advancements in this area are progressing faster than expected. He personally estimates the chance of AI potentially killing all humans in the next decade at more than one in ten. He also states, "We truly believe that AI could kill all humans."

Le brief IA que lisent les pros

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

Self-improvement: a pursued goal, not yet achieved

For several years, industry players have expressed concerns about AI systems capable of self-improvement, which could escape human control through recursive enhancement. So far, such an AI system has not been designed. However, some companies are actively working toward this goal, and a significant portion of the existing AI code is already generated using AI.

A high-profile resignation and a history of departures for security reasons

Just hours before Evan Hubinger's statements, Jacob Coxon announced his resignation from Anthropic on X. This researcher, who trained AI systems at Anthropic after working on similar systems at OpenAI, cites a perceived lax approach to safety as the reason for his departure. He accuses Anthropic and OpenAI of rushing toward self-improving superintelligence and "playing with our lives," while asserting that some AI designers genuinely believe it could kill all humans by the end of the decade. His departure is presented as one of the most publicized exits from Anthropic, a company founded by former OpenAI members in response to safety concerns. In recent years, several researchers have also cited safety reasons for leaving OpenAI.