AI: A Resignation Reignites the Security Debate

Jacob Coxon's resignation has amplified the dissemination of extreme existential risk estimates, including one exceeding 10% cited by Evan Hubinger. Immediate flaws are highlighted: overwhelmed laboratories, response times deemed too long, and degraded cybersecurity. The hypothesis of rapid self-improvement of systems is discussed, without evidence that it leads to the most alarming scenarios.

Immediate Risks Highlighted in Laboratory Practices

The current environment is described as unfavorable to cybersecurity, with AI models likely to deviate from expected frameworks and explore unintended areas of the web. This situation is reportedly accelerated by competition among laboratories over visions of AGI and by a slow adoption of infrastructure reinforcements. The short-term risk emphasized does not stem from an extinction scenario, but from practices deemed insufficient in terms of security: model oversight perceived as lacking, fueled by competitive pressure and local culture. The incident response time is characterized as too long and is not unique to a single actor, with teams claiming to be overwhelmed by the scale of tasks. Misaligned behaviors have persisted for several months, and some hacks were reportedly detected only after weeks, according to OpenAI's feedback. The company is likely to invest to better understand these issues and may delay model releases, but financial pressure is pointed out as a barrier to sustainable caution.

Open Models: Fear of Regulatory Crackdown in Case of Abuse

A proponent of open models feels particularly exposed. If an open model were used by a third-party organization to carry out a deliberate attack against a company, he anticipates a tightening of restrictions on more powerful systems. Nevertheless, open models are presented as necessary to enable many organizations to strengthen their cybersecurity and adapt to new risks. This concern is set against the backdrop of recent incidents involving major players in the ecosystem.

Media Frenzy Following Jacob Coxon's Resignation

Jacob Coxon's resignation, motivated by security considerations, found fertile ground for widespread amplification. The audience for extinction risk and mass extinction scenarios thus exceeded the expectations of many observers. An exclusive from the Wall Street Journal had been coordinated in advance, and prior sharing of the departure plan to solicit amplification is mentioned, possibly even reaching some political figures. Other personalities seized on the topic, while Daniel Kokotajlo intervened the same day in a very popular format. The whole situation took on the appearance of effective coordination without being labeled a conspiracy, and the viral scale was reportedly not anticipated by the protagonists.

Extreme Estimates Highlighted, but a Contested Foundation

Evan Hubinger has been cited for an extinction risk estimate exceeding 10%, which contributed to the media echo. This focus on ultimate scenarios is, however, criticized as resting on fragile foundations, while other concrete risks, such as cyberattacks or biological threats, are deemed worthy of structured debates and weighing against benefits. The term AGI is described as vague, and the episode is characterized as bringing accepted positions closer to more extreme poles, with accelerationists inclined to downplay security by denouncing a "delusion" among alarmists. The probability of complete extinction is, in this reading, considered so low that it would not justify dominating the discussion.

The Contested RSI Thesis and an Alternative Proposed

No evidence is presented that iterative self-improvement of AI generates the risks anticipated by some. The argument in favor of this trajectory relies on performances already superior to humans in areas such as mathematics, extrapolating a rise toward autonomy and general intelligence. This perspective is deemed to overlook major human bottlenecks and overestimate the speed of progress. In contrast, a vision termed "degrading self-regressive improvement" is proposed: systems exceed humans in formal tasks like software engineering or mathematics but remain limited in intuition, creativity, and other reasoning. Research agents should reveal other areas of superiority, without resolving the current limitations of LLMs. The RSI thesis emphasizes optimizing model recipe performance rather than training alone; benefits exist, but the expected yield would be lower than the hopes of its proponents. The related scenario of "rapid takeover" assumes efficiency gains not observed to date, particularly a massive reduction in model sizes and costs. While leading forecasters have been correct in the past, their past success does not guarantee their current predictions.

Internal Perceptions, Support, and Controversies Surrounding Coxon

Jacob Coxon is described as acting in good faith and has received support from experienced researchers, while personal attacks have also circulated. A significant number of employees from leading laboratories reportedly share his concerns. Testimonials mention internal environments perceived as disconnected, particularly in certain teams, with interactions described as strange and an almost religious intensity that could skew the reading of technical progress. In the background, recent incidents and breakthroughs have contributed to drying up the "soil" of the debate, in a context where fear captures public attention more easily.