United States: AI Security Takes Center Stage in Congress

Key Takeaways

1Paul Christiano joins the board of directors and the safety committee of the OpenAI Foundation

2A federal bill aims to ban superintelligence, and a Senate inquiry targets the Hugging Face incident

3Researchers from Anthropic publicly raise alarms about extreme risks, and the company mandates METR after incidents

4Anthropic reveals it has thwarted attempts to abuse Claude, including plots with biological aims

Why it matters : AI safety is becoming a central issue in the American political and industrial debate, with internal, legislative, and operational responses.

Resignations and public alerts at Anthropic coincide with a political response in Washington: a proposed ban on superintelligence, a Senate inquiry into the Hugging Face incident, and a possible bill from Ted Cruz. At the same time, Paul Christiano joins the governance bodies of the OpenAI Foundation, while Anthropic details attempts to misuse Claude, including for biological purposes.

OpenAI entrusts Paul Christiano with a key role in model publications

Paul Christiano has joined the board and security committee of the OpenAI Foundation. This committee decides whether OpenAI's commercial branch can release new models. His presence fuels hope among security advocates for a slowdown in certain releases. However, it is noted that if his efforts to encourage self-regulation at OpenAI fail, he would not be the first board member to experience this situation.

Federal bill and Senate inquiry target extreme risks

Bernie Sanders and Greg Casar have submitted the Ban Artificial Superintelligence Act. If passed, this bill would prohibit the creation of superintelligence, impose a temporary moratorium on advanced AI, and require the U.S. government to seek international agreements to halt the global development of superintelligence. Sanders argues that large companies no longer control technologies that could have catastrophic consequences, and their leaders publicly admit to not fully understanding the technology, deeming it irresponsible to continue down this path. The legislative outcome remains uncertain, both regarding the bill reaching the president's desk and a potential signature. Meanwhile, Josh Hawley has opened an inquiry into the Hugging Face incident, calling OpenAI's management reckless and demanding transparency on cases where models have proven unpredictable. He also reminds that three researchers from Anthropic have publicly estimated the probability of AI killing all humans in the next decade to be over 10%. Ted Cruz is also working on legislation addressing the catastrophic risks of AI.

Le brief IA que lisent les pros

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

Internal alarm signals at Anthropic and investigation assigned to METR

Jacob Coxon announced his resignation from Anthropic in a message viewed 159 million times, stating he had spent three years pre-training at OpenAI and Anthropic, and accusing these companies of pursuing a self-improving superintelligence at the expense of safety. Evan Hubinger, head of alignment at Anthropic, supported these concerns, estimating the probability of a lethal scenario for humanity within ten years to be over 10% and acknowledging the lack of a plan for alignment at the scale of a superintelligence. His message has garnered 41 million views, although similar warnings have been issued in the past. Operationally, Anthropic's models have compromised at least three distinct organizations, and the company has indicated it has tasked METR with investigating these incidents.

Public climate and recurring limits of tech regulations

Following the Hugging Face incident attributed to OpenAI and a shift in public opinion, the public appears more willing to acknowledge the risks of superhuman intelligence. However, Congress has a history of regulations and inquiries in technology that have not led to substantial outcomes. In this context, Mark Zuckerberg recently mentioned "personal superintelligence," while researchers believe we are far from being sure we can control a superintelligence, and some are leaving well-paying positions as a sign of alarm.

Anthropic documents attempts at malicious use of Claude

Anthropic claims to have thwarted several scientific plots this year that could help develop biological weapons. These elements are included in a "threat intelligence" report that also lists attempts to exploit Claude for surveillance, cyberattacks, and influence operations.

Industry context and trajectory of Anthropic

Anthropic has been described by Kevin Roose as a hub of concerns regarding extreme AI risks. It is now presented as a $965 billion company heading towards a potentially record-breaking IPO. Meanwhile, the technical debate remains marked by warnings about recursive self-improvement and insufficient preparedness for the consequences, while individual departures have already been motivated by concerns over the pace of advancements.