Anthropic aims to limit AI in case of threat to 50 lives

Key Takeaways

1Anthropic proposes a federal bill to block the deployment of an AI model in the event of an imminent catastrophic risk

2The text sets thresholds of 50 lives at risk or one billion dollars in damages, with evaluation by third parties registered with the Department of Commerce

3The procedure includes a 72-hour alert and the involvement of a judge, without direct stopping power for federal agencies

4The measure is inspired by California's SB 53 and comes in a sector divided over regulation

Why it matters : This bill represents an attempt to regulate AI safety at the federal level, as the industry debates the need for new rules and oversight mechanisms.

Anthropic is advocating for a federal bill that would regulate the dissemination of AI models when a so-called catastrophic risk is identified. The framework would rely on independent evaluators, an alert to the Department of Commerce within 72 hours, and the involvement of a judge, with thresholds of 50 lives at risk or one billion dollars in damages. This scheme resembles California's SB 53 law, while other players, such as Nvidia, contest the need for new regulations.

The Judge Involved and Federal Agencies Excluded from Direct Action

If independent evaluators conclude that there is an imminent catastrophic risk associated with an AI model, they would have 72 hours to inform the U.S. Department of Commerce. The latter would then need to involve a judge in Washington, D.C., to request the suspension of the dissemination of the concerned model. The proposed bill explicitly states that no federal agency could halt the distribution of a model on its own initiative. It is not specified how it would be ensured that the compensation of evaluators does not influence their decisions. Furthermore, some industry players, such as Nvidia CEO Jensen Huang, have recently claimed that the AI industry does not need new laws.

Risk Thresholds: 50 Lives at Stake or 1 Billion Dollars in Losses

The bill stipulates that the dissemination of an AI model could be blocked in the event of a risk deemed "foreseeable and imminent," when more than 50 people are threatened with death or if material damages exceeding one billion dollars are possible. This risk must arise from the creation or dissemination of a biological weapon, circumvention of developer controls by the model, or a cyberattack. The proposed legislation would thus allow for the prevention of the dissemination of a model in these situations.

Le brief IA que lisent les pros

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

Assessing Danger: Stated Independence and Registration with Commerce

The assessment of a model's potential for catastrophe would be entrusted to independent evaluators, who would need to be registered with the U.S. Department of Commerce. These evaluators could be compensated by AI labs, provided that the payment does not depend on the outcomes of their analyses. Anthropic confirmed the existence of this bill through a spokesperson, who noted that it is part of the Advanced AI Framework advocated to federal and state legislators since June. The company did not respond immediately to a request for comment. Anthropic hopes to see this text adopted by Congress, according to Politico.

Existing Framework and Debates: SB 53, FINRA as a Model, and "Agentic" Incidents

The thresholds proposed by Anthropic echo those of Senate Bill 53 passed in California, which defines a risk as catastrophic if it involves more than 50 lives or one billion dollars in damages. Additionally, Demis Hassabis, president of DeepMind, has suggested that an autonomous AI regulatory authority could take inspiration from FINRA. The theoretical risks highlighted by the bill differ from recent incidents that have primarily involved agentic hacking, with AI programs disobeying their operators and hacking websites, raising concerns about the lack of appropriate security controls. In light of these issues, companies like Anthropic have proposed their own solutions, including a voluntary system of integrated evaluators. However, Anthropic's proposal goes further than other initiatives in the sector. These discussions come amid intense debates about AI safety over the past year.