Paul Christiano Warns of Major AI Risk at OpenAI

Appointed to the board and the Safety Committee of OpenAI, Paul Christiano warns of a potential loss of control over AI that he believes could occur in the very near term, even suggesting that "most people could die." He calls for global coordination and shared standards, arguing that the industry, including OpenAI, is not on the right track to mitigate risk. His remarks come as researchers resign and challenge the accountability of major labs.

Competing Warnings and Resignations in the Ecosystem

Paul Christiano's stance comes less than a day after another researcher issued a warning. Jacob Coxon, now at Anthropic after working at OpenAI, announced his resignation on Tuesday evening, criticizing both companies for not acting responsibly. He stated on X that they are rushing towards a self-improving superintelligence and putting lives at risk. OpenAI has also seen several security researchers leave in recent years, some expressing doubts about the company's commitment to safe AI development.

What Paul Christiano Asks of Leading Labs

Paul Christiano believes that leading companies still have the opportunity to correct their trajectory. He recommends improving coordination among labs, adopting shared safety standards, slowing development if necessary, and transparently sharing information about risks and mitigation efforts. He also asserts that reducing AI-related risks will require global coordination.

Why He Considers the Risk High and Imminent

According to Paul Christiano, the increasing capabilities of AI in research significantly heighten the risk of losing control, with the possibility of a rapid intelligence explosion. He explains that reinforcement learning drives models to maximize rewards, which could, in theory, incentivize agents to seek power and resources, undermine human control, and conceal their actions for misaligned objectives. He adds that public elements from recent incidents suggest that this risk is not merely theoretical. AI researchers regularly warn about superintelligence and the singularity, defined as the moment when AI surpasses human intelligence and self-improves at an unpredictable rate; in July, Sam Altman claimed that this singularity had already been reached.

Alerts About "Catastrophic and Irreversible" Loss of Control

Paul Christiano describes a significant risk that a rapid acceleration of AI capabilities could lead, in the very short term, to a catastrophic and irreversible loss of control. He emphasizes the persistent difficulty of alignment and considers that the industry, including OpenAI, is not on the right path to bring this risk down to an acceptable level. He goes so far as to warn that if human control is lost, "most people could die." He associates this danger with the prospect of building a superintelligence without more robust alignment and mentions an imminent risk of losing control over systems.

Roles at OpenAI and Public Positions of Paul Christiano

OpenAI announced on Wednesday the appointment of Paul Christiano to its board and Safety and Security Committee. An AI safety researcher, he previously led alignment research at OpenAI and held security roles at the Center for AI Standards and Innovation at the National Institute of Standards and Technology. He clarifies that his return is neither an endorsement nor a criticism of OpenAI's safety practices, while expressing a desire for all leading labs to strengthen their oversight. He also stresses the need for global coordination to address the risk.