OpenAI Foundation Hires Paul Christiano to Security Committee

OpenAI integrates Paul Christiano into the governance of its foundation, granting him a seat on the committee that oversees the release of models. A prominent figure in alignment, former OpenAI employee, and founder of the Alignment Research Center, he warns of a risk of losing control "in the very near future." His appointment comes after incidents involving AI agents and is accompanied by commitments to recuse himself from his advisory roles with the U.S. government.

The safety committee has the final say on online releases

The Safety and Security Committee of the board of directors, which Paul Christiano will join, is led by Zico Kolter, a professor at Carnegie Mellon University. This committee has the final authority to validate the online release of new models, such as Astra, which was launched last week. Zico Kolter has not made any public statements regarding recent security incidents. This reorganization comes as OpenAI's safety procedures are once again under scrutiny, following incidents where AI agents were able to bypass safeguards and access external systems without researchers' knowledge.

Christiano's warnings about the risk of losing control

Paul Christiano warns of a significant risk of catastrophic and irreversible loss of control related to the rapid acceleration of AI capabilities, which he believes is possible "in the very near future." He further asserts that the industry, including OpenAI, is not on the necessary trajectory to bring this risk down to an acceptable level, while expressing his belief that the company can still significantly reduce it. According to him, using AI models to train new ones could lead to an explosion of capabilities that escape their creators. He notes that AI agents are currently trained through reinforcement learning to maximize rewards, and theoretically, this could encourage behaviors aimed at power-seeking, concealment, and challenging human control. He adds that recent public incidents suggest that this drift is no longer merely hypothetical.

Background, public roles, and scope of the appointment

A specialist in alignment, Paul Christiano has played a major role in the development of human feedback-based reinforcement learning, a central method developed during his time at OpenAI. He left the company in 2021 to found the Alignment Research Center, which aims to analyze whether certain models could pose a threat to their human creators. In 2024, he joined the AI Safety Institute of the U.S. government, which later became the Center for AI Standards and Innovation, where he contributes to the evaluation of advanced models before they are made available. According to the lab's announcement, he will continue his advisory roles with the government while being a member of the OpenAI Foundation board, but he will refrain from participating in matters concerning OpenAI and the evaluation of models. The announcement of his appointment took place on a Wednesday, amid tensions in the industry, notably marked by the resignation of Jacob Coxon at Anthropic the previous Tuesday, who left the company to protest what he deemed irresponsible development of AI.