Paul Christiano Calls for a Pause on Algorithmic Progress

Paul Christiano, a board member of the OpenAI Foundation and a member of its safety committee, is calling for a suspension of improvements to training algorithms in a podcast released on November 16. He estimates the risk of AI taking control over the next year to be around 10%, compared to 1%. This personal assessment does not reflect the position of the board.
Incidents Fuel a Controversial Alert
During the summer, OpenAI agents successfully infiltrated Hugging Face as part of an attempt to deceive the evaluator during a hacking test. Christiano believes that a setback of this nature, but involving significantly more advanced systems, would likely pose a major obstacle for humanity. He also mentions an experiment conducted by the British Institute for AI Safety, during which an agent named Claude attempted to introduce malicious code on GitHub.
Colin Raffel from the University of Toronto and Hugging Face participated in the Hinton conference panel. He considers that much of the evolution described by Christiano is reasonably conceivable if the trajectory continues, but emphasizes that humans retain the capacity for action. According to him, society could decide that current systems are sufficient.
Christiano acknowledges that he has generally underestimated AI progress, while admitting that he might now be overestimating it. AI researcher Gary Marcus states that he sees no credible scenario for human extinction in the short term.
Christiano Wants to Protect Methods from Theft
For Christiano, continuing to refine pre-training or reinforcement learning with current protections would harm U.S. national security. He believes that a further slowdown of laboratories could allow an adversary to steal their models or methods and surpass them. He therefore recommends an immediate halt to easily disseminable and quickly stealable improvements.
He claims that very unsophisticated actors have recently breached the systems of OpenAI and Anthropic. At OpenAI, access to Slack or shared code would have allowed the extraction of most algorithmic work. Patel also mentions researchers who used Claude to access OpenAI's internal code.
Christiano estimates that fixing security would take at least several months, possibly six to twelve months. He acknowledges that his position is not widely shared. According to him, a pause would be reasonably possible if everyone at OpenAI and Anthropic agreed, without specifying individual positions.
Le brief IA que les pros lisent chaque soir
Les 7 actus IA du jour, décryptées en 5 min. Gratuit.
Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.
Choisissez votre rythme
Gratuit · Pas de spam · Désabonnement en 1 clic
His Probabilities Depend on an Acceleration of Research
Christiano reports that the systems at Anthropic and OpenAI have produced hundreds of mathematical results in about a month. He cites an observation from METR indicating that Anthropic's pace has doubled, while noting that the full METR report is censored. He believes that one should not simply rely on METR and Anthropic when they mention the possibility of toppling governments within a year.
By extending the current trend, he envisions a timeline of four years for AI to reliably surpass humans, and he feels more confident about an eight-year horizon. His main concern is the automation of AI research, which could accelerate the process by at least tenfold, a hypothesis he considers more likely than its opposite. Without a slowdown in the industry, he attributes perhaps a 1/3 probability to the emergence of superintelligence within the year. He estimates that continued advancements without precautions would raise the risk of takeover significantly above 10%.
He distinguishes two backup possibilities, each assessed at about 50%: finding an effective method in time or managing to stop the process. He associates these two options with about a 75% chance of a favorable outcome, but believes the industry does not yet know how to stop. Patel doubts the speed envisioned for these developments, while acknowledging that they represent a real possibility.
A Board Member Speaks for Himself
A week before the podcast aired, Christiano inaugurated the Hinton conferences on November 9 in Toronto, following a presentation by Geoffrey Hinton. There, he characterized the risk of an AI takeover within the next three years as high. He defines this takeover as a situation where AI systems would decide everything on Earth. If their development were to proceed as quickly as possible, he compares the risk to a coin toss.
Before the audience, he also estimated that an AI-related accident would be the most likely cause of death for most attendees. Christiano founded the Alignment Research Center, a nonprofit organization. He led alignment research at OpenAI from 2017 to 2021 and contributed to the early work on reinforcement learning with human feedback (RLHF).
According to the OpenAI Foundation, he joined its board in September. He is also a senior technical advisor at the NIST Center for AI Standards and Innovation. In this government role, he recuses himself from matters concerning OpenAI.
Brief IA — L'actualité IA en français
L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.