AI Security: Miles Brundage's "Triage" Against Counterpoints

Miles Brundage defends a "triage" approach in AI safety: addressing 80% of risks with a fraction of the available efforts, without claiming to avoid all severe scenarios. This position is based on signals of accelerating capabilities, while other voices highlight uncertainty regarding timelines, challenge the 80/20 idea, and propose different resource allocation strategies.

Contested Timelines and "Radical" Options in the Face of Uncertainty

Several researchers believe that AI timelines do not lend themselves to a single estimate. Toby Ord advocates for maintaining a broad distribution of time horizons, deeming it reductive to condense it into a single figure. He indicates that he has maintained this position, including recently. Eliezer Yudkowsky expresses reluctance to announce precise dates and emphasizes the historical rarity of accurate dated predictions, criticizing titles like "AI 2027" or invented p(doom) figures. In this context of uncertainty, Christoph Winter and Charlie Bullock advocate for a "radical option" to address the possibility of transformative AI within the next 10 years: investing extraordinary amounts of resources, contrary to more moderate approaches.

Miles Brundage's "80/20" and Scenarios Hard to Dismiss

Miles Brundage argues that in practice, one must accept a "triage" logic: mitigating 80% of risks with a fraction of the efforts that a world of slower progress and earlier governance would have allowed. He considers it likely impossible to completely rule out very bad scenarios, such as an unjust economic transition involving suffering and suicides, addiction and AI-related disorders, a rushed overhaul of education, AI-facilitated terrorist attacks causing thousands or millions of deaths, or enduring totalitarianism enabled by AI in certain countries. A former senior advisor for AGI preparedness at OpenAI, he left this role, believing he could be less biased outside the industry.

Perceived Urgency: Rising Capabilities and Shifting Bets

To support the urgency, Miles Brundage refers to his December 2024 paper, Time’s Up for AI Policy, where he deems it almost certain that AI will surpass human capabilities in most cognitive domains in the coming years. He sees the announcement of o3 as an indication that superhuman coding and mathematics are arriving sooner than many anticipated, and he believes the implications have hardly been explored. His positioning also fits into a landscape where expectations are evolving: the bet opposing him to Gary Marcus has seen, according to Metaculus, Marcus's probability of victory drop from 90% in July to 30% more recently.

Algebraic Proofs and Debates: Claimed Results in Mathematics

OpenAI has claimed to have solved the Millennium Prize problem of Navier–Stokes. If this claim were confirmed, it would represent a significant advancement beyond the trend; however, there is debate about the potential integration of another person's work into the training data. Observers report that the mathematical community sees signs of progress, while diverging on the automation of other components of the profession beyond proof production and the associated timelines. OpenAI also states that it has published 372 other results online, with a few more released just days ago.

Counter-Readings: Limits of Models and Effects of Scale

Voices are calling for caution in focusing on short-term scenarios. Richard Ngo fears a counterproductive scarcity dynamic and compares this stance to that of first responders, a role many are ill-suited for. Others consider that progress may be slower than some labs estimate, citing the absence of continuous learning and low sampling efficiency, as Dwarkesh Patel argues. Toby Ord contends that moving from pre-training scale to RL and inference changes the game, mentioning an order of magnitude of 100x in pre-training equating to 1000x in inference or 1,000,000x in RL, an effect that has been masked until now but is already prominent in inference and RL computing uses. Steve Newman adds that a longer "distance" to cover may only become apparent over time, citing models like Claude 5 or o3 achieving feats without succeeding in tasks like writing a tightly plotted novel.

Where Brundage's Alarm on Timelines Comes From

The alarm on timelines expressed by Miles Brundage stems from his thesis that an AI surpassing human capabilities in most cognitive domains is almost certain to emerge and be deployed in the coming years. He associates this horizon with recent announcements regarding performance in coding and mathematics that he considers ahead of expectations, and he has anchored his argument in a December 2024 paper supported by concurrent publications, including a tweet and a retrospective in March.