Kevin Roose quantifies the risk of AI and suggests inspections

Journalist Kevin Roose advocates for strict oversight of AI models, even suggesting UN-style inspections. He estimates his own probability of a catastrophe at 10%, while reporting much higher estimates in Silicon Valley. He sees a window of opportunity to slow down and regulate, despite the global race and political divergences.

Risk Estimates Ranging from 10% to 99.9% Depending on the Sources

Kevin Roose estimates the probability of an AI-related catastrophe at 10%. He notes that he knows people who suggest much higher figures, up to 50%, 60%, and even 99.9%. According to him, it is not uncommon to hear assessments of at least 50% in San Francisco. He claims to be more optimistic than he was a few months ago, as the issue of AI risks is now a topic of public discussion. He believes it is still feasible to intervene, slow down, enact laws, and coordinate efforts to maintain control over this technology.

Laboratory Inspections and Integrated Evaluators to Regulate Models

Kevin Roose compares the current situation to nuclear arms control, while reminding that such control required resources that only a few countries possessed. He believes an inspection regime may be necessary, with a UN-like body tasked with verifying that AI models do not pose dangers. Proposals exist for integrated evaluators to work within OpenAI and Anthropic, having access to the systems without being employed by these companies. Roose states that there is virtually no regulation on these companies and considers this situation unacceptable. He also mentions specific risk probabilities related to AI, such as P(doom), P(disempowerment), P(pandemic), and P(job loss).

AI Race, Rivalries, and the "Bigger, Smarter" Hypothesis

Kevin Roose describes a paradox where the leaders most concerned about the power of AI are also those developing it the fastest, citing Dario Amodei of Anthropic and Sam Altman of OpenAI. He reports that these leaders observe a rapid progression of the technology. Dario Amodei, in his essay The Big Blob of Compute Hypothesis, argues that making systems smarter means making them larger. Roose adds that discovering a simple recipe combining computing power and data could trigger a race, making the technology strategically indispensable. He recalls that conflict has always been part of the AI field, while some critics accuse leaders of exaggerating apocalyptic scenarios to promote their companies.

Shifting Political Context and Risk of International Evasion

Kevin Roose observes that public opinion on AI is rapidly evolving, with growing concern among voters. He states that some advisors to Donald Trump oppose any regulation, while others from the national security community are more vigilant regarding AI risks. Roose discusses the possibility of an industrial pact to slow research on the most advanced models and believes that leading companies could act on their own in the absence of government regulation, despite their rivalries. He warns that even if major American companies slow down, other players, particularly in China, will continue to advance. He notes that Chinese models are inspired by existing models and that the technology is quickly becoming commonplace.

Roose's Recent Journey and the Subject of His Latest Book

Kevin Roose has left The New York Times to co-host a podcast for NPR. He has published "The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence," a book that compiles stories and biographies of technologists who have shaped the recent history of AI.