Nate Soares (MIRI) Wants to Halt Training of General AI

The president of MIRI, Nate Soares, estimates the probability of human extinction related to AI at over 50% within the next ten years and says he has stopped contributing to his 401(k). He calls on governments to suspend the training of general-purpose AIs and redirect efforts towards targeted uses. This position contrasts with that of Elon Musk, who predicts an abundance that will make retirement savings "irrelevant" in 10 or 20 years.

Suspend Training of Generalist AIs, Redirect Towards Targeted Uses

Nate Soares urges governments to halt the training of general-purpose AI systems. He proposes focusing technological capabilities on specific applications, such as cancer research and safer autonomous vehicles. He summarizes his red line as follows: "The solution is not to build the thing that would kill everyone." He also claims to be taking action to prevent this scenario and does not view it as inevitable.

A Drift Scenario: Adverse Goals, Self-Improvement, and an Uninhabitable Earth

According to Nate Soares, a sufficiently capable AI could pursue goals contrary to those of humans and escape oversight. It could self-improve, build automated infrastructure, and operate without human intervention. Such a system could reshape the planet according to its own objectives, potentially making Earth uninhabitable for the human species. Soares has been working on AI alignment for over 12 years and is co-author of "If Anyone Builds It, Everyone Dies." As president of MIRI, he previously worked as an engineer at Google and then at Microsoft.

Extinction Probabilities and Retirement: The Gap Between Soares, Hinton, Hubinger, and Musk

Nate Soares posits a probability of over 50% for human extinction due to AI within a decade. Geoffrey Hinton mentions 10%, and Evan Hubinger suggests a probability greater than 10% over the same timeframe. Soares explains that he has stopped contributing to his 401(k), a position detailed in an interview conducted by Nicholas Thompson in the series "The Most Interesting Thing in AI." In contrast, Elon Musk states that people may not need to worry about saving for retirement in 10 or 20 years, anticipating that advancements in AI, robotics, and energy will render these savings "irrelevant." He describes an era of abundance with high universal income, cheap goods and services, better health and education, and envisions that money itself could become unimportant if AI and robots surpass humans in nearly all tasks.