Richard Sutton reignites the debate on scaling LLMs

Richard Sutton asserts that computational power alone is not sufficient and calls for new and continuous learning methods. While laboratories are emerging to explore alternative algorithms, Sam Altman and Dario Amodei advocate for the continued scaling of LLMs. Researchers like Jerry Tworek and Jayden Teoh warn against a reductive interpretation of the "bitter lesson."

Leaders Advocate for Continued LLM Scaling

Sam Altman, CEO of OpenAI, believes that the large-scale development of language models has not yet reached its ceiling and criticizes detractors for hindering a new generation of researchers by being overly assertive about the limits of scaling. He cites an OpenAI model that recently succeeded in disproving a long-standing mathematical conjecture as an example. Altman finds world models useful, particularly in robotics, but asserts that the data clearly supports the continuation of LLM scaling. Dario Amodei, CEO of Anthropic, shares this perspective and describes the progress of scaling as a never-ending rainbow.

A Wave of 45 New Labs Exploring Alternative Learning Paths

Forty-five new AI laboratories are attempting to explore the boundaries of massive scaling. Among these is Core Automation, led by Jerry Tworek, which focuses on learning algorithms that go beyond pre-training and reinforcement learning. Richard Sutton is also involved in this dynamic through Oak Lab. These founders share the belief that a much more capable AI is possible while expressing doubts about the currently dominant method.

What Richard Sutton Demands from AI Systems

Richard Sutton claims that mere computational power is insufficient to achieve intelligence and calls for more ingenious methods, arguing that the science of AI is far from complete. He has criticized the limitations of simply scaling language models and labeled synthetic data as a major mistake, due to a real world he deems infinitely complex, where any simulation would only be a microscopic representation. Sutton views human expertise as a bottleneck in scaling and reminds us, as he wrote back in 2019, that general methods that progress with computational power eventually surpass hand-crafted approaches. Today, he advocates for agents equipped with world models, capable of continuous learning without forgetting previous knowledge, noting that current language models do not continue to learn after their training and that online texts reflect only a tiny fraction of the world's complexity.

A Community Divided on the Interpretation of the "Bitter Lesson"

Jerry Tworek believes that the Bitter Lesson remains relevant, but notes that as the computational power dedicated to training increases, the selection of algorithms becomes crucial. He observes that some algorithms continue to progress while others hit roadblocks, and that many aspects of scaling remain to be explored. In his view, the temporary adoption of transformers and a misinterpretation of this lesson have marked a generation of scientists. Jayden Teoh reports having seen this interpretation hinder academic work, with promising ideas being dismissed as too out of touch, without it being Richard Sutton's fault. He emphasizes that these ideas, although later replaced by more general approaches, often enabled advancements at the time they were proposed, and calls for them not to be outright rejected, reminding us that an idea does not need to survive scaling to contribute to the progress of AI.