Mustafa Suleyman: AI Continues to Rise Despite Doubts

The Exponential Evolution of AI According to Mustafa Suleyman

In a world where we have evolved with a linear understanding of progress, artificial intelligence (AI) defies this intuition. Mustafa Suleyman, co-founder of DeepMind, emphasizes that since its inception in 2010, the amount of training data for AI models has exploded dramatically. Early systems required about 10¹⁴ flops, while current models exceed 10²⁶ flops. This exponential growth is at the heart of AI's evolution.

Skeptics, however, continue to predict obstacles to this progression. They often point to the slowdown of Moore's Law, the lack of data, or energy limitations. Yet, Suleyman insists that the combination of driving forces behind this revolution makes the exponential trend predictable. To understand this dynamic, it is crucial to examine the complex and ever-evolving realities that lie behind the headlines.

Technological Advances Driving AI

Training AI can be likened to a room full of people using calculators. Historically, increasing computing power simply meant adding more people with calculators. However, this approach was inefficient, as many remained idle waiting for the necessary data. Today, the revolution goes far beyond just adding faster calculators. It involves ensuring that all these calculators work together continuously and efficiently. Three major advances enable this transformation.

First, the basic calculators, that is, the chips, have become much faster. Nvidia chips, for example, have seen their raw performance increase by more than seven times in six years, rising from 312 teraflops in 2020 to 2,250 teraflops today. The Maia 200 chip, launched in January, offers 30% better performance per dollar than any other hardware in their fleet.

Second, high-bandwidth memory technology, or HBM, has revolutionized the speed at which data is processed. By stacking chips vertically, the latest generation, HBM3, triples the bandwidth of its predecessor, allowing processors to remain constantly busy.

Third, connections between computing units have been enhanced through technologies like NVLink and InfiniBand, transforming hundreds of thousands of GPUs into giant supercomputers. These advances enable the processing of data volumes unimaginable just a few years ago.

A Computing Power Explosion

These combined innovations have led to a dramatic increase in computing power. In 2020, it took 167 minutes to train a language model on eight GPUs. Today, it takes less than four minutes with modern hardware. While Moore's Law would have predicted a 5x improvement over this period, we have observed a 50x improvement. The largest current clusters use more than 100,000 GPUs, each significantly more powerful than those of the previous generation.

Simultaneously, software advancements have also played a crucial role. According to Epoch AI, the computing power required to achieve a given level of performance halves approximately every eight months, much faster than the traditional doubling of 18 to 24 months predicted by Moore's Law. The service costs of some recent models have dropped by up to 900 times on an annual basis, making AI much more affordable.

The Future of AI: Almost Human Agents

The prospects for the near future are equally impressive. Leading labs in the field are increasing their capacity by nearly 4x per year. Since 2020, the computing power required to train cutting-edge models has grown by 5x each year. It is projected that the global computing power relevant to AI will reach 100 million H100 equivalents by 2027, representing a tenfold increase in three years. By the end of 2028, we could see a 1,000x increase in effective computing power.

This surge in power could enable the development of AI agents that are almost at human level. These semi-autonomous systems would be capable of performing complex tasks over extended periods, such as writing code, managing projects, making calls, or negotiating contracts. The implications of this transition extend far beyond mere technology, potentially transforming all sectors based on cognitive work.

The Energy Challenge of AI

However, this rapid expansion of AI poses a major challenge: energy. A single AI rack the size of a refrigerator consumes 120 kilowatts, equivalent to 100 households. But this growing energy demand coincides with another exponential trend: the declining costs of renewable energy. Solar costs have fallen by nearly a factor of 100 over 50 years, and battery prices have decreased by 97% over 30 years.

A path toward clean power is beginning to take shape. Massive investments are underway to build $100 billion clusters with energy needs of 10 gigawatts. These warehouse-sized supercomputers are no longer science fiction. The groundwork is being laid for these projects, both in the United States and internationally. At Microsoft AI, the superintelligence lab is preparing for a world of cognitive abundance.

Conclusion: A Revolution in Progress

Skeptics, accustomed to a linear world, will continue to predict diminishing returns. Yet, the explosion of computing power is the technological story of our time. And according to Mustafa Suleyman, this is just the beginning.