NVIDIA and the Crucial Memory Challenge for the Future of AI

The Crucial Importance of Memory in Modern AI

In the development of artificial intelligence algorithms, attention is often focused on the power of the processors capable of executing them. NVIDIA's graphics processing units (GPUs), for example, are at the heart of today's large language models, and companies are competing to create ever-faster AI accelerators. These new chips promise increased computing power with more cores and the ability to perform more operations per second.

However, it would be misguided to think that the future of AI relies solely on creating more powerful processors. A persistent problem remains: many AI systems are limited not by computational speed, but by the speed of access to the data necessary for those computations. In other words, the future of AI may be more closely tied to memory management than to sheer computing power.

An Analogy to Illustrate the Problem

To better understand this challenge, imagine hiring the fastest and most skilled chef in the world. This chef can prepare dishes at lightning speed. However, if every ingredient is stored in a warehouse several kilometers away, the chef will have to wait for someone to fetch those ingredients before being able to cook.

No matter how skilled the chef is, they will often find themselves waiting for the necessary ingredients to arrive. Modern AI systems face a similar problem. Processors can perform calculations, but they cannot work on data that is not yet available. If the processor is capable of calculating faster than the memory can provide the information, performance is limited by the movement of data rather than by the computation itself.

This phenomenon is known as the memory bottleneck in the field of computing. It is one of the major challenges, albeit often underestimated, of modern AI.

The Scale of Contemporary AI Models

To grasp why memory has become such a crucial issue, it is essential to consider the size of current AI models. Early machine learning models contained thousands, or even millions, of parameters. In contrast, modern models contain billions, or even trillions.

Each parameter is a numerical value that must be stored in memory and accessed regularly during training and inference. Take, for example, a model containing 70 billion parameters. Before even starting a calculation, the system must have space to store all these parameters.

Add to this thousands of users interacting simultaneously with the model, and the hardware must continuously move vast amounts of information between memory and processors. Thus, the challenge lies not only in the ability to perform calculations but in the ability to feed the hardware with data quickly enough.

The Cost of Data Movement

Moving data can prove more costly than processing it, a counterintuitive reality in computing. Over the decades, processor performance has significantly improved, with engineers designing chips capable of performing calculations increasingly quickly. In contrast, improvements in memory systems have progressed at a slower pace.

This growing imbalance has become more evident as AI systems have developed. Modern processors can execute trillions of operations per second but often spend a lot of time waiting for data to arrive.

This data bottleneck manifests in various ways within AI systems. It is visible when moving data between memory and processors, between GPUs, across servers, or between data centers.

As these models continue to grow, the problem of data movement will play a major role in the overall performance of the system.

Understanding Memory Types in AI

So far, the term "memory" has been mentioned multiple times. You may be wondering what type of memory is being referred to. Most people are familiar with RAM, the memory installed in laptops and desktops. However, AI systems use different types of memory for various functions.

• RAM: Random Access Memory stores data used by the CPU. It is relatively large but not particularly fast compared to memory specialized for AI.

• VRAM: Graphics processing units have dedicated memory called Video Random Access Memory (VRAM). It is used during training and inference to store model parameters, training batches, activations, and intermediate calculations. The amount of VRAM available often determines whether a model can fit on a GPU.

• High-Bandwidth Memory (HBM): Modern AI accelerators increasingly rely on High-Bandwidth Memory (HBM). HBM is specifically designed to move large amounts of data extremely quickly. Rather than simply increasing memory capacity, HBM focuses on increasing memory bandwidth, or the rate at which information can be transferred.

While memory size (its capacity) is important, its bandwidth plays an even more significant role. One way to understand these two concepts is to compare them to a highway. Capacity is the number of cars that can exist on the highway, while bandwidth is the number of lanes available.

You can have a massive parking lot, but if all vehicles must exit through a single lane, traffic becomes the limiting factor.

Memory Challenges in Training and Inference

As mentioned earlier, AI systems use different types of memory in different ways. The memory challenge manifests differently during training and inference.

• Training: Training requires storing model parameters, gradients, activations, and optimizer states. As a result, memory needs become enormous, often necessitating the distribution of memory across multiple GPUs.

• Inference: Inference generally requires less memory than training, but it introduces a different challenge. The model must continuously respond to requests while quickly retrieving parameters and generating outputs. For interactive systems like chatbots, latency is a major issue.

The faster memory can provide information, the quicker the model can respond. This is one of the reasons why memory technologies remain essential even after training has concluded.

Perspectives and Potential Solutions

Most discussions about AI performance optimizations focus on larger models and faster processors. Yet, hardware engineers are increasingly recognizing a different reality.

Building smarter AI systems is not just about adding more computing power. It also requires solving the problem of data movement. Fortunately, many researchers are focusing on improving this data movement issue. To do so, they are exploring various approaches:

• Enhanced memory architectures.

• Faster interconnections.

• Memory-efficient algorithms.

• Model compression techniques.

• Computing near memory.

• Optical and photonic communication technologies.

Each of these approaches attempts to answer the same question: how to move large amounts of information efficiently?

The answer could determine the future trajectory of AI.

Modern AI systems depend on both computation and communication. While processors perform calculations, memory systems determine how quickly data can reach them. As models continue to grow in size and complexity, memory capacity and bandwidth become increasingly important factors for overall performance.

The next major breakthrough in AI hardware may not come from a processor with more cores or higher clock speeds. It could come from a better way to move data.

So, which of these approaches (if any) is the solution? Well, for now, we do not know, but we are slowly working towards it.