GPU and AI: Resource Fragmentation Hinders Innovation

The Illusion of GPU Utilization

In the field of artificial intelligence, the use of GPUs is often seen as a performance indicator. A high GPU utilization rate gives the impression of a healthy infrastructure. However, this appearance can be misleading. Utilization averages often hide a fragmentation of resources, where the remaining capacities are not effectively usable.

Imagine a city where some roads are empty while others are congested. Similarly, in an AI cluster, resources may be available but poorly distributed, preventing the efficient handling of new workloads.

A cluster may show high GPU occupancy, active workloads, and high memory usage while having mediocre effective capacity. The problem is often not that resources are exhausted, but that the remaining resources only present themselves in unusable combinations.

Resource Fragmentation: An Invisible Problem

Let's take the example of three nodes after various workloads:

• Node A: Available GPU, HBM nearly full, storage and I/O bandwidth available.
• Node B: Available GPU and HBM, saturated storage, available I/O bandwidth.
• Node C: Limited GPU, available HBM and storage, saturated I/O bandwidth.

A new workload requiring healthy storage bandwidth and I/O capacity finds no suitable place. This is resource fragmentation: the cluster is not empty, but the resources are fragmented into inefficient combinations.

This situation becomes particularly dangerous in GenAI systems, as modern workloads heavily rely on retrieval pipelines, KV cache growth, storage throughput, and overall data path efficiency. A cluster may appear healthy from a distance while silently degrading beneath the surface.

A Cluster Can Have Available GPUs and Rising Queues

This is the most counterintuitive part of the entire problem. A cluster can simultaneously have rising queue times, worsening latency, and declining throughput. At first glance, this seems contradictory. It is not.

If the only "free" GPUs are on nodes whose storage bandwidth is already overloaded, the SSD queue depth skyrockets, or the CPU I/O is consumed by background jobs, then these GPUs are not actually available for the next useful workload.

A greedy scheduler may still place tasks there. These tasks then increase contention, prolong queue times, and leave behind even worse fragmentation. This creates a vicious cycle: more fragmentation → more bottlenecks → longer execution times → more fragmentation. From an operational standpoint, the system is slowly choking.

Why GenAI Has Changed the Bottleneck Landscape

Traditional schedulers were designed for environments where CPU, memory, GPU, and network dominated placement decisions. Modern GenAI systems have changed the nature of the pressure on infrastructure:

• Heavy retrieval pipelines can saturate SSD bandwidth.
• Inference jobs accumulate KV cache over time.
• Loading checkpoints can hit object storage.
• Multimodal workloads create bursty data movements.
• Background maintenance tasks silently steal CPU cycles that would otherwise power the GPUs.
• Node-level storage degradation can reduce effective throughput long before a node technically fails.

This creates a new category of infrastructure pathology: the GPU is no longer the sole bottleneck. The path to the GPU matters just as much. This changes what "healthy utilization" truly means.

A Starving GPU Is Still an Expensive GPU

This is where the economic stakes become serious. Modern AI infrastructure is costly. Public pricing for access to NVIDIA H100 in 2026 generally ranges from a few dollars per hour of GPU to over $10/hour, depending on the provider and engagement model.

At the scale of a large fleet, a cluster of 1,000 H100 GPUs operating at an average cost of about $3/GPU-hour costs approximately:

• $26 million per year, before networking, storage, orchestration, and maintenance costs.

Now imagine that fragmentation and I/O bottlenecks silently waste just 10% of the productive time of the GPUs. This amounts to about:

• $2.6 million per year in inefficient infrastructure spending. Not because the GPUs have disappeared, but because the system has failed to utilize them effectively.

Conscious Residual Planning

Most schedulers pose a deceptively simple question:

• "Can this workload fit on this node?"

Conscious residual planning asks a more important question:

• "What kind of residual cluster does this assignment create?"

This idea is at the heart of Residual-Aware Geometric Planning (RAGP). Instead of reducing a node to a few scalar counters, RAGP treats residual capacity as a multidimensional form.

At a high level:

• non-viable nodes are eliminated,
• the scheduler simulates the remaining resources after placement,
• it prioritizes placements where the vectors of residual resources remain useful for future workloads.

This last step is crucial. Two placement decisions may both seem correct immediately while creating completely different future cluster states. One preserves healthy residual capacity. The other leaves resources in unusable fragments. Traditional utilization metrics often cannot distinguish between these outcomes.

A Concrete Example

Suppose an incoming workload requires:

• moderate storage bandwidth,
• and healthy CPU availability.

After tentatively placing the workload:

| Residual Resource | Node A | Node B | |------------------------|--------|--------| | CPU | 0.2 | 0.1 | | GPU | 0.35 | 0.4 | | HBM | 0.3 | 0.3 | | Storage Bandwidth | 0.25 | 0.02 |

A scalar scheduler might consider both placements acceptable because:

• the GPU remains available,
• memory remains available,
• and the workload technically fits.

Conscious residual planning prefers Node A. Because Node B leaves behind almost unusable storage capacity. This becomes dangerous once the next heavy retrieval or cache growth workload arrives. This is the subtle failure hidden within many modern GenAI clusters: a placement may be locally feasible while being globally harmful.

Extending RAGP to RAGP-I/O

The original formulation of RAGP primarily reasoned about:

• compute capacity.

This worked reasonably well in compute-dominated environments. GenAI workloads have changed the bottleneck landscape. In real systems:

• storage queue depth,
• degraded RAID states,
• retrieval pressure,
• and CPU I/O saturation

can influence throughput just as strongly as GPU utilization itself. A node may appear healthy in terms of compute, memory, and network while silently starving workloads through storage bottlenecks.

RAGP-I/O extends the planning space by explicitly incorporating:

• storage bandwidth,

into feasibility and placement logic. Instead of reasoning through five dimensions, the scheduler reasons through seven: CPU, RAM, GPU SM, HBM, network, storage bandwidth, and CPU I/O.