Inactive GPUs and Queues: Allocation, Memory, Placement

Pods may be waiting while the graphs show a GPU almost idle. The reason is not necessarily a hardware shortage, but often exclusive allocation, occupied VRAM, placement constraints, or a pipeline that is blocked elsewhere. Cast AI reports an average utilization of 5% on clusters analyzed in 2025-2026, but this does not mean that 95% of the capacity is recoverable without fine diagnostics.

When the GPU is Not the Bottleneck: CPU, I/O, and Placement Blockages

Low computational activity on a GPU does not always mean it is available for other workloads. The CPU, PCIe transfers, data loading, or the network can leave the GPU unoccupied while the workload waits elsewhere. For example, an LLM inference server waiting for a slow tokenization step or a fetch from a distant KV cache will show a low percentage of active streaming multiprocessors, without freeing the device for other pods. These situations are difficult to diagnose from GPU metrics alone. Sustained low computational activity on an active load generally indicates a bottleneck outside the GPU; adding more cards does not provide a solution. Fixes focus on CPU resources, batching adjustments, storage latency, or redesigning the data pipeline. Tools like nvidia-smi dmon or DCGM metrics allow for observing computational usage over time: alternations between 0% and short bursts suggest a CPU limit, while a constant weakness under load points to I/O or network issues. These profiles differ from allocation shortages and require different remedies.

Moreover, scheduling failures can stem from placement. Node affinities targeting a specific GPU model, topology distribution constraints, or intolerable taints can exclude otherwise available hardware. A request for nvidia.com/gpu: 1 accompanied by a selector demanding an A100 will not be scheduled on H100s, even if they are idle. Requesting more replicas in time-slicing than a node can provide, without distribution, can also cause deployment failures despite total capacity. The command kubectl describe pod <pending-pod> and its events (MatchNodeSelector, InsufficientResource, TaintToleration) help identify these cases, which are resolved through scheduler configuration, not by adding hardware.

Exclusive Allocation by Default: The First Lock to Lift

The Kubernetes scheduler treats GPUs as whole devices exposed via nvidia.com/gpu, without considering the percentage of active silicon. By default, it assigns entire GPUs to pods: a pod requesting nvidia.com/gpu: 1 gains exclusive access to the device for its entire lifecycle, preventing any other pod from accessing it, regardless of actual usage. This behavior explains most queues next to underutilized GPUs in inference clusters: low-frequency served models monopolize each card, and an incoming load sees nvidia.com/gpu: 0 available, even if the aggregated activity of the node remains below 10%. The device plugin focuses on allocation, and the auto-scaler can add nodes, but recovering capacity on an existing node does not occur. The issue is one of scheduling: the way GPUs are presented to the scheduler needs to change, which sharing mechanisms allow.

Saturated VRAM: Heavy Models, KV Cache, and Quantization

Low computational utilization does not mean that GPU memory is available. A 70 billion parameter model in FP16 occupies about 140 GB of VRAM for its weights, to which the KV cache adds 15 to 20% at 4K context and can exceed the weights at 128K. It then requires two A100 80 GB or more to load it. A 13B FP16 model fits on an A100 80 GB with about 26 GB but occupies VRAM continuously; activity may only indicate 8%, and the device remains unavailable for other loads. Before configuring sharing, quantization needs to be assessed: a 7B in INT4 can drop to about 4 GB compared to around 14 GB in FP16; INT8 often reduces needs by about half compared to FP16 with acceptable quality for inference.

The choice of sharing method depends on memory. Time-slicing multiplexes computation but does not partition VRAM: all replicas share the same address space, and two loads that together exceed memory cannot coexist safely. Diagnosing memory constraints first avoids unstable deployments. A preliminary check with nvidia-smi is essential: if free memory approaches zero, the limit is VRAM rather than scheduling. MIG provides isolated memory partitions at the cost of compatible hardware and appropriate scheduling.

What GPU Utilization Metrics Really Indicate

The indicators grouped under "GPU utilization" cover at least four distinct measures. Most dashboards display computational activity, which is the share of active streaming multiprocessors over a given window; it informs about silicon occupancy, not allocation status. A model loaded in VRAM occupies that memory continuously, and Kubernetes will not schedule anything else on an allocated device, even if computational activity remains, for example, at 5% with a service responding rarely. Other metrics complete the picture: VRAM memory occupancy indicates how much memory is consumed without specifying computational activity or the ability of pending loads to hold; scheduling wait time measures the duration of waiting without clarifying the source of the blockage; throughput and latency reflect perceived performance (up to the first token or end-to-end) without informing about resource utilization or allocation efficiency. Confusing these scales leads to erroneous diagnostics.

Quantifying Without Overinterpreting and Diagnosing in Order

Cast AI indicates, in a report covering tens of thousands of clusters between January 2025 and April 2026, an average GPU utilization of 5% before optimization. One cluster maintained 49% on 136 H200, a gap attributed mainly to technique rather than hardware. These elements describe real overprovisioning and underutilization at the fleet level, without meaning that 95% of the capacity is immediately mobilizable. Each cluster and each GPU requires its own diagnosis.

On the remedy side, several sharing methods coexist with different trade-offs in memory isolation, hardware requirements, observability, and cloud support; none is universal. Some platforms, like Cast AI, claim to support time-slicing, MIG, and MPS with automatic bin-packing that does not require modifying manifests. Regardless of the tooling, the diagnostic approach must follow a precise sequence: check allocation status, then memory occupancy, followed by placement, and finally application bottlenecks. It is in this order that the causes unravel and technical fixes take effect.