LLMs on Limited Hardware: 7 Levers and Their Quantified Trade-offs

Training a multi-billion parameter model without an H100 cluster requires balancing memory, bandwidth, and computation. From QLoRA to RingAttention, seven approaches enable operation on RTX 4090, A10G, or L40S. Here’s what they offer and where they encounter issues — supported by figures and conditions.

I/O and Drivers: What Causes GPU Usage to Drop Below 30%

Offloading via PCIe Gen4/Gen5 consumption introduces I/O bottlenecks. When host-to-device transfers do not complete in time, streaming multiprocessors wait, and computation usage can drop below 30%. PCIe bandwidth can also starve loading processes sending batches from NVMe SSDs. On the optimized kernels side, compiling FlashAttention in non-standard environments or drivers can fail due to ABI incompatibility, silently reverting to slower non-fused PyTorch kernels, or degrading precision if sequence lengths are misaligned without appropriate masks. Despite these risks, using hardware-optimized kernels is essential to maximize SM occupancy and eliminate memory bandwidth bottlenecks.

Dynamic Memory: Recomputations and Tiling Instead of Storing

Activation checkpointing eliminates memory-intensive tensors, particularly GeLU/SwiGLU, layer normalizations, and dropout masks, then reconstructs them during backpropagation from transformer block boundaries. This strategy adds about 30% FLOPs per step and, without fine management of allocations, can fragment CUDA memory up to OOM errors. To limit memory round-trips, FlashAttention-2 splits Q, K, V into blocks that fit in L1/SRAM cache and computes softmax inline, avoiding writing the N×N attention matrix in HBM. Fused kernels combine LayerNorm, biases, and activations into a single CUDA launch.

Static Memory: Quantifying or Projecting Optimization States

To reduce the footprint of weights and states, QLoRA freezes weights to 4 bits (NF4), applies double quantization that saves 0.37 bits per parameter, dequantizes to BF16 on the way in, adds a low-rank update ΔW=B·A, and immediately removes weights from the cache. DoRA separates the magnitude and direction of updates. The trade-off is a 20% to 35% drop in throughput compared to 16 bits and the need to revert to 16 bits to merge adapters, which prevents direct deployment in 4 bits without composite precision loss. This approach targets fine-tuning of 7 to 70 billion parameters on 24 GB. An alternative without layer freezing, GaLore projects gradients into a low-rank subspace and only tracks momentum and variance for the projected matrices, with periodic updates to amortize the SVD. Calculation pauses and sensitivity to hyperparameters T and r can lead to divergences, but the approach aims for full pre-training or domain adaptation under tight memory constraints.

Sharding and Offload: Extending Beyond Aggregated VRAM

With ZeRO-Stage 3 or FSDP in full sharding mode, the distribution of optimizer states, gradients, and parameters occurs between GPUs and the host's main memory. Each GPU retains only a fraction 1/N of the state in inactive memory; the weights of a layer are reconstructed using an All-Gather just before computation and then released before moving to the next layer. In host offload configuration, fragments are stored in pinned CPU RAM and transferred asynchronously via CUDA streams. This scheme allows, for example, training a 30 billion parameter model on four 24 GB GPUs when the total VRAM is insufficient.

Mixed Precision FP8: Doubled Throughput, Narrow Margins

Running contractions in FP8 can halve memory bandwidth and activation size compared to 16 bits. Weights and activations use E4M3 (1/4/3 bits) and gradients E5M2 (1/5/2 bits). Dynamic scaling factors per tensor or tile prevent under- and overflows before conversion to FP8 Tensor Cores. The narrow dynamic range exposes the risk of gradient vanishing and divergence if deferred scaling or channel quantization is not rigorous. Acceleration is limited to modern architectures, such as RTX 4090, L40S, and H100, where throughput can be doubled and activation VRAM reduced by half.

Ultra-Long Sequences: Segmenting and Circulating Blocks

To exceed the capacity of a single GPU with contexts of 64k+ tokens, RingAttention segments the sequence across K devices and circulates Query, Key, and Value blocks in a ring topology during attention computation. Each device calculates attention between its local block and those received from others, allowing for sequences much longer than what a single GPU's memory would permit over commodity interconnections.

Hardware Context: Why the Naive Approach Fails

At scale, full pre-training requires H100 clusters connected at 3.2 Tbps. In practice, many teams have setups with two or four RTX 4090, A10G, or L40S GPUs, constrained by PCIe and 24 to 48 GB of VRAM. Naive training in 16 bits with AdamW and preserved autograd graph fails: 14 GB for the weights of a 7B in FP16/BF16, about 56 GB for AdamW states at 8 bytes per parameter, plus 14 GB for gradients and activations that swell with context length. Hence, the need to distinguish between static and dynamic memory and to qualify the bottlenecks between Tensor Core power and VRAM bandwidth.