LLM in Production: Choosing the Right Optimization for the Bottleneck

Model compression, prefix caching, in-flight batching, PagedAttention, speculative decoding, or pre-fill/decoding separation: each lever addresses a specific symptom. A framework of six questions guides the choice and clarifies the costs and limitations, from inter-token latency to GPU memory constraints.

What Each Lever Costs and What It’s For

Compression techniques such as quantization, sparsity, and distillation lead to a loss of accuracy, and obtaining correct quantized activations is more challenging than quantized weights. Prefix caching fails whenever there is even a slight difference in the shared prefix. Continuous batching has few downsides and primarily depends on its activation. PagedAttention adds a page table layer to the attention computation. Speculative decoding is suitable for latency-sensitive generation for a single user, but it is only cost-effective with a sufficiently high acceptance rate. Pre-fill-decoding disaggregation targets traffic where pre-filling and decoding compete for the same GPU, incurring network costs to move the KV cache and adding routing complexity. Tensor or pipeline parallelism can lead to pipeline bubbles and communication overhead between devices. Offloading the KV cache addresses long or highly concurrent contexts that lack GPU memory, at the cost of latency during retrieval. Finally, offline batch inference maximizes throughput but is unsuitable for cases requiring strict response times.

Recognizing Failure Signals and Initial Fixes

When the success rate of prefix caching remains very low despite identical prompts being sent multiple times, it usually indicates a lack of determinism in serialization or the presence of dynamic content inserted too early in the prompt. To fix this issue, static content should be placed first, and the order of keys in the serialized context should be rearranged. If the batch size remains unchanged while available memory is not saturated, it may mean that PagedAttention is not enabled or that the page size is not correctly configured. Most modern engines already support PagedAttention and enable continuous batching by default, which justifies checking these settings before adding complexity.

When to Reduce the Model Before Everything Else

If the model occupies nearly all the GPU memory, the available space for the KV cache becomes very limited, constraining the batch size regardless of the scheduler used. When the model is too large or inference, with a batch size of a single request, is bottlenecked by computational power, it is advisable to start by applying compression methods to the model. Quantization involves reducing the precision of weights and activations, structured sparsity removes certain weights deemed less important to reduce computations, and distillation involves training a more compact model to replicate the behavior of a larger model. These different methods, which remain largely independent of the software stack used, set the remaining hardware capacity.

Leveraging Prompt Repetition and Wave Arrivals

In chat applications, the complete history is transmitted with each exchange, agent frameworks heavily employ the same system prompt, and RAG pipelines frequently use the same upstream instructions. When a significant portion of the traffic uses an identical prefix, prefix caching proves to be the most effective method, as it avoids recalculating the KV cache for already-seen tokens, with benefits increasing as conversations lengthen. This technique only works if the shared prefix matches exactly character by character, hence the importance of placing stable content at the beginning and variable content at the end of the prompt. If prompts are mostly unique, the variability of response lengths becomes the issue. A support chatbot may produce one sentence while a code assistant may generate four hundred lines, and under naive batching, the entire batch waits for the slowest request. Continuous batching addresses this idle time by immediately removing completed sequences and admitting new requests without waiting for a batch limit, an option generally enabled by default.

KV Cache Memory: Fragmentation and Allocation

When traffic is well grouped but memory limits the batch size, KV cache management may be at fault. Some engines reserve a contiguous block of memory per request sized for the maximum sequence, leaving large portions unused and blocking other requests. PagedAttention addresses this issue by breaking the cache into fixed-size pages and allowing allocations in smaller chunks. Most modern engines support it, and its activation should be checked before complicating the architecture. If memory is not the limiting factor, inter-token decoding latency becomes the next priority.

Inter-Token Latency or GPU Contention: Two Distinct Paths

To accelerate user token streaming, speculative decoding uses a draft model that proposes multiple tokens, verified in a single pass by the main model. When these proposals are correct, several sequential steps are validated together, reducing inter-token latency. However, this mechanism introduces the cost of an additional model and verification, and it is only relevant if the draft model is suitable and the acceptance rate is sufficient. At scale, another concern is contention between pre-filling and decoding on the same GPU: long pre-fills can delay decoding. Disaggregation separates these phases onto dedicated hardware, with pre-fill nodes passing the KV cache to optimized decoding nodes. This separation adds a transfer and routing cost that only makes sense if the traffic justifies it. For smaller volumes or non-interactive processing, offline batch inference is often easier. In the absence of specific conditions, it is preferable to start with continuous batching and prefix caching.

The Six-Step Decision Framework and Its Assumptions

The choice of an optimization is organized around six questions that sequentially test the effective size of the model against the hardware, the existence of a recurring prompt prefix, the presence of simultaneous arrivals with variable outputs, fragmentation or over-provisioning of the KV cache, the priority given to inter-token decoding latency, and then contention between pre-filling and decoding on the same GPU. Each technique relies on an explicit assumption: compression if the model is too large for the hardware or the targeted latency, caching if prompts share a prefix, continuous batching if lengths vary, PagedAttention if KV memory is blocking, speculative decoding if the bottleneck is decoding, disaggregation if the phases compete, parallelism if the model does not fit on a single device, KV offloading if GPU memory is lacking, and offline batches if interactivity is not required.