Accelerating LLM Inference: Key Metrics and Levers

Serving a language model is not just about generating tokens: it involves measurement, planning, and optimization. Between computation, bandwidth, and memory, pre-filling and decoding have different constraints. Quantization, FlashAttention, and KV cache management are among the tools used to reduce latency and increase throughput.

Quantifying, optimizing attention, and better batching for speed gains

Several optimization levers target different limits of inference. Quantization involves representing the model's weights or activations with reduced numerical precision. For a model with 8 billion parameters, the weights occupy about 32 GB in FP32, 16 GB in FP16 or BF16, 8 GB in INT8, and 4 GB in 4 bits, excluding metadata, cache memory, and execution overhead. This reduction can enable execution on smaller hardware and decrease memory bandwidth, at the cost of potential accuracy loss, with gains depending on the hardware and implementation. Regarding attention, traditional implementations move costly intermediate tensors between GPU memory and on-chip storage. FlashAttention improves efficiency by processing attention in blocks with tiling, avoiding the need to materialize the full attention matrix in high-bandwidth memory, and preserving exact attention results to within numerical differences. Finally, traditional fixed batching groups requests but suffers from inefficiencies when durations diverge; modern engines optimize pre-filling and decoding separately to better adapt to these profiles.

Measuring inference: TTFT, TPOT, throughput, and long queues

Before optimizing, it is essential to observe the right indicators. TTFT measures the time until the first token, including queuing, pre-filling, and initial decoding. TPOT quantifies the average interval between tokens after the first. Throughput accounts for the total volume served over time, which can remain high even if users receive tokens more slowly. In addition to averages, p95 and p99 latencies are necessary to identify slow requests. Specific step-by-step measurements are also important: pre-filling time and time per output token.

Memory and concurrency: KV cache becomes central

The key-value cache, reused during decoding to avoid recalculating history, saves computation but increases memory consumption. Its size grows with context length, and serving multiple users simultaneously intensifies this pressure, making its management a key element of efficient service. During decoding, repeated access to weights and the cache encounters memory bandwidth limitations; for small batches, data transfer can take longer than the computation itself. Increased computational capacity does not guarantee proportionally faster generation. Under concurrent load, request scheduling, cache management, and memory allocation become critical, justifying the existence of specialized engines.

Performance varies based on GPU computation and memory management

Performance depends on GPU computation, memory bandwidth, VRAM, context length, and scheduling efficiency. Pre-filling relies on large matrix multiplications and primarily encounters computational limits. Decoding is more often constrained by memory bandwidth. Long prompts intensify attention and increase cache size, a common phenomenon in RAG use cases, document analysis, or coding assistants, where inputs often consist of thousands of tokens.

A request goes through several stages of conversion and processing

A request goes through several stages. The text is first converted into numerical identifiers by the tokenizer, then transformed into embeddings for the transformer. During pre-filling, the prompt traverses the model's layers, potentially in parallel across many positions, and initializes the key-value cache. This phase involves significant matrix operations and intensive computation. Next, the model generates output tokens sequentially: at each step, it must choose a token without access to future context, recalculating only the attention information for the new token and relying on the existing cache for history.

What inference does — and does not do

Inference utilizes a trained model, whose billions of learned parameters remain fixed, to predict the next token based on the input and the context already generated. Generation progresses token by token according to a decoding strategy, without backpropagation or gradient updates. Without a cache, past representations would need to be recalculated at each step. In attention, each token is projected into Q, K, V, the basis of the mechanism reused during decoding. The main challenges are response speed, GPU memory, throughput, and operational costs. Pre-filling is particularly crucial for long documents and RAG, while decoding is heavily weighted for chatbots and code generation.