LLM: Why Batching Changes Everything in Decoding

The bottleneck of large models is not computation but memory during decoding. Three ways to batch requests — fixed batch, queue-shaped batch, and replenishment at each step — lead to very different behaviors on the server and GPU.

When a batch waits for the longest request, utilization drops to 33%

In a static batch of four requests producing 20, 50, 100, and 500 tokens, execution stretches over 500 steps because it waits for the longest one. The 20-token request finishes at step 20, and then its slot remains idle or fills for an additional 480 steps. In total, the useful work accounts for only 670 token steps out of 2,000 possible, which is about 33% utilization. A request arriving at step 1 must wait 499 steps before it can start. This situation corresponds to head-of-line blocking, where a long request delays those that follow.

The real ceiling is memory: decoding, weights, and KV cache

During generation, the GPU is limited more by memory read speed than by computation. Inference alternates between prefill and decode: the prefill processes all tokens of the prompt in parallel and effectively utilizes the computing units, while decoding rereads the weights for each token, which is the main source of waste. For a 7B model in FP16 on an H100, the weights occupy about 14 GB; with a bandwidth of around 3.35 TB/s, reading them takes nearly 4 ms, which limits a single request to ~240 tokens per second. Each token requires about 14 GFLOPs, or ~3.5 TFLOPS in 4 ms, while the H100 offers nearly 1,000 TFLOPS of dense BF16. With a batch size of 1, computation remains idle over 99% of the time. In practice, each sequence maintains a KV cache that grows with each token, and the batch size is therefore limited by memory. Running 32 sequences on the same steps loads the weights once and produces 32 tokens instead of one, and batching increases throughput while reducing the cost per token without adding GPUs.

Forming the batch on the fly: the queue and the Triton example

Dynamic batching changes the way a batch is formed but not its execution. Requests stack in a queue; a batch starts when the maximum size is reached or when the oldest request exceeds a timeout. Thus, in light traffic, no request waits indefinitely for the batch to fill, and in sustained traffic, the GPU remains busy. NVIDIA Triton Inference Server provides an example of this, activated in the model's config.pbtxt file. A configuration illustrates max_batch_size set to 8, preferred sizes of 4 and 8, and a maximum queue timeout of 100 microseconds; it instructs Triton to aim for batches of 4 or 8 and never hold a request beyond 100 microseconds. In contrast, static batching gathers a fixed set of requests, refuses new ones until the batch is completed, and corresponds to the behavior of model.generate(), which becomes problematic when output lengths vary significantly.

Continuous scheduling to keep the GPU nearly full

Continuous batching makes decisions at each decoding step. As soon as a sequence emits its end-of-sequence token, it is returned, and its slot is freed. A pending request occupies this slot at the next step, ensuring that the batch never empties. This approach, also known as iteration-level scheduling, was introduced by Orca and is referred to as in-flight batching in NVIDIA TensorRT-LLM. Specifically, a 20-token request is returned at step 20, and a new one starts at step 21 in the same slot, keeping the GPU nearly full and starting new requests almost immediately.