LLM Optimization: Inference Caching Explained

Introduction to Inference Caching in LLMs

In the realm of large language models (LLMs), inference caching is a crucial technique for optimizing performance. When a user sends a prompt to an LLM, the model performs a substantial amount of computation to process the input and generate each output token. This process is not only time-consuming but also financially resource-intensive. Inference caching allows for storing the results of these computations at various levels of granularity and reusing them during similar or identical requests, thereby reducing both cost and processing time.

The Three Types of Caching

There are three main types of inference caching, each operating at a different level of the stack:

• KV caching: This type of cache stores internal attention states, specifically key-value pairs, computed during a single inference request. This allows the model to avoid recalculating these states at each decoding step. KV caching is automatic and always active.

• Prefix caching: Also known as prompt caching or context caching, it extends KV caching across multiple requests. When a prefix of tokens is shared among different requests, the KV states for that prefix are stored and reused across all those requests. Prefix caching is the most effective optimization you can add to most production applications.

• Semantic caching: This application-level cache stores complete input/output pairs from LLMs and retrieves them based on semantic similarity. Unlike prefix caching, which operates on the attention states being computed, semantic caching completely bypasses the model call when a sufficiently similar request has already been seen.

These types are not interchangeable alternatives but rather complementary layers. KV caching always works, prefix caching is the most effective optimization for most production applications, and semantic caching is an additional enhancement when the volume of requests and similarity are high enough to justify it.

Understanding KV Caching

KV caching is the foundation upon which everything else is built. To understand it, a quick overview of how transformer attention works during inference is necessary.

The Attention Mechanism and Its Cost

Modern LLMs use the transformer architecture with self-attention. For each token in the input, the model computes three vectors:

• Q (Query) — What is this token looking for?

• K (Key) — What does this token offer to other tokens?

• V (Value) — What information does this token carry?

Attention scores are calculated by comparing each token's query with the keys of all previous tokens, then using these scores to weight the values. This allows the model to understand the context across the entire sequence.

LLMs generate outputs in an autoregressive manner — one token at a time. Without caching, generating token N would require recalculating K and V for all previous N-1 tokens from scratch. For long sequences, this cost accumulates at each decoding step.

How KV Caching Solves This

During a forward pass, once the model has computed the K and V vectors for a token, these values are saved in GPU memory. For each subsequent decoding step, the model consults the stored K and V pairs for existing tokens instead of recalculating them. Only the newly generated token requires fresh computation. Here’s a simple example:

• Without KV caching (generating token 100):

  • Recalculate K, V for tokens 1–99 → then compute token 100

• With KV caching (generating token 100):

  • Load stored K, V for tokens 1–99 → compute only token 100

This is KV caching in its original sense: an optimization within a single request. It is automatic and universal; every LLM inference framework enables it by default. You do not need to configure it. However, understanding it is essential for grasping prefix caching, which extends this mechanism across requests.

Using Prefix Caching to Reuse KV States Across Requests

Prefix caching — also referred to as prompt caching or context caching depending on the provider — takes the concept of KV caching a step further. Instead of caching attention states only within a single request, it caches them across multiple requests — specifically for any shared prefix that these requests have in common.

The Main Idea

Consider a typical production LLM application. You have a long system prompt — instructions, a reference document, and few-shot examples — that is identical for each request. Only the user message at the end changes. Without prefix caching, the model would recalculate the KV states for the entire system prompt at each call. With prefix caching, it calculates them once, stores them, and each subsequent request that shares this prefix goes straight to processing the user message.

The Strict Requirement: Exact Prefix Match

Prefix caching only works when the cached portion of the prompt is identical byte for byte. A single character difference — a space at the end, a changed punctuation mark, or a reformatted date — invalidates the cache and forces a complete recalculation. This has direct implications on how you structure your prompts.

• Place static content first and dynamic content last. System instructions, reference documents, and few-shot examples should precede each prompt. Per-request variables — the user message, a session ID, or the current date — should appear at the end.

• Also, avoid non-deterministic serialization. If you inject a JSON object into your prompt and the order of keys varies between requests, the cache will never be hit, even when the underlying data is identical.

Provider Implementations

Several major API providers expose prefix caching as a prominent feature.

• Anthropic calls it prompt caching. You need to enable it by adding a cache_control parameter to the content blocks you wish to cache.

• OpenAI automatically applies prefix caching for prompts longer than 1024 tokens. The same structural rule applies: the cached portion must be the stable prefix of your prompt.

• Google Gemini refers to it as context caching and charges for the stored cache separately from inference. This makes it most cost-effective for very large and stable contexts that are reused many times across requests.

• Open-source frameworks like vLLM and SGLang support automatic prefix caching for self-hosted models, managed seamlessly by the inference engine without modifications to your application code.

Understanding How Semantic Caching Works

Semantic caching operates at a different level: it stores complete input/output pairs from LLMs and retrieves them based on meaning, rather than exact token matches.

The Practical Difference

The practical difference is significant. Prefix caching makes processing a long shared system prompt less costly for each request. Semantic caching completely avoids the model call when a semantically equivalent request has already been answered, regardless of whether the exact wording matches.

Here’s how semantic caching works in practice:

• A new request arrives. Compute its embedding vector.

• Search a vector store for cached entries whose request embeddings exceed a cosine similarity threshold.

• If a match is found, return the cached response directly without calling the model.

• If no match is found, call the LLM, store the request embedding and response in the cache, and return the result.

In production, you can use vector databases such as Pinecone, Weaviate, or pgvector, and apply an appropriate TTL so that expired cached responses do not persist indefinitely.

When Semantic Caching Justifies the Additional Cost

Semantic caching adds an embedding step and a vector search to each request. This additional cost is only justified when your application has a sufficient volume of requests and repeated questions such that the cache hit rate justifies the added latency and infrastructure. It works best for FAQ-type applications, customer support bots, and systems where users ask the same questions in slightly different ways at high volume.

Choosing the Right Caching Strategy

These three types operate at different levels and solve different problems.

CACHING STRATEGY BY USE CASE

• All applications, always: KV caching (automatic, nothing to configure)

• Long system prompt shared among many users: Prefix caching

• RAG pipeline with large shared reference documents: Prefix caching for the document block

• Agent workflow with broad and stable context: Prefix caching

• High-volume application: Semantic caching to avoid redundant calls and optimize performance.