OpenAI Revolutionizes Efficiency with Prompt Caching

Prompt Caching: An Asset for the OpenAI API

Understanding Prompt Caching

Prompt caching is an advanced feature offered by language model API services, such as the OpenAI API. It allows for the storage and reuse of certain parts of the language model (LLM) input that are frequently repeated. These parts typically include system instructions or recurring queries sent to the model each time an AI application is executed. For caching to be effective, these repeated segments must be placed at the beginning of the prompt, forming a prompt prefix. Additionally, this prefix must exceed a certain length threshold, for example, more than 1,024 tokens for OpenAI.

Prompt caching operates at the token level, within the internal processes of the LLM. The inference process of the LLM is divided into two distinct phases:

• Pre-filling: the language model uses the user's prompt to generate the first token.
• Decoding: the model then recursively generates the output tokens, one by one.

In summary, prompt caching involves storing the computations performed during the pre-filling phase so that the model does not have to recalculate them when the same prefix is used again.

Implementation in the OpenAI API

OpenAI's API introduced prompt caching starting October 1, 2024. Initially, this feature offered a 50% reduction on cached tokens, but this discount can now reach up to 90%. Furthermore, using prompt caching can lead to additional savings on latency, potentially up to 80%.

When a prompt is submitted with caching enabled, the API service attempts to access the cache by directing the prompt to an appropriate machine where the cache is expected to exist. This process is known as cache routing, and to do this, the API typically uses a hash of the first 256 tokens of the prompt.

The OpenAI API also provides the option to explicitly set a prompt_cache_key parameter in the API request to the model. This parameter determines which cache is used, thereby increasing the chances that the prompt is directed to the correct machine and reaches the cache.

Additionally, the API offers two types of caching based on duration, defined by the prompt_cache_retention parameter:

• In-memory caching: this is the default caching type, available for all models. Cached data remains active for 5 to 10 minutes between requests.
• Extended caching: available for certain specific models, it allows cached data to be retained for longer, up to a maximum of 24 hours.

Regarding costs, OpenAI charges the same price per input token, whether cached or not. However, if the cache is accessible, cached tokens benefit from a significant discount, which can reach up to 90%.

Practical Implementation of Prompt Caching

To illustrate how prompt caching works, let's take a simple example in Python using the OpenAI API. Imagine a scenario where a long system prompt (prefix) is reused in multiple requests. Make sure you have your OpenAI API key and have installed the necessary libraries.

The first step is to import the OpenAI library and initialize an instance of the OpenAI client:

from openai import OpenAI
client = OpenAI(api_key="your_api_key_here")

Next, we define our prefix, which is the tokens we want to cache because they will be repeated:

long_prefix = """
You are a highly competent assistant specializing in [machine learning](/glossaire/machine-learning).
Answer questions with detailed and structured explanations, including examples when relevant.
"""

We artificially increase the length to ensure that the 1,024 token threshold is reached, then we set up a timer to measure our latency savings:

start = time.time()
response1 = client.responses.create(
model="gpt-4.1-mini",
input=long_prefix + "What is overfitting in machine learning?"
)
end = time.time()
print("Initial response time:", round(end - start, 2), "seconds")
print(response1.output[0].content[0].text)

For models starting from gpt-4o, prompt caching is enabled by default. Since our 4,616 input tokens far exceed the 1,024 token threshold, we are on the right track. The first request checks if the input is a cache hit (which is not the case here, as this is the first time we are making a request with this prefix), and since it is not, it processes the full input and caches it. The next time we send an input that partially matches the initial tokens of the cached input, we will gain cache access.

Let's verify this by making a second request with the same prefix:

start = time.time()
response2 = client.responses.create(
model="gpt-4.1-mini",
input=long_prefix + "What is regularization?"
)
end = time.time()
print("Secondary response time:", round(end - start, 2), "seconds")
print(response2.output[0].content[0].text)

Indeed! The second request runs much faster (23.31 seconds compared to 15.37 seconds). This is because the model has already performed the calculations for the cached prefix and only needs to process the new part, "What is regularization?". As a result, thanks to prompt caching, we achieve significantly reduced latency and lower costs, since cached tokens benefit from a discount.

Potential Pitfalls of Caching

While enabling prompt caching seems straightforward, several factors can lead to a cache access failure. Here are some points to consider:

• A prefix of fewer than 1,024 tokens will not activate caching. This can be easily resolved by artificially increasing the number of tokens in the prefix.

• It is crucial not to break the prefix. Even when using appropriately sized system instructions and prompts, care must be taken not to add variable content at the beginning of the model input. This will break the cache, regardless of the length of the subsequent prefix. Dynamic data, such as adding a user ID or timestamps at the beginning of the prompt, are typical examples.

• Caching only pertains to the pre-filling phase; decoding is never cached. This means that even if we ask the model to generate responses following a specific pattern, the initial tokens will not be cached.

In some use cases, it may not be wise to use prompt caching, such as for highly dynamic prompts, unique queries, or real-time custom systems.

Prompt caching can significantly enhance the performance of AI applications.