Context Windows: The False Memory of AI Agents Revealed

Context Windows: A Deceptive Memory for AI Agents

Context windows in artificial intelligence models, particularly those relying on agents and language models, are often misunderstood. They are comparable to a stateless notebook, where each interaction starts from scratch. Unlike persistent memory, these windows do not retain any information between API calls. Thus, even with a conversation history exceeding 200,000 tokens, the agent does not remember previous interactions. Each new request requires a complete re-reading of the history, which can lead to complications.

AI models, when faced with long prompts, often behave like a student who only pays attention to the beginning and end of a text, neglecting the intermediate information. This can create a snowball effect, where the agent must constantly re-read the entire history, including details that are often outdated. In terms of performance, this can cause slowdowns, a "brain freeze" effect, where the model takes time to generate a response to a lengthy text.

To illustrate this process, consider a typical API call. The model, retaining no memory between calls, must receive all previous interactions to respond to a new question:

model.generate(
    messages=[
        {"role": "user", "content": "Step 1: Let's call this variable `session_id`."},
        {"role": "assistant", "content": "Understood, I will use `session_id` from now on."},
        # ... each intermediate input must be returned, every time ...
        {"role": "user", "content": "Step 47: What variable name did we agree on in step 1?"}
    ]
)

Retrieval: A Tool for Contextual Memory

Retrieval-augmented generation (RAG) systems act like a well-organized shelf, allowing for the retrieval of relevant data at the right moment, in a "Just-In-Time" manner. These systems identify the most pertinent documents for a given question and integrate them into the context window. However, vector similarity, used to determine relevance, does not always guarantee semantic truth.

For example, if a user requests to move a meeting to Friday and then mentions that Alice is sick on Thursday, a vector search engine might retrieve both pieces of information, even if they contradict each other. An effective agent must be able to discern which information is the most current and relevant. A naive RAG pipeline simply concatenates the retrieved information, leaving the model to guess which instruction is valid. A more sophisticated model resolves these contradictions before generating the response, prioritizing the most recent information:

retrieved_chunks = [
    {"text": "Move the meeting to Friday", "timestamp": "2025-01-10T09:00:00"},
    {"text": "Cancel Thursday, Alice is sick", "timestamp": "2025-01-12T14:30:00"}
]
# Reconcile contradictory chunks before they reach the prompt
latest_relevant = max(retrieved_chunks, key=lambda chunk: chunk["timestamp"])

This approach allows the agent to provide accurate and up-to-date responses.

Compression: Optimizing Space in Context Windows

Compression in the context of AI agents is similar to ZIP file compression. It involves reducing the size of data while preserving essential information. This frees up space in the context window for other tasks. Techniques such as removing stop words or using specific compression models, like LLMLingua, are employed for this optimization.

For example, a large API response of 15,000 tokens can be compressed to 5,000 tokens, thus leaving more room for processing the main data:

raw_payload = json.dumps(large_api_response)  # about 15,000 tokens
compressed_payload = compress_with_llmlingua(
    raw_payload,
    target_token_count=5000
)
[prompt](/glossaire/prompt) = f"Data: {compressed_payload}\n\nRespond to the user's question."

This method ensures that essential information is retained while reducing its footprint in the context window.

Synthesis: An Abstraction of Data

Synthesis differs from compression in that it replaces the original data with an abstraction, an irreversible process, a one-way journey. It requires bifurcated storage, where raw transcriptions are saved in economical storage, such as S3 buckets, while the synthesized summary is used in the active prompt.

This bifurcated storage model results in double writing: one for cold storage and one for the active prompt:

def summarize_turn(raw_transcript, session_id, turn_id):
    # 1. Persist the raw, unabridged transcription in cold storage
    s3_client.put_object(
        Bucket="agent-transcripts",
        Key=f"{session_id}/turn_{turn_id}.json",
        Body=raw_transcript
    )
    # 2. Generate a compact summary for the active prompt
    summary = summarizer_model.generate(raw_transcript)
    # 3. Only the summary reintegrates into the context window
    return summary

If the original detail is required later, it can be retrieved from cold storage. Synthesis, unlike compression, does not require reconstruction from the active prompt.

Towards True Memory Persistence

Memory persistence in AI agents is often misinterpreted. For an agent to have true memory, it must act like a database manager, rather than being a database itself. For example, if a user mentions that their dog is named Goofy, but it could also be called Pluto, the agent must be able to update this information in a structured manner:

{
    "tool": "update_entity_graph",
    "params": {
        "subject": "User_Dog",
        "attribute": "Name",
        "value": "Goofy",
        "notes": "Considering Pluto as a potential new name"
    }
}

By adopting this approach, AI agents can effectively manage information and provide a more coherent and reliable user experience.