AI Agency: When Recovery Meets Memory

Introduction

In the field of artificial intelligence, an agent that fails to remember its previous interactions loses much of its utility. Each language model is limited by a fixed context window. This means that once a conversation or a series of documents exceeds this limit, choices must be made: abandon certain information, summarize it, or retrieve it again. Developers designing agents for long-term interactions often encounter this problem. It sometimes happens that the agent asks questions it has already answered, contradicts its previous decisions, or ignores the existence of relevant documents.

To solve this problem, two mechanisms are primarily used: retrieval and memory. Each addresses different aspects of the challenge. Retrieval involves integrating external knowledge that the model has not learned by default, such as documents, code, or database records. Memory, on the other hand, allows the agent to retain what it has learned or accomplished, whether during a session or across multiple sessions, to avoid starting from scratch each time. Confusing these two mechanisms or implementing only one is often the cause of the failure of many agent architectures.

Understanding Why Context Forces a Separation

The context window represents the set of tokens that the model can process simultaneously: this includes the system prompt, conversation history, tool outputs, and any other pre-inserted elements. This window is limited, and each token is considered at each pass, meaning that simply enlarging the window does not solve the problem. Thus, contextual engineering has become a discipline in its own right, focusing on managing this limited resource, regarded as the complete state available to the model at any given moment.

Faced with this constraint, the agent must manage two types of information that it cannot keep permanently in context:

• Information that exists outside the model and the current conversation, such as a knowledge base, code, or policy documents. This is where retrieval comes into play.

• Information that the agent has generated or learned itself, which must persist beyond the current context window, such as a decision made several exchanges ago or a fact concerning a specific user. This is the role of memory.

These two mechanisms are often implemented with similar tools such as embeddings, vector search, and structured stores. The key distinction lies in what they store and where the information comes from. Retrieval searches an external corpus to the agent, while memory stores information derived from the agent's past interactions and actions.

Defining Retrieval in Agentic Systems

Retrieval allows an agent to answer the question: "What does the world know about this that I do not have in my weights or current context?" One of the most common implementations is retrieval-augmented generation (RAG):

• Source documents are broken down into passages small enough to be useful.
• Each passage is converted into an embedding and stored in a vector index.
• When a query is made, the incoming question is embedded in the same way, and the index returns the closest matches.
• These matches are inserted into the prompt alongside the user's question.

This model typically operates on managed data stores with an orchestration layer that connects the retrieval step to the rest of the agent's reasoning. The corpus itself is shared: each user asking questions about the same documentation accesses the same index, which is updated according to its own schedule, independent of any individual conversation.

Defining Memory in Agentic Systems

Memory allows an agent to answer the question: "What have I already learned or done that I need to retain?" It is divided into two distinct layers:

• Short-term memory represents the state of the current session: the conversation so far, as well as anything the agent has noted during the current task. It is low-cost and disappears at the end of the session.

• Long-term memory persists across sessions. It must answer a more complex question than retrieval: not just "What is relevant?" but "What is worth retaining in the first place?"

Some agent memory systems automatically extract useful facts, preferences, and context from conversations to store for later use. At the start of a new session, the agent can query this memory in the same way it would query a retrieval index, but the results are specific to a user, a task, or an agent, rather than a shared document corpus. When designing this layer, teams can explore different strategies and frameworks for agent memory based on what they need to store and retrieve.

A concrete example illustrates this separation. When a customer contacts a support agent regarding a delayed order, the agent first checks its memory for the customer's history. It finds a note from three weeks ago indicating that the customer prefers follow-up via email and that a similar shipping issue was resolved with a partial refund. This is memory, as it comes from the agent's record concerning this specific customer.

Next, the agent needs the current shipping policy, which changed last month. It therefore searches the company's documentation and retrieves the relevant section. This is retrieval, as the information comes from an external source and applies to all customers. Both results are added to the same prompt, but they answer different questions.

Comparing Retrieval and Memory

When comparing retrieval and memory side by side, the differences become more evident:

| Dimension | Retrieval | Memory | |-------------------------------|----------------------------------------------|---------------------------------------------| | Source of information | External corpus not created by the agent | The agent's own past interactions or reasoning | | Scope | Shared among all users and sessions | Specific to a user, task, or session | | What it answers | "What does the world know about this?" | "What have I already learned or done?" | | Freshness mechanism | Reindexing the corpus on a schedule or upon writing | Consolidating, updating, or expiring stored facts | | Typical failure mode | Outdated or missing documents in the index | Contradictory or outdated facts concerning a user | | Cost model | Focused on reading; a query-based search | Reading and writing; extraction after each interaction |

The failure modes listed in the table above explain why an agent built with only one of the two tends to fail predictably, and why most functional systems end up needing both.

Combining Retrieval and Memory into an Effective System

An agent with retrieval capabilities but lacking memory will derive the same conclusions in each session and cannot personalize anything. Conversely, an agent with memory but without retrieval knows its own history but has no way to refer to external information; it cannot answer questions about a policy that changed after its training data ended. Achieving the right combination relies on several factors:

• Filtering is more important than window size. Adding more retrieved documents or memory entries does not necessarily improve responses. Beyond a certain point, additional context can make responses less accurate, as the model must process and evaluate each additional token. Small, targeted searches are often more effective than a single broad search and can keep retrieval efficient in terms of tokens.

• Obsolescence works differently for retrieval and memory. A retrieval index becomes obsolete when the underlying documents change without being reindexed. Memory becomes obsolete when information about a user changes—such as a preference or plan—but the stored fact is not updated or removed.

• Memory adds a writing cost. Retrieval typically involves searching for information when the agent needs it. Memory also requires deciding which information is worth retaining after an interaction, which can add calls to the model and processing time. This extraction is often managed asynchronously to avoid slowing down the agent's response.

• Both sources must be merged carefully. Retrieval and memory can return overlapping or conflicting information. The agent must have clear rules for deciding how much weight to give each source and how to use both in the same context.

The design work for retrieval and memory comes down to deciding what belongs in each category, how much to prune both, and how they combine into a single prompt without giving the model tokens it does not need.

Summary

Retrieval and memory solve different problems in long-term agent systems. Retrieval integrates external information that the agent needs at a given moment, such as documentation, policies, code, or database records. Memory retains information from previous interactions, such as decisions, preferences, and user-specific context. The distinction is important because the two systems have different scopes, freshness concerns, and failure modes. Retrieval depends on updating external sources, while memory relies on the decision of what is worth storing and when the stored information is no longer valid.

The most effective agent architectures use both. They filter what enters the context, keep information reasonably fresh, and merge retrieved knowledge with relevant memory instead of treating one or the other as a complete record of everything the agent needs to know.

The goal is therefore to provide the agent with the context it needs.