Optimizing AI Memory: The Essential Decision Tree

Choosing the Right Memory Strategy for an AI Agent

In the field of artificial intelligence, determining the optimal memory strategy for an agent is a crucial task. This article details how a decision tree can be used to select the best approach by analyzing each type of information separately.

The Different Types of Agent Memory

AI agents rely on four types of memory: working memory, semantic memory, episodic memory, and procedural memory. Each of these memories has specific characteristics regarding the information they contain.

• Working Memory: It manages information relevant to the present moment, based on an active conversation and a limited token budget. It allows for summarizing or shortening previous exchanges without losing essential elements for the agent.

• Semantic Memory: This memory stores stable and reusable information, such as a user's name, role, or preferred language. It also includes domain knowledge like business rules or product specifications.

• Episodic Memory: It retains the history of past events, such as decisions made, complaints recorded, or transactions completed, which can influence future interactions.

• Procedural Memory: It is designed to enhance the agent's efficiency by repeating similar tasks, allowing the agent to become faster or more reliable with each attempt.

These different layers of memory address varied information needs, which is why most production agents use multiple types of memory simultaneously.

The Decision Tree for Memory Strategy

To choose the right memory strategy, a five-question decision tree is used. This tree helps determine the specific needs of a given category of information. Each category, such as the "current ticket" of a support agent or the "complaint history," may require a different approach.

Question 1: Does the information need to persist beyond the current turn?

• Standalone, no continuity needed: This pertains to unique classification requests or intermediate outputs from a tool to answer a current question.

• Must be retained, memory required: For example, a problem resolved by a support agent or the status of an ongoing coding project.

If the information is standalone, no memory is needed. If it must be retained, move on to Question 2.

Question 2: Does it need to survive beyond a single session?

• Only within the session: This includes what has been requested, the tools used, or the problems solved. A conversation buffer is sufficient, maintained by a summary or reduction. Session-based memory management in the OpenAI Agents SDK handles this directly.

• Beyond the session: Returning customer preferences or the status of an ongoing project require memory beyond the session.

If only continuity within the session is necessary, working memory is sufficient. Otherwise, proceed to Question 3.

⚠️ A common design error is mismatching the information with its lifespan, either treating a session-limited state as permanent or building a persistent memory infrastructure for information that only needs to exist during a conversation.

Question 3: Is it a stable fact or an evolving event?

This question is crucial to avoid mixing different types of information in the same storage.

• Stable facts (semantic memory): Includes items like a name, subscription level, or default shipping address.

• Evolving events (episodic memory): Includes past complaints, project decisions, or behavior patterns.

Stable facts are stored in a persistent knowledge store, while evolving events are recorded in a log. Some frameworks directly integrate the temporal dimension into their storage layer. For example, Zep models facts on a knowledge graph where each fact carries a validity window, so a replaced fact is invalidated rather than silently contradicting the more recent fact.

Question 4: How will this memory be retrieved?

The retrieval method depends on the size and structure of the memory store.

• Small and limited store: Reading the entire store at the beginning of a session is possible. Anthropic's memory tool operates this way because the store remains small enough for a complete read to be cost-effective.

• Large and searchable store: Use semantic or hybrid search to retrieve relevant entries. Google’s Memory Bank is designed for this scale, and memory frameworks like Mem0 offer a comparable provider-independent approach that can be used with frameworks like LangGraph or CrewAI.

It is common for an agent to need both retrieval models: a full-read profile for a small semantic store alongside similarity search on a larger episodic log or semantic knowledge base.

Once retrieval matches the actual size and structure of each store, move on to Question 5.

Question 5: Does the agent need to learn reusable procedures?

Procedural memory comes into play here, complementing semantic and episodic memories.

• Recurring tasks: These should be stored in procedural memory to enhance the agent's efficiency.

• Unique tasks: The previously chosen semantic or episodic memory is sufficient.

The memory module of an agent, combined with its planning and tool layers, feeds the learned context to develop future plans. The key is to decide what is written, with raw logs capturing episodic memory and procedural memory storing learned routines.

Common Pitfalls and How to Correct Them

When implementing an agent's memory, several pitfalls can arise. One of the most common is the misalignment of the information's lifespan with its memory, which can lead to inefficiencies. For example, treating a session-limited state as permanent can waste resources, while creating a persistent memory infrastructure for ephemeral information is also ineffective. To correct these errors, it is essential to understand the nature of each type of information and associate it with the appropriate memory.