AI Agents: A Text File Outperforms Memory Tools

Recommendations suggest measuring simple references first—full context and text search—before investing in a vector database, which is reserved for specific needs. On the LoCoMo benchmark, an agent storing data in text files and querying with grep is noted to outperform funded memory tools. The instinct to integrate a vector database for every conversational agent is being questioned due to challenges related to time and state, as well as costs and integration latency.

When to Keep a Vector Database and How to Decide

The use of vector databases is only deemed appropriate in specific situations: out-of-context extraction from very large datasets, relational reasoning involving multiple steps or time dimensions, or centralized structured storage with access management for multiple agents. A recommendation accompanies these particular cases: one should first evaluate full context solutions and file system text search, and only consider paid or vector-based solutions if they clearly outperform these two reference methods.

Why the Vector Instinct Often Fails for an Agent

Many teams default to a vector database as soon as an agent requires memory, but this habit is generally unsuitable for a conversational agent. Used as "memory as retrieval," the vector database poorly manages changes in time and state, leading to obsolescence and contradictions. Additionally, there are costs and latency associated with integration and repeated retrieval loops.

Simple References Outperform Products on LoCoMo

On the LoCoMo benchmark, simple references are reported to surpass complex memory products, including funded tools. Among these references is an agent that saves its memories in a directory of text files and queries them using tools like grep. Pre-trained language models perform classic file system tasks and standard queries more efficiently than custom memory APIs or queries on proprietary graphs, according to their designers. Memory is presented as an uninterrupted cycle of writing, management, and reading, where the model chooses what to retain and retrieve, and this file system-based method can be implemented with very little code.