Vector Search in AI Does Not Ensure Reliable Recall

Vector search brings ideas closer together, not the truth of facts. A practical guide details where cosine similarity goes off track for agent memory, what to actually index, and how to activate it in a LangGraph. The key takeaway is recommendations for transitioning from a verbose memory to relevant recalls.

Activating Indexing in LangGraph and Avoiding the Trap of Non-Indexed Episodes

Semantic search can be activated in a LangGraph through an index that explicitly specifies which fields are converted into embeddings. An existing agent is then upgraded to use this indexed search for both facts and episodes, altering the way it retrieves information. A practical pitfall is highlighted when episodes do not contain a selected field for indexing: these contents will not be found semantically. An exercise also proposes evaluating the robustness of the system against contradictions, comparing disordered text to distilled formulations, and measuring the retrieval of recent episodes against those that are actually indexed.

When Similarity Misleads: Four Recurring Modes of Failure

Proximity in the vector space answers the question "What is this text about?" rather than the validity of a retrieved fact. Four typical modes of failure are described: thematic similarity that does not imply relevance to the task, errors related to negation or contradictions of facts, discrepancies between the embedding of a query and that of the memory, and incompatibilities between embedding models. These limitations explain why a seemingly close query can yield an inappropriate or incorrect recall.

What to Index, Illustrated by the Priya Example

A use case contrasts the request "Suggest something for dinner tonight" with a relevant fact "Priya is vegetarian," without any common words; a keyword search would fail to connect these elements, and injecting all memories into the prompt ceases to be viable when moving from ten to a thousand items. Embeddings convert text into vectors that bring semantically similar contents closer together, typically ranked by cosine similarity. To make this memory usable, it is recommended to index distilled and standalone statements, and for episodes, to target the situation or request rather than the outcome. This framing improves retrieval by similarity without relying on lexical overlap.