Agent Memory Series: Long-Term Memory Is More Than Vector Search

Reading memory projects by starting with embeddings, graphs, BM25, or RRF quickly turns into term matching. A better route is to follow one request through the system: whether the new message should become durable memory, whether an old memory should be preserved or softened, which tool results belong only to this task, and what finally enters the prompt as text, summary, entities, or ranked records.

This first piece is the route map. It explains the write and read loops as an executable path, then places Mem0, Letta, Graphiti, LangMem, TencentDB-Agent-Memory, OpenViking, Cognee, and Supermemory on one map. Later source chapters can then focus on which responsibility each system takes over, not on isolated feature names.

Start with one preference to understand the basic path: “when analyzing an error, show reproduction steps before repair suggestions.” A memory system must decide whether this sentence is worth keeping, what shape it should take, and when it should reappear. Later chapters switch to the representative unit that best exposes each system boundary: Letta uses a state block, Graphiti uses a temporal relationship, TencentDB-Agent-Memory uses tool logs and long-term facts, OpenViking uses a shared tree for resources, memories, and skills, and Cognee / Supermemory turn documents into platform context.

Two basic paths in agent memory

Write side:
  raw conversation / file / tool result
  -> extract reusable information
  -> normalize into record / block / edge / document
  -> deduplicate, update, timestamp, or assign ownership
  -> store under a scoped memory store

Read side:
  current question or task
  -> retrieve, filter, rank
  -> choose a small set of evidence, profile, or task structure
  -> inject into this turn's model view
  -> recover raw source through references when needed

1. Follow One Request: What Memory Does For An Agent

Imagine a coding agent working with you for three weeks. In week one, you tell it not to mock the database. In week two, a test failure teaches it that billing depends on real tenant configuration. In week three, you ask only: “continue the reimbursement fix from last time.” Without memory, the agent has to rediscover the chat and tool traces. If it pastes everything into the prompt, logs and stale conclusions drown out the useful facts.

This is where the memory layer enters: write reusable facts from conversation into memory records, then read a small set of relevant records back into the current model context. The write side may be called extraction, add, upsert, or memory write. The read side may be called search, retrieve, recall, or context assembly.

1.1 Write: Turn Raw Conversation Into Durable Facts

Writing memory is not a transcript dump. A conversation contains small talk, temporary instructions, tool output, user preferences, project facts, and changing timelines. The memory layer first decides what may be useful later, then turns that content into a more stable record. The write path usually has five steps:

Tokenization and embeddings are infrastructure in this loop. Keyword indexes need tokens; Chinese search often needs a segmenter such as GSE before BM25 can work well. Embeddings turn text into vectors so later semantic search can find similar meaning. These steps help indexing, deduplication, and similar-memory lookup. The judgment about whether a sentence deserves durable memory still needs extraction prompts, rules, schema, timestamps, and application boundaries.

The source makes that boundary concrete. In the current Mem0 OSS code, add() builds storage metadata and query filters from user_id, agent_id, and run_id, then sends normalized messages into _add_to_vector_store(..., infer). With the default infer=True, _add_to_vector_store() gathers recent messages, retrieves a small set of similar old memories, and then calls the LLM once for extraction.

processed_metadata, effective_filters = _build_filters_and_metadata(...)
vector_store_result = self._add_to_vector_store(
    messages, processed_metadata, effective_filters, infer, prompt=prompt
)

last_messages = self.db.get_last_messages(session_scope, limit=10)
existing_results = self.vector_store.search(..., top_k=10, filters=search_filters)
response = self.llm.generate_response(...)

The write entry already carries scope. The LLM sees new messages, recent messages, and a small set of similar old memories, not the entire memory store. Embeddings and vector search find nearby evidence; the LLM turns candidate facts into memory records.

1.2 Read: Assemble the Model View for This Turn

Reading memory has the opposite pressure: the memory store can grow, while the prompt can only accept a small working set. When the user asks “continue the reimbursement fix from last time,” the system should not paste three weeks of history. It narrows scope, recalls candidates through several signals, and then compresses the result into the model view.

That is why BM25, RRF, rerankers, and hybrid search often appear together. BM25 scores keyword matches. Vector search scores semantic similarity. RRF can fuse lists by rank instead of raw score. A reranker can make a finer relevance decision over the candidate set. The goal is simple: the memory store can be large, but the model view must stay small, precise, and relevant to the current question.

Mem0's read path shows the same shape. search() validates top_k, threshold, and filters, then calls _search_vector_store(). The lower function runs semantic retrieval, keyword retrieval, entity boosts, and a final scoring pass.

query_lemmatized = lemmatize_for_bm25(query)
query_entities = extract_entities(query)
embeddings = self.embedding_model.embed(query, "search")

internal_limit = max(limit * 4, 60)
semantic_results = self.vector_store.search(..., top_k=internal_limit, filters=filters)
keyword_results = self.vector_store.keyword_search(..., top_k=internal_limit, filters=filters)

scored_results = score_and_rank(
    semantic_results=candidates,
    bm25_scores=bm25_scores,
    entity_boosts=entity_boosts,
    threshold=threshold,
    top_k=limit,
)

This gives readers a concrete way to answer questions about BM25, RRF, and rerank. The visible OSS path combines semantic candidates, BM25-side keyword scores, and entity boosts inside score_and_rank(). rerank is an optional outer branch in search(); RRF is not an explicit step in this path.

1.3 Put The Concepts Back Into The Path

These concepts are easier to read when they are attached to a pipeline. When source code mentions a mechanism, first ask whether it helps persist facts, recall candidates, combine candidates, or control the model-visible view.

With that foundation, the projects below become easier to compare. Some are strongest on extraction, some on retrieval, and some are really about ownership: who owns raw history, profiles, relationships, workflow state, and the model-visible view.

2. Use Five Boxes To Locate Project Boundaries

Return to the reimbursement example. The agent is really holding five boxes. The first stores evidence: messages, files, events, and tool results. The second stores stable profile material: user preferences, team habits, and project constraints. The third stores relationships between people, projects, documents, and tasks. The fourth stores the timeline: what used to be true, what is true now, and what is planned. The fifth stores the model view for this one turn.

A useful first question for any project is which box it treats as the main battlefield. Mem0 puts write, merge, and retrieval between the application and the model. Letta binds profile, tools, messages, and agent state into one stateful agent. Graphiti organizes entities, relationships, episodes, and temporal validity as a graph. LangMem and LangGraph put memory work into workflow runtime, with hot-path and background paths. TencentDB-Agent-Memory starts with current-task context offload before building L0-L3 long-term memory. OpenViking unifies resources, memories, skills, and sessions under a viking:// context tree. Cognee and Supermemory productize knowledge, connectors, files, and profiles.

3. Seven Chapters, Eight Memory Routes

4. Why Mem0 Comes First

Mem0 is useful as the opening chapter because its algorithm history exposes the central tradeoff. The paper mem0 algorithm asks an LLM to classify similar old memories with ADD, UPDATE, DELETE, or NOOP. mem0g adds an entity and relationship graph. The April 2026 algorithm materials then emphasize single-pass ADD-only extraction, entity linking, hybrid retrieval, temporal reasoning, and memory decay.

That sequence gives readers a concrete tension. Updating old memories keeps the memory set smaller, but moves the judgment about staleness into the write path. Append-only writes preserve history, but retrieval has to do more work: ranking, time interpretation, and noise control. Once that tradeoff is clear, Letta, Graphiti, LangMem, TencentDB-Agent-Memory, OpenViking, Cognee, and Supermemory stop looking like minor variations around a vector database.

5. The Article Order

The series has seven chapters and eight project routes. The last chapter pairs Cognee and Supermemory because both move memory toward a platform boundary. The sequence follows ownership, not popularity: from an external memory layer, into agent state, then temporal graphs, workflow memory, and finally productized knowledge or context stacks. Each article should inherit a question from the previous one instead of becoming a feature checklist.

6. Ask Who Owns History

The most reusable takeaway will not be a single benchmark number. It will be an ownership checklist: who stores raw history, who maintains long-term profiles, who explains relationships and temporal changes, and who assembles the final model view. Without those boundaries, embeddings, BM25, graph traversal, and reranking can all end up pushing unresolved decisions back into the prompt.

The series will read from public code and contracts in that order: entry API, data model, write path, retrieval path, then engineering tradeoffs. The goal is to make each memory record easy to classify in a real agent: evidence, profile, relationship, workflow state, or context for this turn only.

Sources

• mem0ai/mem0
• letta-ai/letta and letta-ai/letta-code
• getzep/graphiti
• langchain-ai/langmem and langchain-ai/langgraph
• TencentCloud/TencentDB-Agent-Memory
• volcengine/OpenViking
• topoteretes/cognee
• supermemoryai/supermemory