Introduction
An agent is only as good as the memory it can reach. Storing everything it has ever seen is the easy part; databases have done that for decades. The hard part is the reverse: out of everything ever recorded, surface the one piece that answers the question in front of you, the moment it is asked, and leave the rest behind.
Get that wrong and the sharpest model in the world still answers from nothing. So that is what we set out to measure: not how well a system stores, but how well it finds.
This report covers three things, each reported the honest way. How often Sift retrieves the right memory. How much context it carries to do it. And how often the final answer is correct, with a blind split held out of all tuning, so the numbers are not the product of peeking.
Why LongMemEval?
A memory benchmark is only useful if it looks like real use. LongMemEval-S does. Each of its five hundred questions is buried in a history of around fifty past conversations, the kind of accumulated context a real assistant carries for a real person, more than a hundred thousand tokens in all.
The questions are deliberately varied. Some ask for a single fact stated once. Some require counting or combining evidence across many sessions. Some turn on dates and ordering. Some depend on a fact that changed over time, where only the latest version is correct. A system that simply memorizes cannot pass; it has to retrieve the right evidence and reason over it.
We split the five hundred into training, development, and a blind held-out set, and never looked at the held-out set while tuning. We report on all five hundred, and call out the held-out split on its own, because a result you can only get by peeking is not a result.
Architecture
Sift turns a question into an answer through the path below, and a language model appears in only one place: the very end, where the reader writes the answer. Everything before it is arithmetic, including the step that decides what is relevant.
flowchart TB Q(["your question"]) --> E(["embed"]) --> S["score"] S --> L["lexical: IDF + adjacency"] S --> V["vector: meaning"] S --> G["graph: connections"] L --> R(["rank"]) V --> R G --> R R --> T(["top 5 sessions"]) --> P(["snippet pack"]) --> O(["~1,900 tokens"])
Anatomy of a query
Here is a real question from the benchmark, and the haystack it lives in. The answer sits in a couple of these conversations. Everything else is noise.
Fifty-one past conversations. Sift surfaces the five that matter, in about 1,900 tokens, and the answer is right there: a sports store downtown.
Lexical ranking
The first signal is lexical, and it does most of the work. Every query term is weighted by how rare it is across the whole corpus, its inverse document frequency, so a specific word like racket counts far more than the, with an added bonus when matched terms sit next to each other. Rare, specific language dominates; filler barely registers.
Switching from raw term-counts to this IDF-proximity ranking, with nothing else changed, gave us the single largest recall gain of any change we made. It is the main reason the numbers in this report are what they are.
Vector and graph
Two more signals refine the ranking. A dense vector from a single embedding model measures meaning, so a memory that says the same thing in different words still surfaces even when no words match. A graph signal measures how connected a memory is to others already scoring well, so corroborating context rises together rather than in isolation.
The three signals are added into one score. None of them is a language model, and none is learned at query time.
Snippet packing
Sift does not hand the reader whole sessions. From the top five, it packs snippets around the matched lines, a tight budget of the required evidence plus a little neighboring text for context. That is how around fifty sessions of history, more than a hundred thousand tokens, collapse to about 1,900 tokens while keeping the session that holds the answer.
No model in the loop
Here is what is missing, and why it matters. Two of the systems we measured, Zep and agentmemory, run a model at retrieval time: a cross-encoder that re-scores every candidate before the reader sees it. Sift runs nothing of the kind. Once the query is embedded, the ranking is pure arithmetic, with no model deciding what is relevant.
Performance results
We measure three things: whether the right memory is retrieved, how much context that takes, and whether the final answer is correct. The first two are where Sift wins cleanly. The third we report carefully, because it is the number the field most often reports wrong.
Recall
Recall asks one question. Of the five sessions we hand the reader, is the one holding the answer among them?
Top five out of five hundred. Supermemory, the only other system that publishes a recall number, needs fifteen candidates to reach 95 and still trails Sift's five. The six questions Sift misses are the ones where the session holding the answer never reached the reader's context at all. Recall is the ceiling on everything downstream.
Context efficiency
Every token of context you hand the reader is a token you pay for, on every query, forever. Sift answers from about 1,900.
One system here is leaner than Sift: Supermemory, at around 720 tokens. It carries less context, and its recall is lower. We would rather spend a little more and find the answer almost every time.
The leaderboard
Line up the public answer-accuracy numbers and Sift looks mid-pack. The comparison is rarely like for like, and two things are worth knowing before you trust it.
Supermemory's headline 95% is a Recall@15 number, placed in a column of answer-accuracy numbers. A different measurement entirely.
Another widely cited 90% is not the figure in the paper it comes from, which reports 71% with a gpt-4o reader.
Here is the full board. The systems use different readers, different context budgets, and different judges, and the order tracks those at least as much as it tracks the memory underneath.
Almost every number above is read by a stronger model than Sift's gpt-4o, on several times the context. Hold the reader fixed, at the same gpt-4o anyone can run, and the order changes.
We have tried to compare like with like throughout. Recall against recall, tokens against tokens, and answers graded by a model that did not write them.
Downstream answers
We report answer accuracy too, because it matters. We just refuse to lead with it, and we refuse to report it the way the field usually does. Three things make that number softer than it looks:
- It swings on the reader. The same system can move ten points just by changing which model reads the context.
- It is usually graded by the same model that wrote the answer, which marks its own work kindly.
- It is tuned and reported on the same questions, with no held-out set to catch overfitting.
So we do the opposite. Each configuration runs on two readers, grok-4.3 and gpt-4o, graded by the official LongMemEval judge and re-checked by a second model, across training, development, and a blind held-out split reported separately. Because provider sampling alone moves a full run by four or five questions, we treat any smaller difference as noise rather than a result.
One number is still pending: the full five-hundred answer accuracy with our benchmark-specific aids stripped out, so what we publish reflects general retrieval and not hand-tuning to the test set. We are finalizing it under the protocol above and will publish it here, held-out split and all.
What we can already say: retrieval is not the bottleneck. On more than nine in ten of the questions Sift gets wrong, the right evidence was sitting in the context the whole time. The gap is the reader reasoning over what it was given, not the memory failing to find it.
Threats to validity
A few things could undercut these numbers. We would rather name them than have a reader find them.
- Answer accuracy depends on the reader. The same retrieval, read by a stronger or weaker model, moves the QA number more than the memory does. We report two readers for exactly this reason.
- QA is a judged metric. Provider sampling alone moves a full run by four or five questions, and any LLM judge has its own bias. We re-grade with a second model and treat sub-run differences as noise.
- Recall is on one corpus. 98.8% is LongMemEval-S, a single benchmark we tuned our packing budgets against. It is a strong result, not a promise that recall holds everywhere, and gold short-answers carry their own punctuation and paraphrase ambiguity.
How to reproduce
Sift is not open source today, but it will be. We are preparing the retrieval stack for a public release, so these results can be run and checked directly rather than taken on our word. Until then, this report is the complete account: the retrieval method described above, every setting that produced these numbers in the appendix, the public benchmark, and the exact definition of each metric. The method we measured is the one that runs in production, unchanged.
Conclusion
Most memory systems are judged on a single answer-accuracy number, graded by their own model, on the data they tuned against. We think that is the least trustworthy thing you can report, so we led with the parts that hold up: recall, cost, and a retrieval path with no model in the loop.
Storing everything is easy. Finding the one thing that matters, cheaply and reliably, is the work. That is the part we have spent ours on.
Citations
- Wu et al. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. ICLR 2025. arXiv:2410.10813
- Rasmussen et al. Zep: A Temporal Knowledge Graph Architecture for Agent Memory. arXiv:2501.13956; getzep.com/research
- Mem0. Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv:2504.19413; mem0.ai/research
- Supermemory. Research. supermemory.ai/research
- agentmemory. github.com/JordanMcCann/agentmemory
Appendix
The frozen configuration and the metric definitions behind every number in this report.
- Recall@5
- the session holding the answer appears among the top 5 retrieved sessions, scored per question over all 500
- Answer accuracy
- the final answer judged correct under the official LongMemEval judge protocol
- Context tokens
- cl100k count of the packed context; reported as the mean per query (median ~1,740)
- Embeddings
- text-embedding-3-large, 1,280 dimensions
- Ranking
- lexical-vector-v1: IDF-proximity lexical, dense vector, graph adjacency
- Retrieval depth
- top 5 sessions
- Snippet budget
- 1,250 session / 460 required / 160 neighbor characters
- Readers
- grok-4.3, gpt-4o-2024-08-06
- Judge
- official LongMemEval protocol, cross-checked with a second model
- Splits
- 297 training, 106 development, 97 blind held-out