Designed agent memory stopped paying off, and the benchmarks missed it

Agent memory stopped paying off this month, and almost nothing in the field noticed. The reason is a measurement problem rather than a modeling one, and the measurement problem is fixable in an afternoon.

The evidence is four papers published in the first three weeks of October 2026. They attack the same question from different angles and land in the same place.

Memory stopped paying off

The clearest result is MemoryTax, posted on 21 October 2026. It asks a blunt question. Agent memory systems such as Mem0 and Zep decide what to keep before any question arrives. Coding agents keep the raw history and let the model search it with its own tools. Which one wins, now that the model is strong enough to search for itself?

The study runs one LoCoMo conversation of 419 turns and 152 questions, across four memory methods (full context, BM25, Mem0, Zep) and three coding agents (Pi, Codex, Claude Code), each driven by GPT-4o-mini, GPT-6 Luna, and Claude Opus 5.5. Every rollout is costed.

The answer is uncomfortable for anyone selling a memory product. Mem0 beats full context only with GPT-4o-mini, the weakest model in the panel, and building the Mem0 index alone costs more than a whole full-context rollout. The paper’s own summary names the mechanism: agent memory is a costly patch for a weak model.

It gets worse. Memory also has a ceiling set by the model that writes it. A stronger reader does not recover what a weaker writer left out. The usual upgrade path, swap in a better model, does not rescue a memory layer that dropped the fact on the way in.

With Claude Opus 5.5 the simplest design wins. Pi, the coding agent that keeps the raw history and lets the model search it, tops the cost-accuracy frontier, scoring 99% for less than half the cost of full context. Its one miss turns out to be a wrong answer key in the dataset.

Two caveats, because the scope is narrow. This is one conversation with one run per cell, so read the percentages as direction rather than precision. And part of Pi’s lead comes from photo captions only the agents could read, which means the gap on the 88 questions with no image in the evidence is one answer.

The null that took four corrections

The second paper is harder to argue with, because it reports no effect at all. Persistent Memory in Multi-Agent LLM Inference, posted on 6 October 2026 and accepted at the NeurIPS 2026 Machine Learning for Systems workshop, measures a three-tier agent architecture whose top tier stores and recalls reasoning traces.

Decomposing the context works. Peak KV working set drops to 14.3 MiB per query, against 35.5 and 35.3 MiB for single-pass and retrieval-augmented baselines.

The persistent memory tier does not work. Across eight controlled dataset pairs at 100 items per arm, it costs an extra 0.368 MiB of peak cache and produces an accuracy change of +0.015, with a 95% confidence interval of [-0.011, +0.046]. The interval contains zero.

The authors argue the null is structural rather than a property of their system. Single-question benchmarks hand each item its own evidence and score it independently. Correctness requires resetting stored traces between conditions. So recall has nothing informative to retrieve, and the memory system is being asked to prove itself in a setting built to make it look useless.

The methodological detail is the part I keep coming back to. Reaching that null took four measurement corrections. Three of them inflated the apparent benefit of memory, and the fourth made an effect that size look resolvable. None of the four appears in the results table. Any ablation reporting an accuracy gain from a memory tier should be read against that: the error bars of the experiment were wider than the effect, and it took a careful analyst to notice.

Score the store, not the answer

Both papers above share a root cause, and the third names it.

Agent memory benchmarks score a final answer. That answer depends on three things: whether the memory store held the right fact, whether retrieval found it, and whether the reader used it well. Score the answer and you cannot tell them apart. Beyond Downstream QA, posted on 15 October 2026, says so and then measures the damage.

Its benchmark is FEB-30: 30 fact-evolution trajectories across 6 diagnostic categories and 3 difficulty levels. The trick is that the documents get generated from a known fact trajectory, so the ground truth is deterministic and needs no human annotation, and the store can be inspected after every ingestion step.

On one representative memory system, intrinsic evaluation (read the store) and extrinsic evaluation (ask the question) diverge by 16 percentage points on average, and 44% of trajectories exceed a meaningful divergence threshold. Decomposed, the system classifies its own memory operations (create, update, reinforce, supersede) correctly only 20% of the time. Extrinsic evaluation cannot see that at all, because a right answer and a right final answer look identical.

MemDerive, posted on 18 October 2026, makes the same point with a different instrument. It derives exact gold answers with Datalog and records the gold derivation DAG, so a wrong answer can be traced to the state that was missed. Across 4,692 events and 1,503 queries, every memory system it tests falls behind plain BM25 retrieval on derived state, even though the same answer model reaches 96.01% when it is simply handed the supporting events.

Read that last number carefully. The reader can answer. The store and the retrieval are what fail. Most failures are ungathered, meaning the system held the missing intermediate state but did not retrieve it for the question that needed it. That is a different bug from the one memory marketing assumes.

What to measure on Monday

If the benchmarks are broken, what do you run instead? Four answers, in increasing order of effort.

MemProbe, accepted at EMNLP 2026, turns cognitive memory research into four reusable experimental paradigms: interference, misinformation, consolidation strength, and reconsolidation window. It evaluates six incremental memory systems on a 56-episode suite and finds that systems with similar aggregate scores have distinct behavioral profiles. Two memory systems can both answer 80% correctly and fail in completely different places.

Benchmark design is starting to catch up. LongMemEval-V2, out since May 2026, moves from chat question answering to web and enterprise agent trajectories, up to 115 million tokens, and scores answer accuracy and query latency together. Its leaderboard ranks systems by how much they improve a fixed reference frontier rather than by their absolute score, and one of the shipped baselines is a vanilla Codex coding-agent memory baseline. That is the control MemoryTax argues for, adopted as a baseline.

Two cheap checks cost an afternoon. Give your system a fake memory, a whole invented life matched in shape to the real one, and see whether your metric scores it the same. Then run your test with the memory removed, because a system with no memory at all can pass a recognition probe by reasoning its way to a plausible answer. The Kairos checks spell out both and release the code. Treat those results carefully. They were run against 2025 models, and nobody has re-run them against Claude Opus 5.5 or GPT-6 yet, which is exactly why you should run them on your own stack rather than quote them.

And do the thing nobody does. Dump what the store actually holds after ingestion and read it. A memory system that keeps evidence lets you see which stored belief a wrong answer came from, instead of guessing between memory and retrieval.