- The paper shows that no memory method dominates across scenarios, with DCI-Lite achieving the best mean generality rank at 3.67 while long context remains a strong, cost-competitive baseline.
- The paper identifies representation loss and retrieval failure as major weaknesses of index-based memory, while structural rate and query amortization predict when indexing can outperform a flexible harness.
- The paper introduces AutoMEM, which combines query-time planning with selective indexing to reach 67.3 on LoCoMo, outperform DCI-Lite by 22.3 percentage points, and reduce query-time tokens by up to 72% in a streamlined variant.
Motivation and scope
Memory systems for LLM agents have proliferated rapidly, but nearly all are tuned to a single deployment regime—typically multi-session chat or one agentic-trajectory format. This paper asks a question that prior benchmark work largely leaves unanswered: do these designs generalize across the heterogeneous trajectories agents actually encounter? The authors evaluate eight representative memory systems plus an off-the-shelf agentic harness (DCI-Lite) across five task families: single-turn QA over a large corpus (HotpotQA), multi-session personal chat QA (LoCoMo), agentic-trajectory QA (AMABench across ALFWorld, Web, and T2SQL domains), memory stress tests (MemoryAgentBench's AR/TTL/LRU/CR subtasks), and dynamic long-horizon agentic tasks (ALFWorld, MemoryArena shop/travel). Crucially, they track token cost and latency alongside accuracy, and they report serving-stack metrics (stable-prefix fraction, prefill-to-decode ratio, decode MFU) that raw token counts obscure.
The baselines span the architectural design space: long context as the no-memory control; note-based SimpleMem; multi-store PlugMem and LightMem; graph-based HippoRAG v2 and AMA-Agent; the DCI-Lite harness that keeps raw text and lets the LLM compose per-question grep/read plans; and RL-trained Mem-T and MemRL. The default backbone is Qwen3-32B for both construction and answering, with a Qwen3-4B ablation showing backbone-dependence of the conclusions.
Headline findings
Three patterns emerge from the main comparison. First, no method dominates: every index-based method (one that pre-builds a structured store) loses to plain long context on at least one benchmark, while DCI-Lite holds the best mean fractional generality rank (3.67). Second, long context is stronger than commonly assumed and remains cost-competitive: heavy-index methods such as HippoRAG cannot recover their preprocessing cost unless many future queries hit the same store, and lighter alternatives such as LightMem score below long context on several benchmarks. Third, token efficiency does not equal system efficiency: HippoRAG and AMA-Agent issue tokens as long chains of small serial LLM calls with KV-cache misses, inflating wall-clock latency far beyond what token counts suggest. The stable-prefix fraction predicts this wall-vs-token gap better than call counts do, and the authors recommend publishing serving-stack metrics alongside accuracy so orchestrator choices cannot dominate comparisons.
A notable methodological contribution is the argument for LLM-judge scoring over token F1 on verbose benchmarks: holding GPT-4o-mini fixed and changing only the answering prompt produces a 34-point swing in LoCoMo F1 driven purely by output verbosity.
Two failure modes of index-based memory on agentic QA
Index-based methods perform poorly on agentic-trajectory QA, several scoring less than half the long-context baseline. Using an oracle probe—with gpt-5.5 checking whether answers remain recoverable from each method's stored representation—the paper localizes two distinct failure modes:
- Representation-level failure: build-time schemas drop step- and action-level evidence outright. HippoRAG's entity-relation KG cannot encode step indices or action labels because they are not entity-relation facts; the information is destroyed before retrieval ever runs, and only widening the schema at build time can fix it.
- Retrieval-level failure: storage retains the answer but passive retrieval cannot surface it. PlugMem keeps over 98% answerability in its episodic store, yet its router picks one branch by embedding similarity and returns a compact note; when the answer lives in step-level detail, the agent has no mechanism to issue a follow-up query.
A third, related problem is inflexibility: committing to a schema at build time requires anticipating which relation types future questions will probe, which does not scale as agentic scenarios diversify.
When the harness wins, and when indexing pays off
Controlled probes isolate three conditions under which DCI-Lite beats structured indexing. On predicate complexity, when HotpotQA questions refer to key entities indirectly rather than naming them, DCI-Lite leads HippoRAG by +11.3 pp; when both entities are named directly, it trails by 3.1 pp. On corpus–window regime, the harness wins decisively only when the corpus exceeds the context window (+29.5 pp on HotpotQA) and loses when it fits (−16.5 pp on LoCoMo's ~26k tokens). On schema expressiveness, where only ~10% of AMABench-ALF predicates are encodable in the index schema, DCI-Lite leads by +21.3 pp.
The paper then characterizes when indexing still pays off via two jointly necessary conditions. The first is amortization: on AMABench, LightMem's per-question cost crosses below DCI-Lite at roughly 2 shared queries, and even HippoRAG at roughly 32. The second is demand, measured by a structural rate—the fraction of questions whose answers cannot be recovered from any single retrieval unit under flat retrieval. This rate cleanly predicts head-to-head outcomes: on LoCoMo (60.7% structural rate), indexed methods beat DCI-Lite by +13 to +14 pp; on HotpotQA (17.7%) and AMABench domains (8.6–10.3%), they lose by 7–23 pp. Schema expressiveness gates whether indexing can help at all; structural rate gates whether it is needed. Both must hold for indexing to beat the harness.
Scope limit: dynamic agentic tasks are policy-limited
On dynamic agentic tasks, the entire memory-design axis collapses: no memory method consistently improves the long-context baseline on ALFWorld. A hand-crafted golden procedural memory distilled from expert trajectories lifts Qwen2.5-7B success from 46.7% to 71.7%, while GRPO post-training reaches 86.7%. The residual gap splits into two parts. Gap 1—an over-entropic action posterior causing step-budget exhaustion—is closable by procedural rules, but agents fail to self-distill such recipes from their own trajectories, implying propositional content must come from a stronger oracle. Gap 2—state-dependent decisions that text rules cannot enumerate—persists even with golden memory in context (17 of 32 failures), delineating a capability ceiling for self-evolving agents whose only update channel is non-parametric memory. Notably, RL-trained memory controllers like Mem-T and MemRL post-train a controller atop an unchanged actor and therefore remain bounded by this ceiling.
AutoMEM
The findings motivate AutoMEM, an agentic loop that migrates selective indexing into the harness. At each step the planner picks one action from {rg, read, dump, index_query}; an executor runs it without LLM involvement; a judge inspects the accumulated trace and either accepts it for answering or returns a hint naming missing evidence, triggering re-planning. The default index_query targets a Cypher graph built from LightMem's write-time summary index.
Results support the design: AutoMEM scores 67.3 on LoCoMo, improving over DCI-Lite by +22.3 pp, over long context by +5.8 pp, and edging the previously strongest Mem-T by +0.8 pp, while achieving the best average rank in the suite (3.10). The largest lift lands on LoCoMo, the highest-structural-rate benchmark—consistent with the diagnostic framework. Two efficiency optimizations preserve most accuracy at lower cost: a KV-cache-friendly chat layout raising prefix-cache hit rates from 5.4% to over 50%, and a single-pass variant (Kmax=1, parallel rg/read/index_query) cutting query-time tokens by 66–72% at −3.8 pp on LoCoMo.
Limitations and open questions
The paper is candid about several constraints. AutoMEM uses a uniform backbone for plan, judge, and answer steps; a tiered router reserving the strong model for answer composition would cut cost roughly 2–3×, but is not implemented, so reported costs are upper bounds. Judge protocol variance is acknowledged: lenient prompts can reverse rankings on conversational QA, so absolute scores should be read against the strict factual rubric used throughout. Evaluation uses fixed deterministic subsets (e.g., n=200 per QA benchmark) due to compute cost, and the MemoryArena environment had to be re-implemented from the released dataset. The oracle answerability probe is an upper bound on retrieval, not an end-to-end score. Open questions include the intrinsic limits of index-based memory for agentic use cases, a memory DSL enabling meta-learned schemas per workload, and infrastructure that batches the serial plan/judge/answer calls dominating wall-clock latency.
Conclusion
This paper reframes memory-system evaluation around cross-scenario generality and delivers a clear empirical verdict: existing structured memories are scenario-narrow, failing on agentic trajectories through build-time schema loss and passive retrieval, while deferring structure commitment to query time yields the best generality. The structural-rate metric offers a predictive rule for when indexing pays off, and AutoMEM demonstrates that combining a harness with selective indexing reaches the best cost–accuracy position among evaluated systems. The finding that dynamic agentic tasks are policy-limited—bounded by parametric rather than non-parametric updates—marks the boundary of what memory engineering alone can deliver.