- The paper introduces a rigorous evaluation of continuity in LLM memory systems by assessing seven distinct properties against standard benchmarks.
- It reveals that existing benchmarks typically cover just 0.43/7 of the continuity properties, underscoring a significant gap in current evaluations.
- The analysis recommends using ATANT v1.0 for genuine continuity assessment, cautioning against conflating retrieval metrics with persistence and stateful memory.
ATANT v1.1: An Authoritative Positioning of Continuity Evaluation in Memory, Long-Context, and Agentic-Memory Benchmarks
Abstract and Motivation
"ATANT v1.1: Positioning Continuity Evaluation Against Memory, Long-Context, and Agentic-Memory Benchmarks" (2604.10981) addresses a critical lacuna in the evaluation of memory systems for LLMs and agentic architectures: the concept of continuity as a system property distinct from memory retrieval, long-context recall, and agentic tool use. Building on the ATANT v1.0 framework, which operationalizes continuity via seven explicit properties and a robust LLM-free evaluation methodology, this companion paper interrogates the landscape of prevailing benchmarks (e.g., LOCOMO, LongMemEval, BEAM, MemoryBench, Zep, Letta/MemGPT, RULER) to clarify what is and is not measured by each. The thesis is formal and precise: no existing public benchmark prior to ATANT v1.0 offers direct, structural evaluation of continuity as defined along these seven axes.
Defining Continuity: ATANT’s Seven Properties
ATANT v1.0 decomposes continuity into seven properties:
- Persistence Beyond Session — State survives across process boundaries.
- Update Handling — Proper supersession semantics for outdated state.
- Temporal Ordering — Correct resolution of historical vs. current facts.
- Disambiguation — Robustness to ambiguous or overlapping entities.
- Reconstruction — Synthesis of state from distributed traces.
- Model Independence — Storage is decoupled from any specific model.
- Operational Usefulness — Utility across diverse, multi-domain surfaces.
These properties collectively exceed the scope of most memory or retrieval benchmarks, which tend to focus on chunk-level or context-window capacity, paraphrastic recall, or tool-call accuracy.
Structural Analysis of Benchmarks
The paper undertakes a granular, property-by-property analysis of seven widely used evaluations. The findings, summarized quantitatively, are decisive: the typical benchmark covers at most one property fully, with a mean coverage of 0.43/7 and none exceeding two properties at any degree of fidelity.
LOCOMO
LOCOMO is dissected as the community’s most-cited long-conversation benchmark. Its evaluation is shown to:
- Suffer from a structural “empty-gold” bug that renders 23% of its corpus unscorable, particularly penalizing correct refusal behavior.
- Rely on substring matching for paraphrastic answers, which rewards verbosity and surface overlap rather than structured memory, and fails to test persistence, update handling, model independence, or robust reconstruction.
- Cover only aspects of temporal ordering and partial disambiguation, demonstrably failing as a continuity proxy.
A particularly strong empirical result—Kenotic’s reference system scoring 96% on ATANT but only 8.8% on LOCOMO—serves as concrete evidence that the two benchmarks gauge orthogonal properties and should not be viewed as interchangeable.
LongMemEval and BEAM
LongMemEval and BEAM focus strictly on single-pass, in-context retrieval over very long token spans. Their scores can be achieved by any model with a sufficiently large context window, independent of any persistent or multi-session memory mechanism. They do not distinguish between continuity layers and base model capabilities, rendering them epistemically orthogonal to continuity evaluation.
MemoryBench and Zep
MemoryBench and Zep’s evaluations emphasize retrieval precision/recall or graph traversal within stored knowledge. While Zep's knowledge graph representation supports persistence and partial disambiguation in real deployments, the evaluation suite does not isolate or directly probe these properties, and it remains silent on update semantics, multi-model interoperability, and reconstruction.
Letta/MemGPT and RULER
Letta and MemGPT evaluate agentic tool use: whether the agent invokes read/write operations correctly during multi-step tasks. However, correctness of the resulting memory state is not tested for continuity properties. RULER is simply a context-window stress test, with no bearing on persistent state or continuity.
Property-Coverage Matrix and Implications
The synthesized property-coverage matrix reveals the stark limitations of current benchmarks:
- No benchmark except ATANT v1.0 exceeds a score of 1/7.
- Update Handling, Reconstruction, and Model Independence are uniformly absent, due to the design choices that prioritize reproducibility and one-pass evaluation.
This structural gap is not reflective of benchmark quality—each serves its articulated purpose—but rather evidences a misalignment between research claims of continuity and their empirical substantiation. The frequent conflation of retrieval- or context-window benchmarks with continuity measurements has led to systematic underinvestment in designing systems and evaluation protocols that address the full continuity property set.
Recommendations and Forward Guidance
The authors concretely delineate when each benchmark is appropriate and emphasize that only ATANT v1.0 currently provides valid continuity adjudication. They recommend that claims of continuity in system papers be supported by ATANT compliance levels, and that high scores on other benchmarks not be interpreted as evidence of continuity per se. This reporting standard, if adopted, would force a precise demarcation between retrieval, context, and continuity claims and clarify progress metrics across LLM memory system research.
Limitations and Future Directions
The evaluation is limited to Kenotic’s internal implementation and does not offer cross-system ATANT benchmarking (addressed in a proposed v2.0). The detailed cell assignments in the coverage matrix involve methodological judgment, defensively documented in the appendix. The paper also proposes a minimal patch to correct LOCOMO’s principal scoring defect, but the central claim stands even with this addressed.
ATANT v2.0 is charted to close partial property gaps in the original standard, particularly on reconstruction and operational diversity, pushing towards a full 7/7 property compliance.
Conclusion
"ATANT v1.1" (2604.10981) offers a rigorous, formal evaluation of the contemporary memory and long-context benchmark ecosystem, introducing needed clarity to the measurement of continuity as a distinct system property. The empirical and structural analysis demonstrates that advances in retrieval, agentic tool use, or context window size cannot be conflated with continuity without risk of misleading performance claims. Adoption of ATANT’s evaluation standard is positioned as necessary for any rigorous, property-complete assessment of stateful memory in LLM and agentic systems. Future work will extend these principles with broader system evaluations and an expanded checkpoint design.