Zero-Mem: Zero-Token Memory Operations for LLM Agents

Abstract: LLM agents need memory to act consistently over long interactions, yet many systems use additional LLM calls to operate that memory. Generating intermediate records and mediating their retrieval adds recurring token and time costs, while omitted or merged details can obscure the original evidence. We ask whether structured memory access requires generation at all. Zero-Mem introduces \emph{zero-token memory operations}: no step outside final question answering invokes an LLM or consumes LLM input or output tokens; encoder computation is accounted for separately. Zero-Mem preserves original interaction traces as its source of record. It organizes the traces in two complementary ways. An entity--context graph exposes connections across interactions, while a temporal hierarchy preserves conversational locality and session state. For each query, Zero-Mem weighs the two views, retrieves from both, and follows their structure to recover supporting relations or surrounding context. Deterministic calibration first discards conflicting evidence and then keeps the reader's answer grounded in the retrieved traces. Only the final-QA reader invokes an LLM. Across long-memory and long-context question-answering benchmarks, Zero-Mem achieves competitive performance while eliminating LLM calls and LLM-token consumption from memory operations. With the same final-QA reader and context budget, it reduces memory-operation time cost by 57.6\% relative to the fastest compared baseline. Ablations support the contribution of the two views and their query-dependent coordination. Overall, the results show that structured agent memory need not generate an intermediate representation of the past. After peer review, the code and implementation details will be available at \textcolor{blue}{https://github.com/TheMoon0815/Zero-mem}.

• The paper introduces Zero-Mem, a memory system that operates entirely without generative LLM calls by leveraging an entity–context graph and temporal hierarchy for trace preservation.
• The paper demonstrates deterministic memory organization and retrieval, achieving superior F1 and BLEU-1 scores (+5.4 F1 and +5.45 BLEU-1) on benchmarks like LoCoMo and HotpotQA.
• The paper evidences a 57.6% reduction in latency and enhanced transparency, cost efficiency, and traceability by isolating memory operations from final inference.

Zero-Mem: Eliminating Generative Memory Operations for LLM Agents

Motivation and Problem Statement

As LLM agents are increasingly deployed for long-term, multi-turn tasks, effective memory systems have become essential for retrieving relevant information from extensive, entity-rich, and temporally structured interaction histories. A prevalent strategy in agentic memory design is to utilize additional LLM calls for summarization, abstraction, organization, and retrieval of historical traces. While this lowers retrieval complexity and can improve downstream answer quality, it introduces substantial token and latency overhead, increases cost, and frequently sacrifices traceability or fidelity due to generative loss or omission in intermediate representations.

Zero-Mem is proposed to address a fundamental question: Can structured, high-fidelity memory for LLM agents operate entirely without generative LLM calls or LLM-token consumption, except for the final answer generation? The objective is to architect a memory pipeline in which all memory organization, indexing, and retrieval operations are non-generative and token-free, thereby isolating memory-operation cost from final inference and improving efficiency, transparency, and provenance preservation.

Methodology

Zero-Mem architectural design is based on retaining fully provenance-bearing raw interaction traces, eschewing generative abstraction at every step. Memory organization and retrieval are conducted through two interdependent but non-generative representations:

• Entity–Context Graph View: A non-parametric, provenance-preserving bipartite graph is built by applying lightweight NER models. Context nodes represent original trace units, connected to entity nodes (detected without LLMs); additionally, adjacency edges encode sequence continuity. Evidence propagation follows a multi-step, dense-similarity seeded PageRank over this graph, incorporating relational cues and lexical refinement.
• Temporal Hierarchy View: The interaction log is partitioned into episodes (reflecting session boundaries), windows, turns (utterances), and local spans. These units preserve fine-grained conversational locality and temporal state. Retrieval is performed in a coarse-to-fine manner, guided by answer-type and boundary constraints deterministically inferred from the input query without invoking an LLM.

  Figure 1: Overview of Zero-Mem, depicting how original traces are organized into provenance-preserving entity–context graphs and a temporal hierarchy, with all intermediate operations being token-free.

For each query, a deterministic query structure profile determines the balance (routing coefficient $\rho$) between the relational (graph) and hierarchical (temporal) retrievals; the two evidence sets are fused, then closed by adding bridging structures (from both views) before deterministic calibration removes spurious or out-of-bound content, ranks, and formats the evidence for the final answer generation. Throughout, only statistical, deterministic, or classical NLP components (e.g., NER, BM25, semantic embeddings) are used outside the terminal QA call.

Comparative Evaluation and Results

Zero-Mem is evaluated on the LoCoMo and HotpotQA benchmarks, representing long-term conversational memory and long-context multi-hop QA, respectively. Two model configurations are explored: GPT-4o-mini and Qwen2.5-14B, reflecting both closed-source and open-source settings.

Zero-Mem achieves the strongest average F1 and BLEU-1 scores in LoCoMo across all question types and both LLM readers. With GPT-4o-mini, it improves over the best baseline (GAM) by +5.4 F1 and +5.45 BLEU-1. With Qwen2.5-14B, it leads on every question type and metric.

For HotpotQA, Zero-Mem shows consistent improvements over all baselines across context-lengths up to 448K tokens, with an average improvement of 5.52 F1 over the previous best (GAM) at these extreme settings.

Figure 2: Ablation on HotpotQA shows both the graph and hierarchy views are necessary for optimal F1 and BLEU-1; evidence closure and calibration provide further measurable gains.

Figure 3: Retrieval budget ablation on LoCoMo. Performance increases rapidly up to top-5 to top-10 retrieval, saturating thereafter; top-5 is used for baseline comparability.

Also critical are Zero-Mem's efficiency results: memory operations incur zero LLM token or call overhead, and its non-generative design halves memory-operation latency compared to the next most time-efficient baseline (LightMem). Importantly, answer quality improves or matches the best generative-memory competitors, disproving the hypothesis that generative abstraction is a requirement for structuring large-scale agentic memory.

Ablations and Architectural Insights

Ablation studies rigorously confirm:

• Complementarity of Dual Views: Removing either the graph or hierarchy view imposes severe quality degradation, especially for cross-entity (relational) and session-local (temporal) queries. Combined, their coordination supports both distributed and local evidence needs.
• Evidence Closure and Calibration: Omitting evidence completion or calibration steps also results in significant drops in F1 and BLEU-1, illustrating the necessity of deterministic post-retrieval processing.

Implications and Future Directions

Zero-Mem demonstrates that provenance-preserving, structured memory for LLM agents can be operationalized entirely outside the scope of generative LLMs, except for the final reader. This delivers large practical and theoretical benefits:

• Practical: Substantial cost and latency reduction; transparency and debug-ability of evidence chains; improved privacy and compliance owing to strict provenance traceability.
• Theoretical: Separates the roles of memory structuring and generation in agent architectures; supports explicit evaluation of non-generative memory pipelines in compositional and multi-hop-agentic settings.
• Contradicts the often unstated assumption that only LLM-driven summarization, abstraction, and reflection can enable scalable and accurate memory for large-horizon agents.

  Figure 4: Contrasts Zero-Mem's operating regime against prior generative-memory and raw retrieval architectures.

Potential future research includes extending this dual-view, zero-token methodology to richer multi-modal traces, dynamic entity tracking, and time-evolving agent behaviors; further, joint optimization of retrieval and answer modules could refine overall effectiveness now that memory cost is amortized.

Conclusion

Zero-Mem establishes that LLM agents can achieve state-of-the-art long-term memory performance with no intermediate generative memory operations. Memory organization, indexing, retrieval, and calibration are entirely token-free; only the terminal QA reader depends on the LLM. Empirical results demonstrate superior answer quality and 57.6% lower latency relative to the most efficient prior art. This work compels the community to reconsider generative dependence in agent memory pipelines, motivating a new class of transparent, efficient, and provenance-maintaining agentic architectures (2607.29377).