Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory

Abstract: Long-horizon language agents must operate under limited runtime memory, yet existing memory mechanisms often organize experience around descriptive criteria such as relevance, salience, or summary quality. For an agent, however, memory is valuable not because it faithfully describes the past, but because it preserves the distinctions between histories that must remain separated under a fixed budget to support good decisions. We cast this as a decision-centric rate-distortion problem, measuring memory quality by the loss in achievable decision quality induced by compression. This yields an exact forgetting boundary for what can be safely forgotten, and a memory-distortion frontier characterizing the optimal tradeoff between memory budget and decision quality. Motivated by this decision-centric view of memory, we propose DeMem, an online memory learner that refines its partition only when data certify that a shared state would induce decision conflict, and prove near-minimax regret guarantees. On both controlled synthetic diagnostics and long-horizon conversational benchmarks, DeMem yields consistent gains under the same runtime budget, supporting the principle that memory should preserve the distinctions that matter for decisions, not descriptions.

• The paper introduces a decision-centric memory compression framework that evaluates memory based on its impact on decision quality, achieving higher evidence recall than description-based methods.
• It formalizes the tradeoff between memory budget and decision performance through a rate-distortion analysis using covering and packing complexity, proving NP-completeness for optimal partitioning.
• The implemented DeMem system leverages certified incompatibility and budgeted encoding to deliver lower distortion and superior benchmark performance in long-horizon, multi-session tasks.

Decision-Centric Memory Compression for Language Agents

Motivation and Problem Formulation

The exponential growth in interaction horizons for LLM-based agents has underscored the necessity for efficient memory management, especially under strict runtime budget constraints. Existing memory systems often prioritize descriptive criteria (semantic similarity, salience, recency, summary fidelity), but empirical analysis reveals a significant mismatch: descriptive similarity is a weak predictor of decision evidence compatibility. For example, description-based retrieval recovers only $66\%$ of gold evidence on LoCoMo, while a principled decision-centric approach achieves $83\%$ recall, and $85\%$ of errors in description-based systems arise from precedence or dilution of critical evidence.

The paper "Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory" (2605.10870) formally addresses agent memory as a decision-centric compression problem. It introduces a theoretical framework in which memory is evaluated by the loss in achievable decision quality induced by compression, rather than by fidelity to descriptive features. The runtime bottleneck is modeled as a fixed number of $K$ internal memory states (slots), and the decision-centric rate-distortion frontier characterizes the optimal tradeoff between memory budget and decision quality.

Theoretical Contributions

Decision-Centric Rate-Distortion Boundary

The memory constraint is formalized as a contextual bandit: for each history-query pair $(h,q)$, the agent maps history into a slot $m$ and selects an action based on $m$ and $q$. The central theoretical question is when histories can be safely merged into the same memory state without degrading downstream decision quality. The paper establishes an exact forgetting boundary:

• Histories may share a memory state if and only if they admit a common $\epsilon$-optimal action for the current query; descriptive proximity is irrelevant unless it aligns with action compatibility.

This motivates the definition of a decision-centric distance $d_{\text{dec}}^q(h, h')$, representing the minimal worst-case gap incurred by merging two histories.

Covering and Packing Complexity

• Covering number: minimum number of memory states required to bound distortion by $83\%$0.
• Packing number: maximum set of mutually incompatible histories at scale $83\%$1.

The authors prove tight bounds for the decision-distortion frontier in terms of these combinatorial quantities. An information-theoretic lower bound is derived: the entropy of task-relevant distinctions is governed by packing, not by descriptive diversity.

Computational Hardness

Optimal partitioning for memory compression in terms of decision distortion is NP-complete. This result constrains the design space for agent memory systems—practical algorithms must use tractable relaxations and approximate cluster compatibility.

DeMem: Certified Online Memory Refinement

To operationalize the theoretical framework, the DeMem system is introduced. It routes each history-query pair into one slot and refines partitions only when empirical evidence certifies a conflict in decision requirements.

Figure 1: DeMem routes histories into bounded slots and splits only on certified conflict.

DeMem is built around two core mechanisms:

• Certified incompatibility: Pairwise confidence intervals and empirical rewards identify when co-routed histories cannot share a near-optimal action.
• Budgeted encoding: Greedy graph coloring partitions contexts under a strict slot budget, using certified conflicts as cannot-link constraints.

This approach yields near-minimax regret guarantees, separating unavoidable compression loss from statistical learning.

Empirical Validation

Synthetic Memory Bandit Analysis

In environments with intentional mismatch between description and decision identity, DeMem achieves the lowest cumulative regret among budgeted methods. The empirical memory-distortion curve shows that DeMem consistently attains the lowest distortion and the largest gains under severe description-decision mismatch.

Figure 2: Synthetic results. (a) Cumulative regret on the Decoupled Bandit. (b) Memory--distortion tradeoff under varying $83\%$2. (c) Performance under increasing description--decision mismatch.

Direct validation of partition quality demonstrates monotonic alignment between induced distortion and regret across methods and seeds.

Figure 3: Direct validation of the theory object in the synthetic environment. (a) DeMem learns the lowest-distortion partition; description-aware baselines degrade as mismatch grows. (b) Downstream regret is broadly monotone in induced partition distortion across methods and seeds.

Benchmark Results

Across LoCoMo, LongMemEval, and MemoryArena, DeMem achieves the highest mean overall score, particularly in categories requiring cross-session integration and long-term recall:

• On LoCoMo, DeMem obtains up to $83\%$3 accuracy, outperforming RAG, Mem0, EMem-G, Mnemis, and other state-of-the-art baselines under matched memory budgets.
• On MemoryArena, DeMem achieves the highest mean success rate (Avg SR $83\%$4), demonstrating robust performance in interdependent multi-session agentic tasks.

Mechanism audits show that DeMem achieves higher support purity, lower within-slot conflict, and greater gold-support coverage than feature-based or description-only memory partitions.

Ablation and Robustness

Certified splitting, decision-aware routing, and slot-memory representation are all critical—removing any significantly degrades performance. Split precision is high ($83\%$5), and splits are triggered selectively ($83\%$6 of routing events), indicating efficiency and accuracy in partition refinement.

Figure 4: Ablation and robustness. (a) Parameter sensitivity over 144 configurations. (b) Component ablations. (c) Accuracy vs. memory budget. (d) Accuracy vs. latency.

The decision-centric selection criterion is modular: drop-in application to RAG and EMem-G improves evidence recall and answer quality without altering other pipeline components.

Implications and Future Directions

The paper conclusively demonstrates that memory systems for agents should be organized around the preservation of decision-relevant distinctions, not descriptive similarity. The theory provides formal limits, practical algorithms, and empirical validation spanning synthetic and real-world benchmarks.

Practical implications are substantial: memory bottlenecks are ubiquitous in long-horizon deployment (clinical, legal, educational, agentic). Compression by description can discard critical actions; decision-centric partitioning is essential for reliability.

Theoretical implications include a unified framework for agent memory as a rate-distortion problem, and tight complexity results defining fundamental limits of budgeted agent memory for downstream decision quality.

Future research should further investigate scalable certified splitting under adversarial or dynamically shifting decision requirements, integrate decision-centric memory within larger agentic harnesses, and extend rate-distortion analysis to settings with adaptive feedback or contextual nonstationarity.

Conclusion

The decision-centric rate-distortion framework fundamentally restructures agent memory by grounding compression in achievable decision quality. DeMem implements certified refinement according to this principle, yielding formal guarantees and consistent empirical advantages across tasks and backbones. Memory should preserve distinctions that matter for action—not just for description. The framework is architecture-agnostic and provides a principled module for integration into future AI agent systems.