Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents

Abstract: LLM agents increasingly rely on external memory systems to accumulate experience across tasks. Yet nearly all existing approaches, from graph-structured memories to reflective insight stores, access memory through fixed, hand-designed heuristics. We argue that this static view of memory is a core bottleneck for agentic learning because optimal memory behavior is fundamentally context-dependent. The early stages of the tasks, benefit from minimal retrieval because memory is sparse; recurring goal types benefit from plan reuse rather than generic nearest-neighbor lookup; stuck agents benefit from re-retrieval with alternative queries; and across long task streams, the memory store itself must be consolidated and pruned to remain useful. We present Memory as a Controlled Process (MemCon), a framework that models memory operations as a Markov Decision Process and learns an online policy that adaptively decides when, what, and how much to retrieve, when to inject a distilled plan, and when to consolidate or forget. MemCon is backend-agnostic: it wraps any existing memory implementation, learns from task-by-task binary feedback with no pretraining and no additional LLM calls, and uses a lightweight tabular contextual bandit with UCB exploration that converges within tens of tasks. Across 6 benchmarks, 3 agent frameworks, and 3 LLM backbones, MemCon consistently outperforms multiple memory baselines by up to 15.2 points in task success while reducing token consumption by 5--20%.

• The paper presents a novel adaptive memory management framework (MemCon) that models memory operations as a controlled process using a Markov Decision Process.
• The paper leverages UCB bandit methods for efficient online learning, achieving up to 15.2 point improvement in task success while reducing token consumption by 5–20%.
• The paper demonstrates backend-agnostic performance across diverse benchmarks and frameworks, paving the way for scalable, adaptive memory management in LLM agents.

MemCon: Adaptive Memory Management for LLM Agents

Motivation and Context

Agentic LLM systems have advanced through the integration of external memory stores, yet their memory access protocols remain overwhelmingly static: retrieval parameters (e.g., top-$k$, graph hop depth) and query templates are fixed regardless of the memory state, task phase, or agent progress. This rigidity introduces fundamental inefficiencies—retrieval can be unwisely aggressive when memory is sparse and overly conservative in situations demanding strategic recall or plan reuse. Existing alternatives either rely on LLM-driven controllers with prohibitive inference costs or hand-designed heuristics that cannot accommodate the intricacies of task evolution. The paper "Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents" (2607.13591) addresses these bottlenecks by proposing MemCon, reframing memory management as a controlled process within a Markov Decision Process (MDP) and leveraging efficient online learning to drive adaptive memory access.

Figure 1: MemCon's architecture wraps multiple agent frameworks and memory backends, enabling backend-agnostic memory control via an MDP policy learned online by UCB bandit methods.

Formulation: Memory as MDP

MemCon posits memory management as a stateless, episodic MDP, where each memory operation—Retrieve, PlanInject, Re-Retrieve, Consolidate, Forget, NoOp—is an action, and the state $\phi(s)$ encodes both agent/task signals (goal type, phase, stuck indicator, objects held, locations visited) and memory signals (store size, plan availability, learning phase). The reward function is episode-level, granting success bonuses, efficiency rewards, and penalizing failures. This tabular contextual bandit setting (with only hundreds of states) enables rapid convergence without deep RL or secondary LLM inference.

State and action discretization yields tractable learning and the UCB exploration rule efficiently selects actions based on visit counts and empirical Q-values, warm-started with domain-informed priors (e.g., retrieval and plan injection are rewarded, NoOp is penalized). Rewards are reverse-discounted at episode end, assigning greater credit to later actions. The controller operates at millisecond latency and adds zero inference cost.

Backend-Agnostic Wrapper and Augmented Operations

MemCon is a wrapper, not a new memory store—it intercepts the retrieve/store calls of any backend and decides how/when/what to retrieve. Two domain-agnostic augmentations are included: generalized plan injection (distills reusable templates from prior successes and injects them when task types recur) and goal decomposition (handles composite tasks by breaking them into object-wise subtasks with appropriate template retrieval). Both augmentations are independent of backend structure and rely solely on transcript manipulation.

Experimental Results Across Benchmarks and Backbones

MemCon is evaluated across 6 benchmarks—ALFWorld, PDDL, ScienceWorld (interactive), TriviaQA, WebWalkerQA, GAIA (QA/tool-use)—three agent frameworks (Lobster, LangGraph, Agent-Framework), and three LLM backbones (GPT-4.1-mini, Claude Sonnet-4, DeepSeek-V3.2). It is compared to nine strong memory baselines spanning vector retrieval, skill libraries, trajectory summarization, generative re-ranking, insight-based learning, graph-based hierarchical memory, and latent-token memories.

The dominant findings are:

• MemCon achieves the highest or near-highest task success/answer accuracy (S/A) across nearly all configurations, outperforming baselines by up to 15.2 points.
• Token consumption is simultaneously reduced by 5–20% compared to baselines; gains are not driven by increased context size but by smarter, context-dependent memory access.

  Figure 2: S/A vs. token cost reveals MemCon achieves superior accuracy with reduced token usage across frameworks and benchmarks.

• MemCon's policy generalizes robustly across agent frameworks and memory storage backends, showing top performance with strong LLMs as well as weaker ones.
• Ablation studies confirm the primary driver is the learned UCB controller, with plan injection and goal decomposition providing additive gains only for relevant task types.

Case Studies and Policy Behavior

MemCon's adaptive mechanism is empirically traceable: early phases trigger minimal retrieval, stuck states evoke re-retrieval with alternative queries, and familiar goal types preferentially inject generalized plans.

Figure 3: Case study of MemCon's stepwise adaptive behavior on two qualitatively distinct queries, demonstrating cold-phase exploration, stuck-state recovery, and plan injection for rapid success.

Robustness and Hyperparameter Sensitivity

A single-knob sweep demonstrates robust performance across reasonable ranges of learning rate, UCB constant, discount factor, and reward weights. The default action preset achieves the best average outcome, and component ablations validate that most gains are achieved by learned control, not simply by plan augmentation or composite goal heuristics.

Figure 4: Policy hyperparameter sensitivity: S/A remains stable across wide sweeps of learning rate, UCB constant, reward discount, and failure penalty.

Theoretical and Practical Implications

MemCon formalizes the agent memory controller as a contextual bandit, yielding provable logarithmic regret bounds and rapid convergence—a practical advantage for scalable deployment. The backend-agnostic design future-proofs agent memory management for emerging memory types (e.g., latent-token, hierarchical, or procedural memories, cf. [latentmem], [memskill], [procmem]). The separation between what is stored and how storage is accessed is critical for scalable, effective agentic learning and resilience in long-horizon, multi-task environments.

Practically, MemCon eliminates inefficiencies of fixed pipelines and costly secondary LLM controllers, enabling adaptive, low-latency memory access in online settings. Theoretical implications include refinements to policy learning for memory control, convergence guarantees, and broad applicability to memory-augmented RL or planning agents.

Speculative Outlook and Future Directions

The decoupling of memory control and storage paradigms paves the way for compositional memory architectures and hybridized memory-access policies. The lightweight policy-learning layer can be expanded toward fine-grained RL, hierarchical control, or meta-learning for memory management. Integration with latent-memory and procedural-memory systems may further boost agent generalization.

A promising direction is joint policy optimization across memory retrieval and tool invocation, combining MemCon-style bandit controllers with active search and planning strategies. The formalism also admits multi-agent settings, collaborative memory access, and federated memory management, crucial for society-of-minds, debate, and teamwork agents.

Conclusion

MemCon presents an efficient, backend-agnostic adaptive framework for memory management in agentic LLM systems by modeling memory operations as a controlled process. Empirical and theoretical results demonstrate that learning when, how, and how much to retrieve significantly enhances task performance and reduces resource consumption, outperforming static pipelines and costly LLM-driven controllers. The approach generalizes across benchmarks, agent frameworks, and LLMs, and provides a robust foundation for future developments in scalable, adaptive memory management for autonomous agents.