Learning to Continually Learn via Meta-learning Agentic Memory Designs

Abstract: The statelessness of foundation models bottlenecks agentic systems' ability to continually learn, a core capability for long-horizon reasoning and adaptation. To address this limitation, agentic systems commonly incorporate memory modules to retain and reuse past experience, aiming for continual learning during test time. However, most existing memory designs are human-crafted and fixed, which limits their ability to adapt to the diversity and non-stationarity of real-world tasks. In this paper, we introduce ALMA (Automated meta-Learning of Memory designs for Agentic systems), a framework that meta-learns memory designs to replace hand-engineered memory designs, therefore minimizing human effort and enabling agentic systems to be continual learners across diverse domains. Our approach employs a Meta Agent that searches over memory designs expressed as executable code in an open-ended manner, theoretically allowing the discovery of arbitrary memory designs, including database schemas as well as their retrieval and update mechanisms. Extensive experiments across four sequential decision-making domains demonstrate that the learned memory designs enable more effective and efficient learning from experience than state-of-the-art human-crafted memory designs on all benchmarks. When developed and deployed safely, ALMA represents a step toward self-improving AI systems that learn to be adaptive, continual learners.

• The paper introduces ALMA, a meta-learning framework that autonomously discovers memory architectures to enable efficient continual learning in agentic systems.
• It leverages open-ended code-space exploration using a Meta Agent (e.g., GPT-5) to optimize modular memory update and retrieval processes across diverse domains.
• Empirical results show superior success rates, adaptation to distribution shifts, and higher sample efficiency compared to manually engineered memory systems.

Automated Meta-Learning for Continual Agentic Memory Design: An Expert Review

Motivation and Problem Formulation

The statelessness of contemporary foundation models (FMs) fundamentally impedes the ability of agentic systems to achieve continual learning, preventing effective reuse of historical experience and degrading adaptive performance in sequential decision domains. While the integration of external memory modules has become standard to address this limitation, prevailing designs remain human-engineered, domain-specific, and static. This constrains scalability and inadequate adaptation to the heterogeneous and non-stationary demands encountered in real-world tasks, as evidenced by disparate requirements in conversational, strategic, and embodied environments.

Recognizing the trajectory of AI design toward learning-based optimization, ALMA ("Automated meta-Learning of Memory designs for Agentic systems") is presented as a paradigm wherein memory architectures themselves are meta-learned via open-ended code-space exploration, rather than manually specified. The objective is to discover, from scratch, memory policies and structures capable of supporting highly efficient, domain-aligned continual learning for downstream agentic systems in diverse interactive environments.

ALMA: Design and Meta-Learning Framework

ALMA leverages a Meta Agent, instantiated by large FMs (e.g., GPT-5), to autonomously search for executable memory designs by open-ended exploration in code space. The search process is formalized around Python abstract classes with two primary interfaces: general_update() and general_retrieve(), encapsulating modular sub-layers for updating and retrieving episodic, semantic, and strategic content from domain-specific interaction logs.

Key aspects:

• Search Space: Turing-complete code representation allows theoretical coverage of arbitrary memory schemas, update logic, and retrieval workflows, including complex database schemas and hierarchical retrieval.
• Open-Ended Exploration: Memory design proposals are sampled from an ever-growing archive, weighted by historic performance (success rates) and visitation, enabling balanced exploitation and exploration rather than myopic greedy optimization. Reflection and debugging loops with code execution validate functional correctness.
• Evaluation Protocol: Systematically decouples the Memory Collection Phase (trajectory sampling for update, no retrieval) from the Deployment Phase (memory-assisted task execution). Static and dynamic deployment modes allow assessment of adaptation to shifts in task distribution, sample scalability, and transferability across foundation models.

Empirical Results: Continual Learning and Adaptation

ALMA is evaluated on ALFWorld, TextWorld, Baba Is AI, and MiniHack—benchmarks encompassing a spectrum of sequential decision-making, text-and-embodied domains, and symbolic rule-based environments. Human-designed baselines span Trajectory Retrieval, ReasoningBank, Dynamic Cheatsheet, and G-Memory. Key findings:

• Success Rates: Learned memory designs consistently outperform all baselines. With GPT-5-nano, ALMA yields a mean success rate gain of 6.2% over no-memory; under GPT-5-mini, ALMA's improvement reaches 12.8%. The generalization of learned designs to more capable FMs is robust, as demonstrated by transferability.
• Domain Specialization: ALMA discovers memory structures attuned to each environment. For spatial domains, designs prioritize object relation graphs, room topology, and trajectory schemas. Symbolic and planning domains favor strategy libraries, plan synthesis modules, and reflex-based subgoal generation.
• Sample Efficiency and Scalability: Learned designs show superior performance scalability with increased task experience during memory collection and are more sample efficient with limited data compared to manual designs.
• Adaptation to Task Distribution Shift: ALMA's dynamic memory updating achieves higher adaptation rates under distributional shift (e.g., ALFWorld: 84.1% success) than baselines, demonstrating its suitability for lifelong agent learning.

Design Analysis and Ablation

Ablation studies reveal the critical role of open-ended exploration. Greedy search, restricted to highest-success-rate designs, yields inferior performance and transfer rates. Archive trees substantiate that moderate-performance stepping stones are necessary for discovering optimal structures, consistent with established principles in novelty-driven search.

Cost efficiency analysis confirms ALMA's superior performance-to-computation trade-off, achieving higher average task success with lower end-to-end FM usage and prompt token size, although optimization of efficiency-performance Pareto is not directly targeted.

Practical and Theoretical Implications

ALMA advances the automation of memory design for agentic systems, freeing practitioners from intensive manual engineering. The approach enables:

• Rapid adaptation of memory paradigms to new domains, including those with complex, non-stationary task distributions.
• Discovery of memory designs not captured by human intuition, supporting broader generalization and continual learning.
• Potential for deployment in specialties such as medicine, finance, and software engineering, where domain specificity and continual adaptation are paramount.

From a theoretical perspective, ALMA contributes to AI-generating algorithm frameworks by meta-learning modular architectural components and potentially lays foundation for systems capable of learning both memory and agentic policies ("learning to continually learn"). Further AI safety considerations incorporate sandboxed code execution and explicit human oversight, acknowledging the challenge that learned components may introduce unintended behaviors.

Limitations and Prospects

Current limitations pertain to offline learning and computational bottlenecks. While ALMA’s framework supports online adaptation, empirical demonstration remains constrained by evaluation cost. Extension to native memory architectures within FMs or full-system meta-learning (joint agent+memory optimization) is a prospective direction.

A systematic inspection pipeline—potentially integrating both AI and human-in-the-loop mechanisms—is essential for scalable safe deployment.

Conclusion

ALMA represents a principled, meta-learning-based step towards continual learning in agentic systems. Learned memory designs provide superior domain specialization, adaptability, sample efficiency, scalability, and cost effectiveness compared to manual baselines, with robust transfer to stronger foundation models. The approach opens paths for automated, scalable, and interpretable memory design, supporting dynamic, lifelong-learning AI agents across diverse domains and task distributions.