MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory

Abstract: The hallmark of human intelligence is the ability to master new skills through Constructive Episodic Simulation-retrieving past experiences to synthesize solutions for novel tasks. While LLMs possess strong reasoning capabilities, they struggle to emulate this self-evolution: fine-tuning is computationally expensive and prone to catastrophic forgetting, while existing memory-based methods rely on passive semantic matching that often retrieves noise. To address these challenges, we propose MemRL, a framework that enables agents to self-evolve via non-parametric reinforcement learning on episodic memory. MemRL explicitly separates the stable reasoning of a frozen LLM from the plastic, evolving memory. Unlike traditional methods, MemRL employs a Two-Phase Retrieval mechanism that filters candidates by semantic relevance and then selects them based on learned Q-values (utility). These utilities are continuously refined via environmental feedback in an trial-and-error manner, allowing the agent to distinguish high-value strategies from similar noise. Extensive experiments on HLE, BigCodeBench, ALFWorld, and Lifelong Agent Bench demonstrate that MemRL significantly outperforms state-of-the-art baselines. Our analysis experiments confirm that MemRL effectively reconciles the stability-plasticity dilemma, enabling continuous runtime improvement without weight updates.

• The paper presents a framework that uses non-parametric reinforcement learning to update episodic memory via a value-based retrieval mechanism.
• It demonstrates significant performance gains over baselines in tasks such as code generation and embodied navigation.
• The approach effectively prevents catastrophic forgetting while enabling safe, continual adaptation without modifying the core LLM.

MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory

Motivation and Background

LLMs have demonstrated strong few-shot reasoning and in-context learning capabilities, yet they lack the distinctive human ability to perform constructive episodic simulation: flexibly synthesizing solutions for novel tasks by leveraging and adapting past experiences. Standard approaches—such as fine-tuning or parameter-efficient continual learning—are often computationally expensive and susceptible to catastrophic forgetting. Non-parametric methods, most notably Retrieval-Augmented Generation (RAG), passively retrieve information based on semantic similarity without any consideration of the historical utility of retrieved knowledge, causing persistent issues when encountering similar but non-transferable or noisy memories.

This paper introduces MEMRL, a principled framework that enables self-evolving, memory-augmented LLM agents through non-parametric reinforcement learning (RL) directly on episodic memory. The central tenet is a strict decoupling of the LLM’s frozen, stable reasoning function from a plastic, utility-optimized episodic memory system. MEMRL formalizes memory-augmented decision-making as a value-based memory retrieval and updates utility estimates online using environmental feedback—thereby reconciling the stability-plasticity dilemma in agentic continuous learning.

MEMRL: Framework and Operational Principle

Memory Structure: Intent-Experience-Utility Triplet

The episodic memory bank in MEMRL is architected as triplets $(z, e, Q)$, where $z$ denotes the intent embedding (semantic representation of the user query), $e$ encodes the experience (solution trace or trajectory), and $Q$ captures the learned utility (expected return of retrieving $e$ under similar intents). This triplet structure enables each memory item to be contextually and functionally indexed.

Two-Phase Retrieval: Semantic and Utility-Driven

Retrieval proceeds in two distinct phases:

1. Phase A (Semantic Recall): For the current intent $s$, a candidate pool $C(s)$ is constructed using cosine similarity above a threshold.
2. Phase B (Value-Aware Selection): Candidates from $C(s)$ are scored via a weighted combination of semantic similarity and normalized Q-value:

$$\text{score}(s, z, e) = (1-\lambda) \cdot \text{sim}(s,z) + \lambda \cdot Q(z,e)$$

where $\lambda$ modulates the trade-off between immediate semantic fit and empirically-learned utility. Top-$k$ memories are composed into the retrieval context.

Non-Parametric RL: Utility Estimation and Memory Update

Upon completion of an action (via the frozen LLM), the agent receives a scalar reward. For memories used in the context, Q-values are updated using a temporal-difference rule:

$Q \leftarrow Q + \alpha (r - Q)$

This non-parametric, Monte Carlo-style update enforces convergence to expected utility under stationary conditions without modifying the LLM weights or other parametric components.

Theoretical Analysis: Stability, Convergence, and Catastrophic Forgetting

• Convergence: The exponential moving average update of Q converges in expectation to the true mean reward for each (intent, experience) pair, as shown through direct analysis of the error dynamics.
• Stability: Under stationary task and frozen model assumptions, the variance of Q-values remains bounded even in the presence of reward noise, avoiding unbounded oscillations in memory utility estimation.
• Global Stability: By modeling the interaction of retrieval policy and value estimation as a Generalized Expectation-Maximization process, the framework ensures that policy and memory utility estimates jointly converge to stationary points. This effectively prevents catastrophic forgetting, observed empirically as a lower regression rate from previously solved to failed tasks compared to other baselines.

Empirical Evaluation and Ablation

Benchmarks and Baselines

MEMRL was evaluated on diverse and challenging benchmarks:

• BigCodeBench (code generation)
• ALFWorld (embodied navigation and multi-step reasoning)
• Lifelong Agent Bench (OS and DB interaction)
• Humanity's Last Exam (HLE, complex multidisciplinary problem solving)

Baselines included standard RAG, reflection-augmented memory, procedural memory (MemP), and retrieval-based critiques (Self-RAG).

Key Results

MEMRL dominates all baselines across domains in both runtime learning and transfer scenarios. On ALFWorld, for example, MEMRL posts a last-epoch accuracy of 0.507, providing a 56% relative improvement over MemP and an 82% gain relative to agents without memory. Crucially, in sequential, exploration-heavy environments, MEMRL's value-aware retrieval acts as a trajectory verifier, with strong correlation ($r=0.861$) between Q-estimates and actual task success rates. Even in low-similarity environments (HLE), MEMRL demonstrates significant performance gains by runtime memorization of specific solutions. Ablations confirm that optimal performance arises from a balanced trade-off between semantic matching and utility weighting ($\lambda=0.5$); memory overcapacity or ignoring semantic similarity impairs robustness and stability.

Stability and Forgetting

MEMRL achieves the lowest empirical forgetting rate (fraction of tasks regressing from success to failure), demonstrating its theoretical guarantees in practice. Removing core design elements (Q-value normalization, similarity gate) leads to instability and increased forgetting.

Implications and Future Directions

MEMRL operationalizes value-based credit assignment within the retrieval process, transforming episodic memory from a passive, similarity-based knowledge base into an active, utility-aware substrate for agentic RL. The approach supports:

• Continual, safe adaptation: Agents improve at deployment without risk to the integrity of their highly-tuned backbone via non-invasive memory updates.
• Trajectory-verifying behavior: Particularly in sequential or multi-step domains, MEMRL's utility mechanism filters brittle or near-miss strategies, promoting transfer of robust procedures and corrective heuristics over superficially similar failures.
• Task-structure awareness: Gains scale with structural repetitiveness, but MEMRL is also effective in low-similarity domains due to its capacity for on-the-fly memorization.
• Generalizability: Learned utility mappings foster improved transfer to held-out tasks and domains without additional parameter updates.

Future research may explore hierarchical or hierarchical memory architectures, more sophisticated utility models (e.g., context-sensitive or compositional Q-functions), and integration with broader metacognitive and planning components. There is potential for MEMRL to underpin safe and interpretable test-time learning regimes and robustify open-ended autonomous agents.

Conclusion

MEMRL establishes a robust, theoretically-sound paradigm for self-evolving LLM agents by combining a utility-driven episodic memory system with a frozen, stable LLM core. It resolves core challenges in runtime continual learning and demonstrates marked, quantifiable gains over state-of-the-art baselines across a spectrum of complex tasks. The framework lays a foundation for further advances in non-parametric agentic learning, bridging RL and memory-augmented reasoning at scale (2601.03192).