Cross-Episode Reflection Memory

• Cross-Episode Reflection Memory is an architectural paradigm that accumulates, retrieves, and reflects on episode-level experiences to inform adaptive learning in agents.
• It employs retention, retrieval, and reflection mechanisms using techniques such as vector embeddings, graph traversal, and rule-based criteria.
• Applications in web navigation, conversational agents, and meta-RL showcase enhanced sample efficiency, adaptability, and decision quality.

Cross-Episode Reflection Memory is an architectural and algorithmic paradigm for integrating long-term, feedback-driven adaptation across episodes or sessions in learning agents—especially those based on LLMs and reinforcement learning (RL). It contrasts with isolated, per-episode learning or stateless in-context learning by accumulating, retrieving, and leveraging knowledge, self-critiques, summaries, or symbolic rules from prior episodes to inform future decision-making, planning, exploration, or reasoning. Approaches span explicit key–value stores, vector databases, structured graphs, and purely attention-based methods, unified by the inclusion of explicit mechanisms for retention, retrieval, and reflection that cross episode boundaries.

1. Fundamental Concepts and Design Principles

Cross-episode reflection memory systems are architected around three essential mechanisms:

Retention: Continuous or event-driven storage of episode-level experiences, summaries, reflections, or structured critiques. Memories may capture raw trajectories, lossy summaries (e.g., key steps and failure modes), strategies, meta-level critiques, or distilled rules.

Retrieval: Query-time selection of relevant past experiences using embedding-based similarity, temporal recency, predicate matching, or graph traversal. Relevance scoring may combine semantic, lexical, structural, and temporal criteria (e.g., cosine similarity of task embeddings in (Liu et al., 17 Nov 2025), entity- and temporal-aware graph fusion in (Latimer et al., 14 Dec 2025)).

Reflection: Incorporation of retrieved memories in current decision-making, typically through prompt augmentation, intervention via “system messages,” dynamic reweighting of candidate actions, or explicit update of latent beliefs and policy state. Some systems instantiate “reflection” as an explicit, LLM-mediated read–update–write cycle (Latimer et al., 14 Dec 2025, Wang, 27 Dec 2025).

Several systems further support self-evolution and adaptation without gradient-based updates, relying solely on the continual accrual and strategic deployment of cross-episode memory (Liu et al., 17 Nov 2025, Wang, 27 Dec 2025).

2. Memory Representations and Storage Architectures

A spectrum of memory representations has emerged:

• Summarized trajectory memories: Systems like WebCoach employ a WebCondenser to compress observation-action-reward traces into 3–5 sentence summaries accompanied by vector embeddings and analytic tags (Liu et al., 17 Nov 2025).
• Structured multi-network graphs: Hindsight organizes memory into world facts, agent experiences, synthesized entity summaries, and evolving subjective opinions, all linked within a temporal, entity-aware graph (Latimer et al., 14 Dec 2025).
• Rule-based predicate memory: Meta-Policy Reflexion accumulates symbolic rules of the form (predicate φ, action a, weight w), offering both soft and hard guidance (Wu et al., 4 Sep 2025).
• Exemplar and feedback factories: Exemplar-Guided Reflection Memory maintains prioritized banks of past exemplars (detailed input-label-CoT traces) and textual feedback items, both scored and filtered for continual relevance (Yan et al., 2024).
• Key–value memory and episodic recall: Systems such as Episodic LSTM (Ritter et al., 2018) and Memento-II’s SRDP (Wang, 27 Dec 2025) store explicit (context, solution, reward/outcome) tuples, permitting differentiable or kernel-based retrieval.

Physical storage typically leverages vector databases (FAISS with HNSW for efficient approximate nearest neighbor search (Liu et al., 17 Nov 2025, Wu et al., 26 Aug 2025)), explicit graph databases (Latimer et al., 14 Dec 2025), or transformer architecture’s intrinsic context window (Lin et al., 3 Feb 2026). Scalability is addressed by compression (summarization, clustering), priority-based eviction, or structured deduplication.

3. Retrieval, Reasoning, and Reflection Protocols

Memory retrieval and reflection protocols are central to cross-episode leverage:

• Similarity and recency–based querying: Agents retrieve top-$K$ episodic summaries or reflections based on current context embedding, often integrating recency penalties (e.g., $$Score(m) = \alpha \cdot Sim(e_t, e_m) + \beta \cdot Recency(m)$$; default $\beta = 0$ in (Liu et al., 17 Nov 2025)).
• Reflection-mediated action selection: Retrieved content is injected into system prompts as advice or lessons, or used to trigger high-level interventions when imminent failure is detected (failure risk, coverage issues, dead ends) (Liu et al., 17 Nov 2025, Azam et al., 2 Jun 2025).
• Reinforcement and belief updating: In multi-network graph systems, new evidence is used to reinforce, weaken, or contradict subjective opinions, supporting dynamic, explainable belief formation (Latimer et al., 14 Dec 2025).
• Rule admissibility and action constraints: Hard admissibility checks enforce that only actions not forbidden by relevant retrieved rules are executed (Wu et al., 4 Sep 2025).
• Meta-optimization integration: Reflection memory can bias prompt gradients in meta-optimization (e.g., retrieved “mistake notebook” records in REMO shape TextGrad-style update directions and optimizer prompts (Wu et al., 26 Aug 2025)).
• In-context learning via full context concatenation: Some systems store entire cross-episode transcripts in the context window, relying on native transformer attention to retrieve and utilize prior experience without an explicit memory module (Lin et al., 3 Feb 2026).

Reflection—beyond mere retrieval—is operationalized via LLM-driven evaluation and update of beliefs, policies, and meta-prompts. This may be implemented as a three-step retain–recall–reflect cycle (Latimer et al., 14 Dec 2025), an explicit SRDP read–write–interact loop (Wang, 27 Dec 2025), or a prompt-augmented planning/inference pipeline (Azam et al., 2 Jun 2025, Yan et al., 2024).

4. Formal Frameworks and Theoretical Foundations

Theoretical accounts formalize cross-episode reflection memory within Markov Decision Processes (MDPs) augmented by external memory and reflection operators:

• Stateful Reflective Decision Process (SRDP): The agent augments the classical MDP with explicit episodic memory, a retrieval policy, and an LLM generative kernel. Reflection is the joint process of reading from memory (policy improvement) and writing new cases (policy evaluation), inducing an augmented MDP over $(s, M)$ (Wang, 27 Dec 2025).
• Entropy-regularized and Parzen-KL policy iteration: KL-regularized Bellman operators on augmented MDPs admit contraction-mapping guarantees and enable Gibbs-form retrieval policies, converging (as memory grows to full coverage) to the optimal policy, given local consistency between LLM kernel and true policy (Wang, 27 Dec 2025).
• Meta-RL objectives over cross-episode returns: Agents optimize $$\displaystyle J(\theta) = \mathbb{E}_{T^{(0:N-1)} \sim \pi_\theta}\Bigl[\sum_{n=0}^{N-1}G^{(n)}\Bigr]$$ with $G^{(n)}$ defined as a cross-episode discounted sum of returns ($\gamma_{traj}$), crediting exploration and reflective learning over multiple episodes (Jiang et al., 18 Dec 2025, Lin et al., 3 Feb 2026).
• Graph-structured evidence vs. inference: Hindsight’s separation of world, experience, observation, and opinion networks clarifies epistemic provenance and mechanisms for belief reinforcement (Latimer et al., 14 Dec 2025).

Empirical validation demonstrates not only increased sample efficiency and solution quality, but also the asymptotic convergence of the effective policy to optimality with sufficiently rich and well-retrieved episodic memory.

5. Applications, Empirical Outcomes, and Architectural Diversity

Cross-episode reflection memory has been systematically applied across several domains and agent architectures:

• Web navigation and browser agents: WebCoach yields success rate gains of up to +14.1 points (from 47.3% to 61.4% with Skywork-38B), reduced steps per episode, and strong transfer to complex sites. Self-generated memory outperforms seeded memory from foreign demonstrations, with little or no benefit for models below a minimum backbone scale (Liu et al., 17 Nov 2025). ReAP yields comparable step-efficiency gains and highlights the particular value of negative reflections (Azam et al., 2 Jun 2025).
• Conversational agents: Hindsight achieves +44.6 points improvement (from 39% to 83.6%) on LongMemEval and leads on multi-session and temporal reasoning, surpassing strong full-context baselines (Latimer et al., 14 Dec 2025). EMMA demonstrates human-level consistency and coherence in multi-session, multi-partner dialogues, yielding 97–99% pass rates on summarization, linking, and tagging metrics (Jang et al., 2024).
• Meta-RL and online adaptation: LaMer’s Meta-RL with reflection yields 11–19% absolute gains across long-horizon environments, with ablations showing reflection is the primary source of adaptation gains. Ablations in ORBIT illustrate that multi-episode meta-RL enables open-source models to match or surpass GPT-5.2 in previously unseen environments, with stepwise success rates (Maze, Mastermind) rising by 30+ points over the base (Jiang et al., 18 Dec 2025, Lin et al., 3 Feb 2026).
• Prompt optimization and continual learning: REMO’s “mistake notebook” closes validation–test generalization gaps and reduces overfitting in prompt optimization for mathematical reasoning (Wu et al., 26 Aug 2025). ERM delivers up to +10 F1 improvement on LIAR and halves optimization steps compared to non-reflective baselines (Yan et al., 2024).
• Symbolic policy abstraction and safety: Meta-Policy Reflexion provides robust, interpretable action constraints and memory-guided adaptation, converging to perfect accuracy on training (AlfWorld) and outperforming per-episode reflection on held-out tasks, especially when hard admissibility checks are employed (Wu et al., 4 Sep 2025).
• Theoretical guarantees and convergence: Memento-II establishes the asymptotic optimality of reflective memory-driven learning, given memory coverage, local LLM kernel consistency, and proper retrieval policy regularization (Wang, 27 Dec 2025).

6. Limitations, Open Problems, and Future Directions

Several challenges and open problems persist:

• Memory scaling and management: As episodic or feedback memory grows, retrieval and storage demands increase. Pruning strategies, clustering, learned key–value indices, and hierarchical summarization are common but still present open trade-offs (Yan et al., 2024, Liu et al., 17 Nov 2025).
• Cross-domain, cross-task transfer: Most current systems construct separate memory banks per task or domain; the unification and robust conditional retrieval across heterogeneous tasks remains an open direction (Yan et al., 2024, Latimer et al., 14 Dec 2025).
• Quality and noise in memory: All approaches depend on the quality of stored episodes, reflections, or rules. Inaccurate reflections, over-specific predicates, or “garbage-in, garbage-out” failures can degrade or even damage agent performance (Azam et al., 2 Jun 2025, Wu et al., 4 Sep 2025).
• Cognitive threshold phenomena: Several studies report that cross-episode memory is only beneficial when the base model exceeds a certain scale (“cognitive threshold”), below which memory signals cannot be reliably exploited (Liu et al., 17 Nov 2025).
• Theoretical–practical alignment: While convergence guarantees exist in the asymptotic regime (Wang, 27 Dec 2025), practical agents face finite context budgets, partial coverage, and nonstationary policies, requiring further study to understand real-world convergence dynamics.
• Human-in-the-loop augmentation: Most systems rely fully on LLM-generated memory; hybrid schemes incorporating human curation or ranking could mitigate degenerate memory accumulation (Yan et al., 2024).
• Failure mode analysis: Predicate-overfitting, negative transfer on already solved tasks, and residual contradiction formation in multi-actor conversational memory highlight the need for future architectural refinements (Wu et al., 4 Sep 2025, Jang et al., 2024).

7. Comparative Analysis and Innovations

Cross-episode reflection memory differentiates itself from Retrieval-Augmented Generation (RAG) and naive context concatenation by introducing structured retention, principled sampling or retrieval, and meta-reasoning over the content and structure of memory:

• Separation of evidence and inference: Systems like Hindsight explicitly differentiate factual evidence, subjective beliefs, and synthesized observations, supporting both robust retrieval and traceable belief revision (Latimer et al., 14 Dec 2025).
• Integration with RL and policy iteration: Memory serves as both a substrate for retrieving value-aligned experience and an external mechanism for continual, on-policy adaptation (Jiang et al., 18 Dec 2025, Lin et al., 3 Feb 2026).
• Explicit reasoning over memory graphs: Structured networks of world facts, opinions, and experiences support multi-hop reasoning and explainability.
• Symbolic rule induction and enforcement: Moving beyond raw experience tapes, MPR and related systems externalize symbolic polices, yielding interpretable and reusable domain abstractions (Wu et al., 4 Sep 2025).
• Meta-controllers and self-adaptive optimizers: Higher-order learning is instantiated through LLM-based meta-controllers summarizing and acting upon epoch-level experience to refine parameter-free optimization policies (Wu et al., 26 Aug 2025).
• Empirical superiority across agent types: In well-controlled experiments, reflection memory consistently delivers gains that cannot be replicated by context scaling alone or naive prompt engineering, especially for sample efficiency, transfer to held-out or complex domains, and temporal/long-horizon reasoning (Liu et al., 17 Nov 2025, Latimer et al., 14 Dec 2025, Lin et al., 3 Feb 2026).

These collective results position cross-episode reflection memory as a central mechanism for robust, continual, and adaptable intelligence in LLM-based and reinforcement learning agents.