Meta-Policy Memory Systems

• Meta-Policy Memory (MPM) is a framework where memory is actively controlled by higher-level policies, evolving from passive storage to dynamic control elements.
• It employs diverse architectures—internal recurrent, external symbolic, episodic, hybrid, and governance middleware—to manage memory update, retrieval, and retention.
• Optimization mechanisms like reflection-distillation, belief-oriented rewards, and hierarchical governance drive MPM's efficiency in long-horizon tasks.

Searching arXiv for papers on Meta-Policy Memory and related memory-policy optimization formulations. Meta-Policy Memory (MPM) denotes a family of architectures in which memory is treated not as passive storage but as an object of higher-level policy control: the system learns or specifies how memory should be written, retrieved, compressed, retained, revised, or ignored in order to improve future decisions. In the recent literature, this idea appears in several closely related forms, including predicate-like reflective rule memories for LLM agents, rubric-grounded reusable guidance, belief-aware memory-policy optimization for long-horizon summarization, policy-driven governance layers for persistent memory, and recurrent or episodic memory as part of a control policy itself (Wu et al., 4 Sep 2025, Li et al., 11 May 2026, Liu et al., 28 May 2026, Kumar et al., 18 Mar 2026, Wang, 27 Dec 2025). A classical antecedent is the finite-memory POMDP controller, where memory is already part of the policy representation through a mapping $$\pi_f : O \times \mathcal{M} \to A \times \mathcal{M}$$ (Lusena et al., 2013).

1. Conceptual scope and terminological boundaries

MPM is not a single standardized formalism. In the agent and reinforcement-learning literature represented here, it refers to systems in which memory changes policy behavior at a higher level than immediate token or action generation. In "Meta-Policy Reflexion," MPM is an externalized set $\mathcal{M}$ of predicate-like corrective rules extracted from hindsight reflections and used through prompt-level guidance plus hard admissibility checks (Wu et al., 4 Sep 2025). In "RubricEM," the analogous object is a rubric bank storing reusable reflections, coupled to a reflection meta-policy that distills judged trajectories into future guidance (Li et al., 11 May 2026). In "Metacognitive Memory Policy Optimization," the higher-level object is the memory-update policy $\pi_{\mathrm{mem}}$, optimized not only for final success but for the clarity of the belief induced by intermediate summaries (Liu et al., 28 May 2026). In "MemArchitect," the meta-policy layer is explicit governance over read, update, decay, contradiction handling, and deletion semantics in an external memory middleware (Kumar et al., 18 Mar 2026).

A useful historical anchor is the free finite-memory policy for finite-horizon POMDPs, defined as $$\pi_f : O \times \mathcal{M} \to A \times \mathcal{M}$$, which maps an observation-memory pair to both an action and the next memory state (Lusena et al., 2013). That formulation already treats memory as a controlled latent state rather than a side channel. The same paper also makes a distinction that remains central to MPM: the amount of memory that is optimal for representing a policy need not equal the amount of memory that is optimal for finding one.

The label is also heterogeneous outside agent memory research. In portfolio management, "MPM" refers to the "Meta Portfolio Method," a supervised selector between HRP and NRP based on historical market features, with only an implicit notion of memory through rolling windows rather than an explicit memory substrate (Kisiel et al., 2021). This suggests that, across fields, the stable core of the term is not a fixed data structure but a policy-over-policies or policy-over-memory-operations perspective.

2. Architectural forms

The literature instantiates MPM through several distinct memory substrates and control mechanisms.

Form	Memory substrate	Representative systems
Internal recurrent memory	Hidden state $h_t$, latent context $z_t$, recurrent reservoirs	(Bao et al., 3 Feb 2025, McKee, 4 Mar 2025)
External symbolic or textual memory	Rule sets, rubric banks, profile memories	(Wu et al., 4 Sep 2025, Li et al., 11 May 2026, Xu et al., 1 May 2026)
Episodic case memory	Persistent store of cases $c=(s,a,r)$ or value-like variants	(Wang, 27 Dec 2025)
Hybrid memory	Trajectory buffers plus distilled predictors and confidence estimators	(Yan et al., 2024)
Governance middleware	Vector-store memories with lifecycle metadata and policy rules	(Kumar et al., 18 Mar 2026)
Memory-policy over recursive summaries	Bounded textual summary $m_t$ updated each turn	(Liu et al., 28 May 2026)

Internal-memory systems treat memory as recurrent controller state. In "Toward Task Generalization via Memory Augmentation in Meta-Reinforcement Learning," an LSTM hidden state summarizes partial observations to infer latent task dynamics and support zero-shot adaptation to unseen actuator failures and payload conditions (Bao et al., 3 Feb 2025). In "Meta-Learning to Explore via Memory Density Feedback," explicit buffers of observations and density values are paired with an Echo State Network that remembers trajectories of novelty feedback, letting action selection depend on both the external memory landscape and recurrent internal state (McKee, 4 Mar 2025).

External-memory systems instead expose memory as an inspectable artifact. "Meta-Policy Reflexion" uses a structured, predicate-like rule memory $\mathcal{M}$ plus hard admissibility checks $a_t \in C(s_t)$, allowing persistent corrective knowledge without weight updates (Wu et al., 4 Sep 2025). "RubricEM" stores accepted reflections in an agent rubric bank indexed by embeddings and exact hashes, while a shared-backbone reflection policy learns to write reusable rubric-grounded guidance (Li et al., 11 May 2026). "MemCoE" uses a structured profile memory updated under an explicit global guideline $\mathcal{M}$0, separating memory governance from concrete writes (Xu et al., 1 May 2026).

A third pattern makes the memory operation itself the optimization target. "Metacognitive Memory Policy Optimization" defines a bounded summary $\mathcal{M}$1 produced recursively by a memory policy and optimizes that policy through belief-oriented process rewards rather than outcome-only reinforcement (Liu et al., 28 May 2026). "MemArchitect" extends this logic to persistent agents by placing policy rules over the memory lifecycle—decay, relevance veto, consolidation, privacy, and planned deletion cascades—outside model weights (Kumar et al., 18 Mar 2026).

3. Formal models and control abstractions

Several papers make MPM technically precise by re-expressing memory as part of the control problem. The classical finite-memory POMDP controller already does this by augmenting policy input with memory state and producing both action and next-memory outputs: $\mathcal{M}$2 (Lusena et al., 2013). That formulation implies an augmented hidden state $\mathcal{M}$3, and the paper’s main interpretive result is that extra memory can improve optimization even after representational sufficiency has saturated.

Recent LLM-agent work generalizes this idea to textual or episodic memory. In MMPO, the environment is a POMDP and the agent no longer conditions on full history $\mathcal{M}$4, but on a compressed memory $\mathcal{M}$5, producing the summary-induced belief $\mathcal{M}$6. The ideal memory objective is to maximize $\mathcal{M}$7, equivalently to minimize $\mathcal{M}$8, so that recursive summarization preserves latent task-state identifiability (Liu et al., 28 May 2026). This makes memory quality a belief-state question rather than merely a storage-efficiency question.

"Memento-II" gives the most explicit MPM formalization in control-theoretic terms. Its Stateful Reflective Decision Process is $\mathcal{M}$9, where the memory $\pi_{\mathrm{mem}}$0 is a multiset of cases $\pi_{\mathrm{mem}}$1, retrieval samples $\pi_{\mathrm{mem}}$2, and the frozen LLM produces the environment action via $\pi_{\mathrm{mem}}$3 (Wang, 27 Dec 2025). The induced composite policy is

$\pi_{\mathrm{mem}}$4

By augmenting state to $\pi_{\mathrm{mem}}$5, the paper constructs an equivalent reflected MDP in which retrieval actions are the control variable, memory write corresponds to policy-evaluation data collection, and memory read corresponds to policy improvement. This is a direct formal statement that, in an MPM system, policy may be externalized into the interaction between memory contents and retrieval.

A related but narrower abstraction appears in AdaMemento, where a successful-trajectory buffer $\pi_{\mathrm{mem}}$6 and failed-experience buffer $\pi_{\mathrm{mem}}$7 are transformed into a prediction network and reflection network. The resulting ensemble policy switches between memory-supported behavior and the base exploration policy according to a confidence threshold $\pi_{\mathrm{mem}}$8, yielding a policy-level arbitration rule rather than simple replay (Yan et al., 2024).

4. Optimization mechanisms

The central methodological question in MPM is how higher-level memory control is learned. The surveyed work answers this in several ways.

Reflection-distillation methods convert experience into reusable memory artifacts. In MPR, failed episodes are retrospectively analyzed and converted into corrective rules through $\pi_{\mathrm{mem}}$9; at inference time, a relevant subset $$\pi_f : O \times \mathcal{M} \to A \times \mathcal{M}$$0 conditions the frozen LLM, and invalid outputs are blocked by hard admissibility checks (Wu et al., 4 Sep 2025). RubricEM goes further by making judged trajectories reusable across episodes: stage-structured rollouts are reflected into rubric-like guidance, the best accepted reflection is written to a rubric bank, and a shared-backbone reflection branch is updated by policy gradient on reflection utility rather than by supervised text targets (Li et al., 11 May 2026).

Belief-oriented memory optimization supplies dense intermediate supervision. MMPO introduces Belief Entropy, $$\pi_f : O \times \mathcal{M} \to A \times \mathcal{M}$$1, measured from an anchor question about task progress and missing information, and uses it inside a dense sub-trajectory reward $$\pi_f : O \times \mathcal{M} \to A \times \mathcal{M}$$2 (Liu et al., 28 May 2026). The key technical move is to supervise memory quality at each turn rather than only at the terminal outcome, thereby improving credit assignment for long-horizon summarization.

Hierarchical optimization separates memory governance from memory editing. MemCoE first induces a global memory guideline $$\pi_f : O \times \mathcal{M} \to A \times \mathcal{M}$$3 through Memory Guideline Induction, using textual gradients $$\pi_f : O \times \mathcal{M} \to A \times \mathcal{M}$$4, batch aggregation $$\pi_f : O \times \mathcal{M} \to A \times \mathcal{M}$$5, and prompt optimization $$\pi_f : O \times \mathcal{M} \to A \times \mathcal{M}$$6 (Xu et al., 1 May 2026). It then fixes $$\pi_f : O \times \mathcal{M} \to A \times \mathcal{M}$$7 and trains the memory evolution policy with Guideline-Aligned Memory Policy Optimization, using a combined reward

$$\pi_f : O \times \mathcal{M} \to A \times \mathcal{M}$$8

so that the learned updater is rewarded both for final correctness and for following the induced schema.

Governance-oriented approaches encode the meta-policy directly as rules and scoring functions. MemArchitect uses explicit formulas such as the adaptive retrieval score $$\pi_f : O \times \mathcal{M} \to A \times \mathcal{M}$$9, the FSRS retrievability rule $h_t$0, and planned conflict arbitration $h_t$1 to decide which memories should survive, compete for context, or be suppressed (Kumar et al., 18 Mar 2026). This suggests a variant of MPM in which the meta-policy is symbolic and auditable rather than learned end-to-end.

5. Empirical behavior and application domains

Empirical results support MPM across several task families, although the benchmarks are not directly commensurate and should be read within each paper’s setup. In MMPO, memory-specific supervision improves long-horizon LLM agents on RULER-HotpotQA, MEM1-QA, and WebShop. At 1.75M-token context on RULER-HotpotQA, MMPO-14B reaches 79.77 versus 78.91 for RL-MemAgent, and at 3.5M tokens 76.47 versus 71.09; the paper also reports average gains from 224K to 3.5M of $h_t$2 for 14B and $h_t$3 for 7B, plus a Belief Entropy reduction that correlates with final accuracy at $h_t$4 (Liu et al., 28 May 2026). These results support the claim that memory-policy quality, rather than context length alone, determines long-horizon robustness.

For reusable reflective memory, MPR reports substantial gains over Reflexion in AlfWorld. On the 60-task training set, MPR rises from 83.9 in round 1 to 100.0 by round 3, whereas Reflexion reaches 87.2 in round 3; on 74 held-out tasks, MPR scores 87.8 against Reflexion’s 86.9, and MPR plus hard admissibility checks reaches 91.4 (Wu et al., 4 Sep 2025). RubricEM, in a different setting, reports an average score of 55.5 across HealthBench, ResearchQA, DRB, and ResearchRubrics, compared with 53.6 for DR Tulu-8B RL, and attributes additional gains to reusable-experience learning through the reflection meta-policy and rubric bank (Li et al., 11 May 2026).

In personalized dialogue memory, MemCoE posts an overall score of 52.02 across PersonaMem, PrefEval, and PersonaBench, compared with 45.00 for MemAgent and 44.19 for Mem-$h_t$5; on PrefEval explicit preference it reaches 81.30, and on implicit preference 69.90 (Xu et al., 1 May 2026). Its retention analysis is especially MPM-relevant: on the explicit retention test, both methods begin at 100%, but by round 10 MemAgent falls to roughly 51% while the full system remains around 74, suggesting that learned memory governance materially slows preference forgetting.

Memory-augmented RL shows similar patterns outside text. In task-generalization for legged locomotion, the recurrent Memory-Aug policy matches Memory-Rand on ID and OOD tasks without collecting real OOD interactions during training, and transfers to real ANYmal hardware under both in-distribution and out-of-distribution joint failures (Bao et al., 3 Feb 2025). In exploration, memory-density feedback yields top coverage of 100% in the random maze and 99% in the continual maze for the combined model, versus weaker observation-only baselines, showing that meta-learned use of memory-derived feedback can dominate fixed novelty heuristics in variable environments (McKee, 4 Mar 2025). AdaMemento likewise reports more than $h_t$6 gain in total rewards over PPO on Montezuma’s Revenge and nearly $h_t$7 in Swimmer by combining advantageous trajectory memory, negative reflection memory, and confidence-gated arbitration (Yan et al., 2024).

6. Limitations, controversies, and open directions

The main limitation of MPM as a research category is heterogeneity. Some systems expose explicit memory banks and rule stores, others rely on recurrent hidden state, and some are better described as meta-policies with only implicit state. "Learning to Deliberate" is illustrative: its Meta-Policy Deliberation Framework learns over high-level actions such as Persist, Refine, and Concede, and conditions those decisions on structured meta-cognitive state, but it does not introduce a persistent external memory store (Yang et al., 4 Sep 2025). This suggests that not every meta-policy is an MPM system in the strict sense, even when it addresses meta-cognitive control.

A second tension is between expressivity and optimization. The classical finite-memory POMDP results show that adding memory increases the size of the search space, yet can make optimal-valued policies easier to find; more memory than is strictly necessary may improve convergence probability, even while slowing computation (Lusena et al., 2013). This trade-off reappears in modern systems as context pollution, memory explosion, or over-conservative averaging across hidden task variants.

Proxy reliability is another recurring issue. MMPO explicitly notes that Belief Entropy is only a proxy for $h_t$8 and may reward confident but wrong summaries if used alone (Liu et al., 28 May 2026). RubricEM’s reusable reflections depend on judge quality and on a single accepted reflection per query, which may discard diversity and concentrate errors in the rubric bank (Li et al., 11 May 2026). MPR notes that extracted rules may contain redundancies or inconsistencies and calls for verification, pruning, and composition mechanisms (Wu et al., 4 Sep 2025). MemArchitect reports mixed results against MemOS because active decay can over-prune one-shot temporal details, turning unchecked hallucination into a tunable over-pruning failure mode (Kumar et al., 18 Mar 2026).

Several papers also point to missing infrastructure for mature MPM systems. MemArchitect still treats conflict resolution, toxic-memory filtering, and right-to-be-forgotten cascades as planned rather than fully realized components (Kumar et al., 18 Mar 2026). MemCoE identifies scorer reliability, compounding update errors, and its fixed-guideline design as open limitations (Xu et al., 1 May 2026). AdaMemento does not maintain an explicit structured policy bank, and RubricEM does not study long-term bank growth, interference, or compositional reuse across heterogeneous domains (Yan et al., 2024, Li et al., 11 May 2026). A plausible implication is that the next stage of MPM research will require more explicit memory provenance, multi-objective governance, and formal policy composition over write, read, update, and delete operations.