Memory-Driven Self-Improvement

• Memory-Driven Self-Improvement Framework is an agent architecture that externalizes adaptation to a dynamic, optimizable memory substrate rather than traditional weight updates.
• It employs explicit memory management techniques—such as retrieval optimization, episodic storage, and self-reflective updates—to enhance adaptive performance across diverse tasks.
• Empirical evidence shows that these systems improve task outcomes in areas like code generation, web navigation, and multi-agent coordination by leveraging utility-based memory updates.

A memory-driven self-improvement framework is a class of agent architectures in which adaptation is shifted from opaque weight updates toward explicit, evolvable memory. In this formulation, a LLM may remain frozen or only partially updated, while external memory is written, retrieved, reweighted, consolidated, pruned, or structurally redesigned during deployment. Recent systems instantiate this idea through episodic memories with learned utilities, self-reflective meta-memory, file-system memory actions, and self-optimizing memory strategies, all of which treat memory as the primary locus of post-deployment plasticity rather than a passive appendage to prompting (Zhang et al., 6 Jan 2026, Ye et al., 11 Feb 2026, Wu et al., 1 Jul 2026, Yang et al., 4 Jul 2026).

1. Conceptual scope

The central premise is that long-horizon adaptation can be achieved by preserving a stable reasoning substrate while externalizing change into memory. In MemRL, the stable component is a frozen LLM and the plastic component is episodic memory with learned utility values; in UMEM, memory is described as non-parametric trainable parameters for LLM-based agents; in AutoMem, memory management itself is treated as a cognitive skill; and in SelfMem, storage, retrieval, and summarization cease to be fixed pipeline stages and become objects of agent-controlled optimization (Zhang et al., 6 Jan 2026, Ye et al., 11 Feb 2026, Wu et al., 1 Jul 2026, Yang et al., 4 Jul 2026).

This framing distinguishes memory-driven self-improvement from several adjacent paradigms. It is not equivalent to passive RAG, because many of these systems optimize which memories are useful rather than merely retrieving semantically similar text. It is not reducible to continual fine-tuning, because several frameworks explicitly avoid weight updates in order to mitigate catastrophic forgetting, preserve general reasoning, or maintain auditability. It is also not simply a consequence of longer context windows: O-Mem shows that semantically irrelevant but critical user facts can dominate correct behavior, and DecentMem argues that centralized memory can impose communication overhead, raise privacy concerns, and collapse agent diversity in multi-agent settings (Hao et al., 21 May 2026, Wang et al., 17 Nov 2025).

A recurring misconception is that memory in these systems is merely archival. In the recent literature, memory increasingly functions as decision policy, retrieval controller, strategic prior, failure detector, or governance artifact. This suggests that the framework is best understood not as “LLM plus notes,” but as a distributed control architecture in which the memory substrate carries a substantial fraction of the system’s adaptive intelligence.

2. Formal architectures and control loops

One of the most explicit formalizations appears in MemRL, which casts the agent as a Memory-Based Markov Decision Process with tuple $(S, A, P, R, \gamma, M)$, where the memory consists of triplets storing an intent embedding, an experience summary, and a learned utility $Q_i$. The joint policy factorizes retrieval and generation as

$$T(y_t \mid s_t, M_t)=\sum_{m\in M_t} p(m\mid s_t,M_t)\cdot P_{\mathrm{LLM}}(y_t\mid s_t,m).$$

In this view, optimization is transferred from model weights to the retrieval policy over memory items, and environmental feedback updates the value of those memories at runtime (Zhang et al., 6 Jan 2026).

A broader abstraction is given by Evo-Memory, which represents a memory-augmented agent as a tuple $(F,U,R,C)$: a base LLM $F$, memory update $U$, retrieval $R$, and contextualization $C$. For a task stream $S=\{x_t\}_{t=1}^T$, the loop is retrieve $\rightarrow$ contextualize $Q_i$0 predict or act $Q_i$1 evolve memory. This formulation makes test-time evolution explicit and provides a common interface for heterogeneous memory modules such as semantic stores, workflow memories, experience replay buffers, and refine-based systems (Wei et al., 25 Nov 2025).

MemEvolve extends the formal picture by decomposing memory architecture itself into four modules,

$Q_i$2

corresponding to encode, store, retrieve, and manage. Its inner loop updates experiential knowledge under a fixed candidate architecture, while its outer loop meta-evolves the architecture through selection and redesign over multiple candidates. By contrast, MetaMem uses a non-RL update pipeline in which reflections over sampled responses generate edit actions, filtered actions produce a consolidated update, and the current meta-memory state is rewritten via $Q_i$3 (Zhang et al., 21 Dec 2025, Xin et al., 27 Jan 2026).

Across these formulations, the shared control structure is cyclical rather than one-shot. Query-time behavior depends on current memory; task outcomes generate feedback; feedback triggers reflection, valuation, or repair; and the resulting memory state changes subsequent behavior. The principal design differences concern where credit is assigned, what is stored, and how retrieval is converted into action.

3. Memory representations and lifecycle management

The representational choices are highly diverse. MemRL stores each past episode as an Intent–Experience–Utility triplet $Q_i$4, where $Q_i$5 is an intent embedding, $Q_i$6 is a compact LLM-summarized trajectory, and $Q_i$7 estimates expected return under reuse (Zhang et al., 6 Jan 2026). REMO instead uses a “mistake notebook” entry

$Q_i$8

designed to preserve optimization-relevant failures and corrective reflections across runs (Wu et al., 26 Aug 2025). MetaMem represents memory as explicit meta-memory units $Q_i$9, described as symbolic rules, templates, or heuristics, and edits them with ADD, DEL, and MOD operations rather than treating memory as a flat vector store (Xin et al., 27 Jan 2026).

Other systems elevate higher-level abstractions. Trajectory-Informed Memory Generation converts trajectories into strategy tips, recovery tips, and optimization tips, each with triggers, ordered steps, priority, context metadata, and provenance to source trajectory IDs and subtask ranges (Fang et al., 11 Mar 2026). ASG-SI promotes successful trajectories into auditable skills with explicit inputs, outputs, preconditions, postconditions, verifier reports, and placement in a directed skill graph, so that memory artifacts become reusable and replayable capabilities rather than unstructured traces (Huang et al., 28 Dec 2025). O-Mem organizes long-term personalized memory into persona attributes $$T(y_t \mid s_t, M_t)=\sum_{m\in M_t} p(m\mid s_t,M_t)\cdot P_{\mathrm{LLM}}(y_t\mid s_t,m).$$0, persona fact events $$T(y_t \mid s_t, M_t)=\sum_{m\in M_t} p(m\mid s_t,M_t)\cdot P_{\mathrm{LLM}}(y_t\mid s_t,m).$$1, working topic memory $$T(y_t \mid s_t, M_t)=\sum_{m\in M_t} p(m\mid s_t,M_t)\cdot P_{\mathrm{LLM}}(y_t\mid s_t,m).$$2, and clue-indexed episodic memory $$T(y_t \mid s_t, M_t)=\sum_{m\in M_t} p(m\mid s_t,M_t)\cdot P_{\mathrm{LLM}}(y_t\mid s_t,m).$$3, thereby separating persistent user profile state from topical continuity and rare cue-triggered recall (Wang et al., 17 Nov 2025).

Lifecycle management is equally central. MemMA treats memory construction, retrieval, verification, diagnosis, and repair as a closed “memory cycle,” adding a backward path in which probe QA pairs are synthesized, failures are classified, and localized repairs are committed before memory is finalized (Lin et al., 19 Mar 2026). MARS and SAGE employ Ebbinghaus-style retention models,

$$T(y_t \mid s_t, M_t)=\sum_{m\in M_t} p(m\mid s_t,M_t)\cdot P_{\mathrm{LLM}}(y_t\mid s_t,m).$$4

together with threshold-based STM/LTM transfer and MemorySyntax-style optimization that increases retention strength for salient items (Liang et al., 25 Mar 2025, Liang et al., 2024). AutoMem makes file-system operations such as READ, WRITE, SEARCH, APPEND, CREATE, and schema-specific actions like <|UPSERT_MAP|> into first-class memory actions, and then separately optimizes the memory scaffold and the model’s proficiency at using it (Wu et al., 1 Jul 2026).

These designs imply that “memory management” is not a secondary engineering detail. The write schema, consolidation rules, provenance model, decay schedule, and update semantics all determine whether experience accumulates as reusable structure or degenerates into retrieval noise.

4. Retrieval, valuation, and self-optimization

A decisive shift in recent work is the treatment of retrieval as policy optimization rather than passive lookup. MemRL’s two-phase mechanism first filters by semantic similarity and then reranks by learned utility, using a composite score

$$T(y_t \mid s_t, M_t)=\sum_{m\in M_t} p(m\mid s_t,M_t)\cdot P_{\mathrm{LLM}}(y_t\mid s_t,m).$$5

with ablations reporting that $$T(y_t \mid s_t, M_t)=\sum_{m\in M_t} p(m\mid s_t,M_t)\cdot P_{\mathrm{LLM}}(y_t\mid s_t,m).$$6 balances grounding and utility, and that normalization plus a strict similarity gate are essential for stability and low forgetting (Zhang et al., 6 Jan 2026). DecentMem uses a different control law: each agent routes between an exploitation pool and an exploration pool according to online weights updated from stage-wise LLM-as-a-judge feedback, yielding theoretical guarantees of global reachability and $$T(y_t \mid s_t, M_t)=\sum_{m\in M_t} p(m\mid s_t,M_t)\cdot P_{\mathrm{LLM}}(y_t\mid s_t,m).$$7 cumulative regret for the routing policy (Hao et al., 21 May 2026).

EvolveNav casts rule retrieval as a bandit problem. Each rule has a momentum-smoothed utility $$T(y_t \mid s_t, M_t)=\sum_{m\in M_t} p(m\mid s_t,M_t)\cdot P_{\mathrm{LLM}}(y_t\mid s_t,m).$$8, retrieval count $$T(y_t \mid s_t, M_t)=\sum_{m\in M_t} p(m\mid s_t,M_t)\cdot P_{\mathrm{LLM}}(y_t\mid s_t,m).$$9, and total episode count $(F,U,R,C)$0, and selection is driven by

$(F,U,R,C)$1

with newly created rules assigned infinite initial UCB to guarantee validation. Retrieved rules are then used in a preflection module that forecasts failure risk before action, allowing the agent to avoid unpromising frontiers rather than relying only on post-hoc correction (Chai et al., 16 Jun 2026). MemAPO applies a dual-memory retrieval scheme in which strategy templates are retrieved by cosine similarity while error patterns function as reusable negative constraints; the reported optimum is $(F,U,R,C)$2 retrieved templates, with larger $(F,U,R,C)$3 increasing noise and token burden (Liang et al., 23 Mar 2026).

The optimization target also varies across frameworks. UMEM jointly trains extraction and management through Semantic Neighborhood Modeling and a neighborhood-level marginal utility reward optimized with GRPO, so that memory updates are rewarded for helping semantically related queries rather than overfitting to a single instance (Ye et al., 11 Feb 2026). “Memory-Driven Self-Improvement for Decision Making with LLMs” stores $(F,U,R,C)$4 pairs with Q-values, retrieves nearest neighbors in embedding space, and then refines the LLM prior through an EM-style objective that upweights high-Q actions in LoRA-based policy updates (Yan et al., 30 Sep 2025). SelfMem dispenses with per-question supervision and instead refines memory strategy notes from aggregate training feedback, including official score, Pass_0.5, token cost, latency, cache hits, and tool diagnostics (Yang et al., 4 Jul 2026).

The common tendency is clear: high-performing systems do not merely have memory. They assign utility to memory, expose that utility to optimization, and refine retrieval rules with feedback signals that are broader than lexical similarity.

5. Empirical domains and benchmark evidence

The empirical literature indicates that memory-driven self-improvement is not restricted to one task family. It has been evaluated in code generation, web navigation, embodied exploration, OS and database tasks, prompt optimization, multi-agent coordination, long-context QA, deep personalization, and long-horizon game-like environments.

Setting	Exact reported outcome	Paper
ALFWorld runtime learning with MemRL	MemRL 0.507 / 0.697 vs MemP 0.324 / 0.456	(Zhang et al., 6 Jan 2026)
BEAM long-context evaluation with SelfMem	official score improvements of 48.7%, 40.8%, and 41.9% at 100K, 500K, and 1M	(Yang et al., 4 Jul 2026)
Decentralized multi-agent memory	average accuracy improved by up to 23.8% over the strongest centralized baseline and by up to 52.5% over the no-memory baseline, while reducing token usage by up to 49%	(Hao et al., 21 May 2026)
Web agents with CER	WebArena average success rate 36.7%; VisualWebArena 31.9%; relative improvement of the GPT-4o baseline by 51.0% on WebArena	(Liu et al., 7 Jun 2025)
Personalized long-horizon memory with O-Mem	51.67% on LoCoMo and 62.99% on PERSONAMEM	(Wang et al., 17 Nov 2025)
Meta-evolution of memory systems	improvements up to 17.06% for frameworks such as SmolAgent and Flash-Searcher	(Zhang et al., 21 Dec 2025)

Additional results reinforce the same pattern. MetaMem reports that it outperforms strong baselines by over 3.6% while specifically improving multi-session and temporal reasoning by teaching the model how to integrate fragmented memories (Xin et al., 27 Jan 2026). EvolveNav reports a 10.1% improvement in success rate with fewer unnecessary steps in zero-shot object-goal navigation by combining self-evolving rule memory with preflection (Chai et al., 16 Jun 2026). Trajectory-Informed Memory Generation improves scenario goal completion on held-out AppWorld tasks by up to 14.3 percentage points overall and by 28.5 percentage points on complex tasks, corresponding to a 149% relative increase in the hard regime (Fang et al., 11 Mar 2026).

A notable empirical regularity is that gains are strongest when environments exhibit recurring latent structure but sparse or costly feedback. This includes exploration-heavy embodied tasks, repeated workflow execution on the web, long conversational memory, and multi-agent coordination with heterogeneous roles. Systems that only store raw experience usually underperform those that compress experience into reusable abstractions, utilities, or skill interfaces.

6. Failure modes, governance, and open problems

The literature is equally explicit about failure modes. MemRL identifies noisy retrieval, semantic drift, sparse or misleading rewards, cold start, memory saturation, long-horizon credit assignment, and overfitting to spurious memories as persistent problems (Zhang et al., 6 Jan 2026). REMO adds that memory-augmented prompt optimization can incur a 3–5× increase in training time relative to TextGrad and remains vulnerable to noisy accumulation, redundancy, cold start, and static thresholds (Wu et al., 26 Aug 2025). O-Mem emphasizes that profiling errors, contradiction handling, drift in user preferences, and privacy-sensitive personalization remain open operational concerns even when retrieval quality improves (Wang et al., 17 Nov 2025).

These concerns have driven a parallel line of work on auditability and verification. MemMA introduces in-situ self-evolving memory construction through probe synthesis, answerability diagnosis, evidence-grounded repair, and semantic consolidation before commit, directly addressing the delayed-supervision problem (Lin et al., 19 Mar 2026). ASG-SI pushes the governance argument further by converting successful trajectories into verifier-backed skills with replayable evidence, decomposed rewards, contract checks, and append-only audit logging, explicitly motivated by reward hacking, behavioral drift, and supply-chain integrity concerns (Huang et al., 28 Dec 2025). AutoMem offers a related but less formal audit channel: because memory operations are first-class actions, each memory decision is visible in the trajectory and can be reviewed by an outer-loop meta-LLM (Wu et al., 1 Jul 2026). DecentMem, meanwhile, frames decentralization itself as a response to communication overhead, privacy risk, and loss of specialization under centralized memory (Hao et al., 21 May 2026).

Several research directions recur across papers. Proposed extensions include richer symbolic and graph-based memory, probabilistic or uncertainty-aware meta-memory, partial sharing protocols for multi-agent systems, adaptive thresholds, multimodal verification, learned memory controllers under bounded context, persistent cross-episode memory, and broader cross-domain transfer studies (Xin et al., 27 Jan 2026, Zhang et al., 21 Dec 2025, Yang et al., 4 Jul 2026). A plausible implication is that future memory-driven self-improvement systems will converge on hybrid designs: explicit external memory for auditability and transfer, learned retrieval and management policies for adaptability, and verifier-backed repair or governance layers for robustness.

In that broader sense, the memory-driven self-improvement framework has become a unifying research program rather than a single algorithmic template. Its defining claim is that the most tractable route to continual improvement in agentic LLM systems may lie not in repeatedly changing the model, but in making memory itself an adaptive, optimizable, and increasingly accountable computational substrate.