---
title: Self-Evolving Experience Harvesting
url: https://www.emergentmind.com/topics/self-evolving-experience-harvesting
type: topic
---

# Self-Evolving Experience Harvesting

Searching arXiv for recent work on self-evolving agents, experience memory, and related frameworks.
Self-evolving experience harvesting is a design pattern for agent systems in which interaction trajectories are not treated as disposable context, but are converted into reusable knowledge that conditions future planning, execution, retrieval, or training. In the recent literature, this knowledge is externalized as structured memory artifacts such as SOPs, workflows, skills, tips, guardrails, guidelines, or tools; updated online or across iterations; and reused to improve long-horizon task solving without leaving the agent permanently test-time static. Representative formulations include MUSE’s hierarchical Memory Module for long-horizon productivity tasks, UI-Mem’s Hierarchical Experience Memory for online mobile GUI RL, EvoNote’s evolving memory of misinformation-correction episodes, and Live-Evo’s online Experience Bank plus Meta-Guideline Bank [2510.08002] [2602.05832] [2606.02215] [2602.02369].

## 1. Conceptual structure and memory representations

Across the surveyed systems, the central abstraction is that raw trajectories are progressively compressed into higher-value units that are easier to retrieve and reuse than full histories. MUSE makes this explicit by separating raw trajectories \(h\) from structured experiences \(e\), with a hierarchical Memory Module \(\mathcal{M}=\{\mathcal{M}_{\text{strat}},\mathcal{M}_{\text{proc}},\mathcal{M}_{\text{tool}}\}\) containing strategic \(\langle\)Dilemma, Strategy\(\rangle\) pairs, procedural SOPs, and tool entries [2510.08002]. UI-Mem likewise departs from flat replay buffers by storing three abstracted template types—High-Level Workflows, Mid-Level Subtask Skills, and Failure Patterns—specifically to support fine-grained credit assignment in long-horizon tasks and cross-task, cross-application transfer [2602.05832].

Other frameworks define the stored unit differently but preserve the same separation between execution traces and reusable abstractions. Mobile-Agent-E maintains persistent long-term memory as natural-language Tips and structured JSON Shortcuts, where Tips are “lessons learned” and Shortcuts are symbolic templates with a unique name, precondition, parameters, and a sequence of atomic operations [2501.11733]. SkeMex distills trajectories into skills \(s=f(T)=(\phi(T),\psi(T))\), where \(\phi(T)\) is the situational trigger and \(\psi(T)\) the reusable procedure, and stores each item as \(m_i=(k_i,c_i,u_i)\) with content, embedding key, and estimated utility [2606.09365]. Metis splits memory into natural-language text and executable code, with text entries typed as environment facts, common pitfalls, or execution plans, and code entries admitted only after sandbox compilation and dependency-closure checks [2606.24151].

A compact comparison of common memory schemas is shown below.

| Framework | Stored artifacts | Organization |
|---|---|---|
| MUSE | strategic entries, SOPs, tool entries | \(\mathcal{M}_{\text{strat}}, \mathcal{M}_{\text{proc}}, \mathcal{M}_{\text{tool}}\) |
| UI-Mem | workflows, subtask skills, failure patterns | Hierarchical Experience Memory |
| SkeMex | general, task-specific, action-level skills | multi-branch repository |
| Metis | env, pit, plan text; callable tools | dual-representation memory |

The same pattern also appears in domain-specific systems. FinAcumen stores each memory entry as the original question, its gold answer, and two distilled texts: “Findings” from correct trajectories and “Cautions” from failed runs [2606.17642]. GSEM stores a clinical experience \(e_i=(c_i,s_i,z_i,Q_i^{(0)})\) inside a dual-layer memory graph, where \(c_i\) and \(s_i\) are textual condition and strategy spans, \(z_i\) marks indication versus contraindication, and graph edges capture inter-experience relations [2603.22096]. AutoSkill uses versioned, human-editable skill artifacts with fields such as name, description, prompt body, triggers, tags, examples, and semantic version number [2603.01145].

## 2. Closed-loop harvesting: from trajectories to reusable knowledge

The operational core of self-evolving experience harvesting is a closed loop that alternates between acting in the environment and rewriting the resulting trace into memory. MUSE presents this loop explicitly as “Plan–Execute–Reflect–Memorize.” A PE Agent decomposes a high-level task \(\tau\) into an ordered queue of sub-tasks, executes each through a ReAct loop, and emits a trajectory \(h_{k:t}=(o_{k:t},a_{k:t-1})\). A Reflect Agent then evaluates the attempt, flags success or failure, and—on success—distills the trajectory into a new SOP via
\[
e_t = \mathrm{Convert}(\tau_t),
\]
with content fields \(\{\text{analysis},\text{precautions},\text{params},\text{steps}\}\) [2510.08002].

Reflection is not merely a binary success check. In MUSE, the Reflect Agent constructs a checklist over truthfulness, deliverables, and fidelity, uses trajectory referencing and active tool verification, and either summarizes successful traces or emits failure-cause analysis \(R_{\text{fail}}\) [2510.08002]. Steve-Evolving strengthens this attribution step by “Experience Anchoring,” storing every subgoal attempt as
\[
e_t=\langle s_{\rm pre}^{(t)}, a_t, \mathcal{D}(s_t,a_t), s_{\rm post}^{(t)} \rangle,
\]
where the diagnosis module returns binary outcome, state-difference summaries, one of 11 enumerated failure causes, and continuous indicators such as coordinate variance and inventory-change norms. Successful sequences are distilled into reusable skills with explicit preconditions and verification criteria, while failures are distilled into executable guardrails at subgoal and task granularity [2603.13131].

Several systems use explicit judges or evaluators to drive memory writing. EvoNote computes trajectory-level feedback \(R(\tau_k)\in\mathbb{R}^4\) over Understandability, Meaningfulness, Usability, and Trustworthiness, assigns each action \(a_i\) a credit
\[
c_i = \sum_{q=1}^4 w_{q,i} R_q(\tau_k),
\]
and then has a Memory Evolver convert positive-credit actions into phase-tagged lessons of the form \((\mathrm{Trigger}, \mathrm{ActionStrategy}, \mathrm{Phase})\) [2606.02215]. UI-Mem updates memory after each training iteration in three stages—Reward & Subtask Verification, Experience Extraction & Abstraction, and Memory Update—using successful trajectories to extract workflows and subtask plans, and failed trajectories to extract a failure diagnosis for the first failed subtask only [2602.05832].

OPD-Evolver generalizes this loop into a lifecycle of retrieval, selection, execution, writing, and maintenance. In its fast loop, the agent reads from a four-level hierarchy \(\{\text{traj},\text{tip},\text{skill},\text{tool}\}\), executes under selected memory, writes new memories \(\Delta_t^\ell\), and periodically performs maintenance such as merge or delete operations; in the slow loop, outcome-calibrated memory attribution provides hindsight supervision for selection, action, writing, and maintenance abilities [2606.17628].

## 3. Retrieval, selection, and runtime utilization

Harvested experience affects future behavior only through a retrieval and utilization policy, and recent work treats this policy as a first-class design dimension. Some frameworks use simple retrieval. MUSE maintains a lightweight procedural index \(I_{\text{proc}}\), matches sub-task descriptions by simple keyword or LLM-prompt criteria, and formalizes retrieval as
\[
R(\mathrm{desc})=\{p.\mathrm{content}: p\in \mathcal{M}_{\mathrm{proc}}, \ \mathrm{sim}(\mathrm{desc},p.\mathrm{index.type})>\theta\},
\]
while relying in practice on simple LLM-based retrieval prompts rather than heavy vector scores [2510.08002].

Other systems make utility part of the retrieval score itself. Live-Evo scores each experience by
\[
\mathrm{Score}(e;q)=w_e \times \mathrm{sim}(e,q),
\]
retrieves the top-\(k\) entries, and then updates weights online by contrastive benefit:
\[
w_e \leftarrow w_e + \bigl(r_q^{\mathrm{off}} - r_q^{\mathrm{on}}\bigr).
\]
Experiences that consistently help are reinforced; misleading or stale experiences are down-weighted and gradually forgotten [2602.02369]. SkeMex similarly combines semantic similarity, context-dependent utility, and recency-weighted memory strength:
\[
\mathrm{Score}_{\mathrm{ret}}(m\mid s_0,\kappa)
=\lambda_{\mathrm{sim}}\mathrm{Sim}(s_0,m)+\lambda_u U(m\mid \kappa)+\lambda_h h(m),
\]
and performs branch-aware Top-\(K\) selection across general, task, and action-level repositories [2606.09365].

A distinct line of work studies not just what to retrieve, but when memory should enter the reasoning process. ExpWeaver modifies runtime utilization by making retrieval optional inside the ReAct loop. At each step, the model emits a reasoning trace \(h_t\); if that trace contains the trigger token “[Retrieve],” then \(g_t=1\), memory is fetched using the current context and reasoning prefix, and the action is generated from \(c_t \oplus h_t \oplus M_t\). This turns memory into an on-demand resource rather than a rigidly always-on or initialization-only prompt component [2605.07164].

Selectivity can also be enforced by thresholding. FinAcumen embeds each query, compares it to stored entries by cosine similarity,
\[
\mathrm{sim}(x,m)=\frac{E(x)\cdot z_m}{\|E(x)\|\,\|z_m\|},
\]
forms the candidate set \(\mathcal{M}_x=\{m\in\mathcal{M}\mid \mathrm{sim}(x,m)\ge \tau\}\), and uses an explicit fallback if no entry passes the threshold. The reported operating point is \(\tau=0.65\), which yields a 79.8% hit rate on large-bank snapshots; if retrieval is empty, the agent reverts to the tool-only rollout [2606.17642].

These retrieval mechanisms are increasingly coupled to memory governance. UI-Mem ranks template use with an Upper-Confidence-Bound-style score over historical success and usage counts [2602.05832]. GSEM performs applicability-aware retrieval by hybrid seed recall followed by multi-seed graph traversal, ranking neighbors by
\[
\mathrm{score}(e_h,e_j)=\tfrac12\bigl(W_{hj}^{(t)}+Q_j^{(t)}\bigr),
\]
so that both relation strength and node quality affect what is reused [2603.22096].

## 4. External memory, internalization, and co-evolution

The literature now distinguishes between two broad ways harvested experience changes an agent. In non-parametric systems, memory remains external and the backbone is unchanged. MUSE, EvoNote, Steve-Evolving, SkeMex, Live-Evo, FinAcumen, and AutoSkill all emphasize test-time evolution through memory updates rather than parameter updates [2510.08002] [2606.02215] [2603.13131] [2606.09365] [2602.02369] [2606.17642] [2603.01145]. By contrast, Evolving-RL treats self-evolution as an end-to-end optimization problem in which a shared policy serves both as extractor and solver. For a source-task rollout, the extractor samples candidate skills \(e_i\sim \pi_\theta(\cdot\mid s^e)\), evaluates them on retrieved downstream tasks, computes extractor-side and solver-side advantages, and jointly minimizes
\[
\mathcal{L}(\theta)=\lambda_e \mathcal{L}_e(\theta)+\lambda_s \mathcal{L}_s(\theta),
\]
so that skill extraction and utilization co-evolve inside a single parameter vector \(\theta\) [2605.10663].

A related issue is how external experience becomes internal capability. The study of continual experience internalization identifies three decisive design choices: experience granularity, experience injection pattern, and internalization regime. Principle-level experience is reported to be more durable than instance-level experience; step-wise injection, which selects relevant experience subsets at each intermediate state, outperforms global injection; and off-policy context-distillation on high-quality teacher trajectories is more stable than on-policy distillation under multi-iteration self-evolution [2606.04703]. In that formulation, the off-policy objective is
\[
\mathcal{L}_{\mathrm{off}}(\theta)
=\mathbb{E}_{H\sim\pi_T}\Big[\sum_{t=1}^T D_{\mathrm{KL}}(p_t \,\Vert\, q_t)\Big],
\]
where \(p_t\) is the experience-conditioned teacher distribution and \(q_t\) the student [2606.04703].

Other papers broaden the notion of co-evolution beyond memory alone. Mem\(^2\)Evolve argues that experience accumulation and asset creation should not be separated. It maintains both Experience Memory \(E\) and Asset Memory \(A\), recruits an existing asset if similarity exceeds a threshold \(\delta\), otherwise creates a new one conditioned on \(E\), and updates both memories after task evaluation:
\[
E_{t+1}=E_t \cup \{\mathrm{Reflect}(\tau_t,r_t,c_t)\}, \qquad
A_{t+1}=A_t \cup \{m_{\mathrm{final}}\}.
\]
This makes harvested experience a guide for future tool and expert-agent generation rather than only a retrieval aid [2604.10923].

Metis provides a different synthesis by letting textual plans remain in memory until recurrence justifies codification into executable tools. When a plan \(p\) has been selected often enough that \(|B(p)|\ge \theta\), it triggers tool generation, validation, and admission into code memory. This architecture is explicitly motivated by the trade-off that text memory has lower construction cost and broader transferability, while code memory has higher execution efficiency but higher construction cost and sharper transfer drop [2606.24151].

## 5. Empirical evidence across domains

Empirical results show that self-evolving experience harvesting is not confined to one benchmark family. On long-horizon productivity tasks, MUSE reports continuous-learning gains on an 18-task subset of TAC, with \(S_{\text{partial}}\) rising from approximately 40% at iteration 0 to approximately 55% by iteration 3, and on the full 175-task TAC benchmark reaches \(S_{\text{partial}}=51.78\%\) and \(PCR=41.14\%\), exceeding Openhands-Versa with Claude-4 Sonnet at \(S_{\text{partial}}=43.19\%\) [2510.08002]. On mobile GUI online RL, UI-Mem reports AndroidWorld success rates of 58.2 for UI-Mem-4B, 62.5 for UI-Mem-4B* with retrieval at inference, 66.8 for UI-Mem-8B, and 71.1 for UI-Mem-8B*, while removing hierarchy reduces success by 5–8%, disabling abstraction drops performance from 66.8% to 58.2%, and disabling self-evolution reduces 66.8% to 62.9% [2602.05832]. Mobile-Agent-E reports a +22.1 pp absolute gain over prior SOTA without evolution and an additional +6.5 pp with self-evolution on its Satisfaction Score metric [2501.11733].

In knowledge-intensive and socially grounded settings, EvoNote reports a human-validated 89.6% pairwise win rate against corresponding human-written notes, 82.0% helpfulness on 300 “Needs More Ratings” posts, and an end-to-end latency under 2 minutes compared with median human note latency over 13 hours [2606.02215]. Live-Evo, evaluated over a 10-week horizon on Prophet Arena, improves average Brier score from 0.19 to 0.14, a 20.8% reduction, and increases average return from 1.24 to 1.46, a 12.9% gain [2602.02369]. FinAcumen reports that adding Financial Memory on top of deterministic financial tools improves FinMMR Easy from 74.08 to 81.67, and that \(k_{\max}=5\) retrieved entries per query gives the best accuracy in its retrieval-depth analysis [2606.17642].

Clinical and medical-agent studies show the same pattern under different memory formalisms. GSEM reaches the highest average accuracy among baselines on MedR-Bench and MedAgentsBench, with 70.90% and 69.24% using DeepSeek-V3.2 and Qwen3.5-35B respectively, and its self-evolution-only ablation raises treatment accuracy from 94.59% to 97.30% after 50 updates [2603.22096]. SkeMex raises average offline in-domain accuracy from 48.20% for no-memory ReAct to 56.08%, gains +13.78 points on out-of-domain transfer relative to ReAct, and in online continual learning reaches 78.56% at epoch 3 [2606.09365]. “Experience Makes Skillful” also reports that compact, governed skill memory can remain small—only a few dozen skills per branch—while still improving online reasoning [2606.09365].

In embodied, web, and agentic-benchmark settings, the evidence is similarly broad. Steve-Evolving reaches 50.09% overall success rate on Minecraft MCU with Qwen3.5-flash, compared with 45.83% for Optimus-1 and 41.75% for Jarvis-1 [2603.13131]. Evolving-RL reports ALFWorld unseen-task success of 88.6% with skills and 81.1% without test-time skill injection, versus 44.6% for a GRPO baseline with skills, and on Mind2Web improves overall action accuracy from 22.83 to 30.87 with skills [2605.10663]. Metis reaches 60.1% accuracy on AppWorld with 97.4K execution tokens and 11.3 turns, compared with 51.8%, 112.6K, and 14.6 for no-memory ReAct [2606.24151]. AgentEvolver reports large gains over vanilla GRPO, including avg@8 improvements from 15.8% to 45.2% on Qwen2.5-7B and from 29.8% to 57.6% on Qwen2.5-14B [2511.10395].

## 6. Failure modes, safety tensions, and governance

Despite these gains, the literature also identifies stability and safety problems intrinsic to experience-driven evolution. A direct safety study shows that memory accumulation can degrade refusal behavior even when all stored experience comes from benign tasks. In offline self-evolution with top-3 retrieved experiences, BrowserART Attack Success Rate rises for GPT-4o from 37.0 to 50.0, for Claude-4.5-Sonnet from 17.0 to 23.0, and for Qwen3-235B-A22B from 39.0 to 53.0; similar increases are reported on Agent-SafetyBench and SafeAgentBench [2604.16968]. In mixed benign-plus-harmful streams, refusal-only traces lower ASR but reduce benign-task Task Success Rate through over-refusal, while mixed memory yields intermediate ASR and TSR, exposing a safety–utility trade-off [2604.16968].

A second concern is long-run instability in capability acquisition. Multi-iteration experience internalization can exhibit “progressive capability collapse rather than compounding improvement” when instance-level experience, global injection, or on-policy context-distillation are used. Reported analyses show instance-level experience giving an initial boost and then falling below base by iteration 3, global injection degrading from 21.0% to 8.5% on WebWalkerQA across iterations, and on-policy internalization causing student turns to jump from 2.5 to 21.9, whereas principle-level, step-wise, off-policy internalization remains stable [2606.04703]. This suggests that the design of the harvesting loop is not separable from the design of the learning regime.

Memory growth and repository contamination are also recurring issues. FinAcumen’s gains per 100 new entries follow a diminishing-returns curve that saturates beyond about 1800 entries, and ablation of retrieval-time deduplication degrades performance [2606.17642]. SkeMex addresses growth through explicit “Govern” operations—merge, promote, deprecate, and capacity-based removal—while Live-Evo relies on reinforcement and decay of experience weights so that misleading or stale experiences gradually fall below the retrieval threshold [2606.09365] [2602.02369]. GSEM similarly keeps original content fixed but calibrates node quality \(Q\) and edge weights \(W\) from online feedback, refining reliability without rewriting the stored experience itself [2603.22096].

The resulting picture is not that self-evolving experience harvesting is a single method, but that it is an organizing principle spanning memory abstraction, feedback attribution, selective retrieval, runtime gating, repository governance, and, in some systems, parameter internalization. The surveyed work consistently treats raw trajectories as an intermediate substrate rather than the final product of learning, and locates long-horizon improvement in the quality of the transformations from trajectory to reusable experience, from experience to action, and from repeated reuse to durable capability [2508.19005].

Source: https://www.emergentmind.com/topics/self-evolving-experience-harvesting