Self-Evolving Experience Harvesting

• Self-evolving experience harvesting is a design pattern that converts raw agent trajectories into structured, reusable memory artifacts such as SOPs, workflows, and tool entries.
• It employs a closed-loop process—plan, execute, reflect, and memorize—to continuously enhance long-horizon task performance and refine decision-making.
• The approach integrates diverse memory representations and retrieval mechanisms, yielding measurable gains in productivity, accuracy, and safety across multiple domains.

Searching arXiv for recent work on self-evolving agents, experience memory, and related frameworks. Self-evolving experience harvesting is a design pattern for agent systems in which interaction trajectories are not treated as disposable context, but are converted into reusable knowledge that conditions future planning, execution, retrieval, or training. In the recent literature, this knowledge is externalized as structured memory artifacts such as SOPs, workflows, skills, tips, guardrails, guidelines, or tools; updated online or across iterations; and reused to improve long-horizon task solving without leaving the agent permanently test-time static. Representative formulations include MUSE’s hierarchical Memory Module for long-horizon productivity tasks, UI-Mem’s Hierarchical Experience Memory for online mobile GUI RL, EvoNote’s evolving memory of misinformation-correction episodes, and Live-Evo’s online Experience Bank plus Meta-Guideline Bank (Yang et al., 9 Oct 2025, Xiao et al., 5 Feb 2026, Fu et al., 1 Jun 2026, Zhang et al., 2 Feb 2026).

1. Conceptual structure and memory representations

Across the surveyed systems, the central abstraction is that raw trajectories are progressively compressed into higher-value units that are easier to retrieve and reuse than full histories. MUSE makes this explicit by separating raw trajectories $h$ from structured experiences $e$, with a hierarchical Memory Module $$\mathcal{M}=\{\mathcal{M}_{\text{strat}},\mathcal{M}_{\text{proc}},\mathcal{M}_{\text{tool}}\}$$ containing strategic $\langle$Dilemma, Strategy$\rangle$ pairs, procedural SOPs, and tool entries (Yang et al., 9 Oct 2025). UI-Mem likewise departs from flat replay buffers by storing three abstracted template types—High-Level Workflows, Mid-Level Subtask Skills, and Failure Patterns—specifically to support fine-grained credit assignment in long-horizon tasks and cross-task, cross-application transfer (Xiao et al., 5 Feb 2026).

Other frameworks define the stored unit differently but preserve the same separation between execution traces and reusable abstractions. Mobile-Agent-E maintains persistent long-term memory as natural-language Tips and structured JSON Shortcuts, where Tips are “lessons learned” and Shortcuts are symbolic templates with a unique name, precondition, parameters, and a sequence of atomic operations (Wang et al., 20 Jan 2025). SkeMex distills trajectories into skills $s=f(T)=(\phi(T),\psi(T))$, where $\phi(T)$ is the situational trigger and $\psi(T)$ the reusable procedure, and stores each item as $m_i=(k_i,c_i,u_i)$ with content, embedding key, and estimated utility (Sun et al., 8 Jun 2026). Metis splits memory into natural-language text and executable code, with text entries typed as environment facts, common pitfalls, or execution plans, and code entries admitted only after sandbox compilation and dependency-closure checks (Dai et al., 23 Jun 2026).

A compact comparison of common memory schemas is shown below.

Framework	Stored artifacts	Organization
MUSE	strategic entries, SOPs, tool entries	$$\mathcal{M}_{\text{strat}}, \mathcal{M}_{\text{proc}}, \mathcal{M}_{\text{tool}}$$
UI-Mem	workflows, subtask skills, failure patterns	Hierarchical Experience Memory
SkeMex	general, task-specific, action-level skills	multi-branch repository
Metis	env, pit, plan text; callable tools	dual-representation memory

The same pattern also appears in domain-specific systems. FinAcumen stores each memory entry as the original question, its gold answer, and two distilled texts: “Findings” from correct trajectories and “Cautions” from failed runs (Guo et al., 16 Jun 2026). GSEM stores a clinical experience $e$0 inside a dual-layer memory graph, where $e$1 and $e$2 are textual condition and strategy spans, $e$3 marks indication versus contraindication, and graph edges capture inter-experience relations (Han et al., 23 Mar 2026). AutoSkill uses versioned, human-editable skill artifacts with fields such as name, description, prompt body, triggers, tags, examples, and semantic version number (Yang et al., 1 Mar 2026).

2. Closed-loop harvesting: from trajectories to reusable knowledge

The operational core of self-evolving experience harvesting is a closed loop that alternates between acting in the environment and rewriting the resulting trace into memory. MUSE presents this loop explicitly as “Plan–Execute–Reflect–Memorize.” A PE Agent decomposes a high-level task $e$4 into an ordered queue of sub-tasks, executes each through a ReAct loop, and emits a trajectory $e$5. A Reflect Agent then evaluates the attempt, flags success or failure, and—on success—distills the trajectory into a new SOP via

$e$6

with content fields $e$7 (Yang et al., 9 Oct 2025).

Reflection is not merely a binary success check. In MUSE, the Reflect Agent constructs a checklist over truthfulness, deliverables, and fidelity, uses trajectory referencing and active tool verification, and either summarizes successful traces or emits failure-cause analysis $e$8 (Yang et al., 9 Oct 2025). Steve-Evolving strengthens this attribution step by “Experience Anchoring,” storing every subgoal attempt as

$e$9

where the diagnosis module returns binary outcome, state-difference summaries, one of 11 enumerated failure causes, and continuous indicators such as coordinate variance and inventory-change norms. Successful sequences are distilled into reusable skills with explicit preconditions and verification criteria, while failures are distilled into executable guardrails at subgoal and task granularity (Xie et al., 13 Mar 2026).

Several systems use explicit judges or evaluators to drive memory writing. EvoNote computes trajectory-level feedback $$\mathcal{M}=\{\mathcal{M}_{\text{strat}},\mathcal{M}_{\text{proc}},\mathcal{M}_{\text{tool}}\}$$0 over Understandability, Meaningfulness, Usability, and Trustworthiness, assigns each action $$\mathcal{M}=\{\mathcal{M}_{\text{strat}},\mathcal{M}_{\text{proc}},\mathcal{M}_{\text{tool}}\}$$1 a credit

$$\mathcal{M}=\{\mathcal{M}_{\text{strat}},\mathcal{M}_{\text{proc}},\mathcal{M}_{\text{tool}}\}$$2

and then has a Memory Evolver convert positive-credit actions into phase-tagged lessons of the form $$\mathcal{M}=\{\mathcal{M}_{\text{strat}},\mathcal{M}_{\text{proc}},\mathcal{M}_{\text{tool}}\}$$3 (Fu et al., 1 Jun 2026). UI-Mem updates memory after each training iteration in three stages—Reward & Subtask Verification, Experience Extraction & Abstraction, and Memory Update—using successful trajectories to extract workflows and subtask plans, and failed trajectories to extract a failure diagnosis for the first failed subtask only (Xiao et al., 5 Feb 2026).

OPD-Evolver generalizes this loop into a lifecycle of retrieval, selection, execution, writing, and maintenance. In its fast loop, the agent reads from a four-level hierarchy $$\mathcal{M}=\{\mathcal{M}_{\text{strat}},\mathcal{M}_{\text{proc}},\mathcal{M}_{\text{tool}}\}$$4, executes under selected memory, writes new memories $$\mathcal{M}=\{\mathcal{M}_{\text{strat}},\mathcal{M}_{\text{proc}},\mathcal{M}_{\text{tool}}\}$$5, and periodically performs maintenance such as merge or delete operations; in the slow loop, outcome-calibrated memory attribution provides hindsight supervision for selection, action, writing, and maintenance abilities (Zhang et al., 16 Jun 2026).

3. Retrieval, selection, and runtime utilization

Harvested experience affects future behavior only through a retrieval and utilization policy, and recent work treats this policy as a first-class design dimension. Some frameworks use simple retrieval. MUSE maintains a lightweight procedural index $$\mathcal{M}=\{\mathcal{M}_{\text{strat}},\mathcal{M}_{\text{proc}},\mathcal{M}_{\text{tool}}\}$$6, matches sub-task descriptions by simple keyword or LLM-prompt criteria, and formalizes retrieval as

$$\mathcal{M}=\{\mathcal{M}_{\text{strat}},\mathcal{M}_{\text{proc}},\mathcal{M}_{\text{tool}}\}$$7

while relying in practice on simple LLM-based retrieval prompts rather than heavy vector scores (Yang et al., 9 Oct 2025).

Other systems make utility part of the retrieval score itself. Live-Evo scores each experience by

$$\mathcal{M}=\{\mathcal{M}_{\text{strat}},\mathcal{M}_{\text{proc}},\mathcal{M}_{\text{tool}}\}$$8

retrieves the top-$$\mathcal{M}=\{\mathcal{M}_{\text{strat}},\mathcal{M}_{\text{proc}},\mathcal{M}_{\text{tool}}\}$$9 entries, and then updates weights online by contrastive benefit: $\langle$0 Experiences that consistently help are reinforced; misleading or stale experiences are down-weighted and gradually forgotten (Zhang et al., 2 Feb 2026). SkeMex similarly combines semantic similarity, context-dependent utility, and recency-weighted memory strength: $\langle$1 and performs branch-aware Top-$\langle$2 selection across general, task, and action-level repositories (Sun et al., 8 Jun 2026).

A distinct line of work studies not just what to retrieve, but when memory should enter the reasoning process. ExpWeaver modifies runtime utilization by making retrieval optional inside the ReAct loop. At each step, the model emits a reasoning trace $\langle$3; if that trace contains the trigger token “[Retrieve],” then $\langle$4, memory is fetched using the current context and reasoning prefix, and the action is generated from $\langle$5. This turns memory into an on-demand resource rather than a rigidly always-on or initialization-only prompt component (Zhao et al., 8 May 2026).

Selectivity can also be enforced by thresholding. FinAcumen embeds each query, compares it to stored entries by cosine similarity,

$\langle$6

forms the candidate set $\langle$7, and uses an explicit fallback if no entry passes the threshold. The reported operating point is $\langle$8, which yields a 79.8% hit rate on large-bank snapshots; if retrieval is empty, the agent reverts to the tool-only rollout (Guo et al., 16 Jun 2026).

These retrieval mechanisms are increasingly coupled to memory governance. UI-Mem ranks template use with an Upper-Confidence-Bound-style score over historical success and usage counts (Xiao et al., 5 Feb 2026). GSEM performs applicability-aware retrieval by hybrid seed recall followed by multi-seed graph traversal, ranking neighbors by

$\langle$9

so that both relation strength and node quality affect what is reused (Han et al., 23 Mar 2026).

4. External memory, internalization, and co-evolution

The literature now distinguishes between two broad ways harvested experience changes an agent. In non-parametric systems, memory remains external and the backbone is unchanged. MUSE, EvoNote, Steve-Evolving, SkeMex, Live-Evo, FinAcumen, and AutoSkill all emphasize test-time evolution through memory updates rather than parameter updates (Yang et al., 9 Oct 2025, Fu et al., 1 Jun 2026, Xie et al., 13 Mar 2026, Sun et al., 8 Jun 2026, Zhang et al., 2 Feb 2026, Guo et al., 16 Jun 2026, Yang et al., 1 Mar 2026). By contrast, Evolving-RL treats self-evolution as an end-to-end optimization problem in which a shared policy serves both as extractor and solver. For a source-task rollout, the extractor samples candidate skills $\rangle$0, evaluates them on retrieved downstream tasks, computes extractor-side and solver-side advantages, and jointly minimizes

$\rangle$1

so that skill extraction and utilization co-evolve inside a single parameter vector $\rangle$2 (Fan et al., 11 May 2026).

A related issue is how external experience becomes internal capability. The study of continual experience internalization identifies three decisive design choices: experience granularity, experience injection pattern, and internalization regime. Principle-level experience is reported to be more durable than instance-level experience; step-wise injection, which selects relevant experience subsets at each intermediate state, outperforms global injection; and off-policy context-distillation on high-quality teacher trajectories is more stable than on-policy distillation under multi-iteration self-evolution (Chen et al., 3 Jun 2026). In that formulation, the off-policy objective is

$\rangle$3

where $\rangle$4 is the experience-conditioned teacher distribution and $\rangle$5 the student (Chen et al., 3 Jun 2026).

Other papers broaden the notion of co-evolution beyond memory alone. Mem$\rangle$6Evolve argues that experience accumulation and asset creation should not be separated. It maintains both Experience Memory $\rangle$7 and Asset Memory $\rangle$8, recruits an existing asset if similarity exceeds a threshold $\rangle$9, otherwise creates a new one conditioned on $s=f(T)=(\phi(T),\psi(T))$0, and updates both memories after task evaluation: $s=f(T)=(\phi(T),\psi(T))$1 This makes harvested experience a guide for future tool and expert-agent generation rather than only a retrieval aid (Cheng et al., 13 Apr 2026).

Metis provides a different synthesis by letting textual plans remain in memory until recurrence justifies codification into executable tools. When a plan $s=f(T)=(\phi(T),\psi(T))$2 has been selected often enough that $s=f(T)=(\phi(T),\psi(T))$3, it triggers tool generation, validation, and admission into code memory. This architecture is explicitly motivated by the trade-off that text memory has lower construction cost and broader transferability, while code memory has higher execution efficiency but higher construction cost and sharper transfer drop (Dai et al., 23 Jun 2026).

5. Empirical evidence across domains

Empirical results show that self-evolving experience harvesting is not confined to one benchmark family. On long-horizon productivity tasks, MUSE reports continuous-learning gains on an 18-task subset of TAC, with $s=f(T)=(\phi(T),\psi(T))$4 rising from approximately 40% at iteration 0 to approximately 55% by iteration 3, and on the full 175-task TAC benchmark reaches $s=f(T)=(\phi(T),\psi(T))$5 and $s=f(T)=(\phi(T),\psi(T))$6, exceeding Openhands-Versa with Claude-4 Sonnet at $s=f(T)=(\phi(T),\psi(T))$7 (Yang et al., 9 Oct 2025). On mobile GUI online RL, UI-Mem reports AndroidWorld success rates of 58.2 for UI-Mem-4B, 62.5 for UI-Mem-4B* with retrieval at inference, 66.8 for UI-Mem-8B, and 71.1 for UI-Mem-8B*, while removing hierarchy reduces success by 5–8%, disabling abstraction drops performance from 66.8% to 58.2%, and disabling self-evolution reduces 66.8% to 62.9% (Xiao et al., 5 Feb 2026). Mobile-Agent-E reports a +22.1 pp absolute gain over prior SOTA without evolution and an additional +6.5 pp with self-evolution on its Satisfaction Score metric (Wang et al., 20 Jan 2025).

In knowledge-intensive and socially grounded settings, EvoNote reports a human-validated 89.6% pairwise win rate against corresponding human-written notes, 82.0% helpfulness on 300 “Needs More Ratings” posts, and an end-to-end latency under 2 minutes compared with median human note latency over 13 hours (Fu et al., 1 Jun 2026). Live-Evo, evaluated over a 10-week horizon on Prophet Arena, improves average Brier score from 0.19 to 0.14, a 20.8% reduction, and increases average return from 1.24 to 1.46, a 12.9% gain (Zhang et al., 2 Feb 2026). FinAcumen reports that adding Financial Memory on top of deterministic financial tools improves FinMMR Easy from 74.08 to 81.67, and that $s=f(T)=(\phi(T),\psi(T))$8 retrieved entries per query gives the best accuracy in its retrieval-depth analysis (Guo et al., 16 Jun 2026).

Clinical and medical-agent studies show the same pattern under different memory formalisms. GSEM reaches the highest average accuracy among baselines on MedR-Bench and MedAgentsBench, with 70.90% and 69.24% using DeepSeek-V3.2 and Qwen3.5-35B respectively, and its self-evolution-only ablation raises treatment accuracy from 94.59% to 97.30% after 50 updates (Han et al., 23 Mar 2026). SkeMex raises average offline in-domain accuracy from 48.20% for no-memory ReAct to 56.08%, gains +13.78 points on out-of-domain transfer relative to ReAct, and in online continual learning reaches 78.56% at epoch 3 (Sun et al., 8 Jun 2026). “Experience Makes Skillful” also reports that compact, governed skill memory can remain small—only a few dozen skills per branch—while still improving online reasoning (Sun et al., 8 Jun 2026).

In embodied, web, and agentic-benchmark settings, the evidence is similarly broad. Steve-Evolving reaches 50.09% overall success rate on Minecraft MCU with Qwen3.5-flash, compared with 45.83% for Optimus-1 and 41.75% for Jarvis-1 (Xie et al., 13 Mar 2026). Evolving-RL reports ALFWorld unseen-task success of 88.6% with skills and 81.1% without test-time skill injection, versus 44.6% for a GRPO baseline with skills, and on Mind2Web improves overall action accuracy from 22.83 to 30.87 with skills (Fan et al., 11 May 2026). Metis reaches 60.1% accuracy on AppWorld with 97.4K execution tokens and 11.3 turns, compared with 51.8%, 112.6K, and 14.6 for no-memory ReAct (Dai et al., 23 Jun 2026). AgentEvolver reports large gains over vanilla GRPO, including avg@8 improvements from 15.8% to 45.2% on Qwen2.5-7B and from 29.8% to 57.6% on Qwen2.5-14B (Zhai et al., 13 Nov 2025).

6. Failure modes, safety tensions, and governance

Despite these gains, the literature also identifies stability and safety problems intrinsic to experience-driven evolution. A direct safety study shows that memory accumulation can degrade refusal behavior even when all stored experience comes from benign tasks. In offline self-evolution with top-3 retrieved experiences, BrowserART Attack Success Rate rises for GPT-4o from 37.0 to 50.0, for Claude-4.5-Sonnet from 17.0 to 23.0, and for Qwen3-235B-A22B from 39.0 to 53.0; similar increases are reported on Agent-SafetyBench and SafeAgentBench (Zhao et al., 18 Apr 2026). In mixed benign-plus-harmful streams, refusal-only traces lower ASR but reduce benign-task Task Success Rate through over-refusal, while mixed memory yields intermediate ASR and TSR, exposing a safety–utility trade-off (Zhao et al., 18 Apr 2026).

A second concern is long-run instability in capability acquisition. Multi-iteration experience internalization can exhibit “progressive capability collapse rather than compounding improvement” when instance-level experience, global injection, or on-policy context-distillation are used. Reported analyses show instance-level experience giving an initial boost and then falling below base by iteration 3, global injection degrading from 21.0% to 8.5% on WebWalkerQA across iterations, and on-policy internalization causing student turns to jump from 2.5 to 21.9, whereas principle-level, step-wise, off-policy internalization remains stable (Chen et al., 3 Jun 2026). This suggests that the design of the harvesting loop is not separable from the design of the learning regime.

Memory growth and repository contamination are also recurring issues. FinAcumen’s gains per 100 new entries follow a diminishing-returns curve that saturates beyond about 1800 entries, and ablation of retrieval-time deduplication degrades performance (Guo et al., 16 Jun 2026). SkeMex addresses growth through explicit “Govern” operations—merge, promote, deprecate, and capacity-based removal—while Live-Evo relies on reinforcement and decay of experience weights so that misleading or stale experiences gradually fall below the retrieval threshold (Sun et al., 8 Jun 2026, Zhang et al., 2 Feb 2026). GSEM similarly keeps original content fixed but calibrates node quality $s=f(T)=(\phi(T),\psi(T))$9 and edge weights $\phi(T)$0 from online feedback, refining reliability without rewriting the stored experience itself (Han et al., 23 Mar 2026).

The resulting picture is not that self-evolving experience harvesting is a single method, but that it is an organizing principle spanning memory abstraction, feedback attribution, selective retrieval, runtime gating, repository governance, and, in some systems, parameter internalization. The surveyed work consistently treats raw trajectories as an intermediate substrate rather than the final product of learning, and locates long-horizon improvement in the quality of the transformations from trajectory to reusable experience, from experience to action, and from repeated reuse to durable capability (Cai et al., 26 Aug 2025).