Experience Distillation Overview

• Experience distillation is the process of converting raw interaction trajectories into streamlined, persistent knowledge structures that improve decision-making.
• It leverages methods such as self-distillation, structured memory extraction, and multi-level retrieval to reduce noise and optimize policy transfer.
• Empirical studies in image generation, reinforcement learning, and tutorial creation demonstrate significant gains in efficiency and strategic abstraction.

Experience distillation is the transformation of raw interaction history into compact, reusable artifacts that preserve the decision-relevant content of experience while reducing trajectory noise, storage burden, or train–inference mismatch. In current research, the term covers several distinct but related operations: converting tool-using rollouts into structured decision guides for privileged self-distillation in image generation, compressing untrimmed human computer-use traces into multimodal tutorials and task graphs, distilling an agent’s memory lifecycle into deployable selection and writing abilities, summarizing past reasoning into retrieved “lessons,” and compressing replay into proxy memories or minimal boundary sets for efficient policy transfer (Chen et al., 20 May 2026, Bai et al., 2 Jun 2026, Zhang et al., 16 Jun 2026, Shen et al., 6 Apr 2026).

1. Conceptual scope and defining properties

Across recent work, experience distillation is defined less by a single algorithm than by a common transformation: transient trajectories are re-expressed as persistent knowledge structures that are easier to reuse than the original traces. In GenEvolve, agent rollouts that use search, image retrieval, internal generation skills, and prompt programming are abstracted into structured “visual experience” and then distilled back into the policy with dense token-level supervision (Chen et al., 20 May 2026). In Demo2Tutorial, synchronized screen frames and interaction logs are mapped by $\phi: X \to (G, T)$ into hierarchical task graphs and multimodal tutorials (Bai et al., 2 Jun 2026). In OPD-Evolver, the distilled object is broader still: the full lifecycle of how an agent selects, uses, writes, and manages experience is converted into deployable capability through on-policy self-distillation (Zhang et al., 16 Jun 2026).

A recurrent misconception is that experience distillation is simply knowledge distillation under another name. Several papers explicitly separate the two. EDV defines experience distillation as third-party, cross-trajectory construction of reusable memory items and states that it does not alter model weights or train a student model (Zhu et al., 23 Jun 2026). By contrast, $\mathcal{X}$-KD reframes distillation as learning in the teacher’s original learning environment and augments standard KD with inferred reward consistency, while EEKD distills from a teacher’s saved training checkpoints rather than only from its terminal outputs (Cai et al., 13 Feb 2026, Wang et al., 2022). This suggests that the modern literature uses “experience” to denote not only samples or logits, but also optimization history, strategic hindsight, tool-use trajectories, and curated memory artifacts.

A second misconception is that distillation always means compressing experience into text for retrieval. Recent systems split between parametric and non-parametric regimes. GenEvolve, OPD-Evolver, the prompting-policy framework, and $\mathcal{X}$-KD all update model parameters from distilled experience (Chen et al., 20 May 2026, Zhang et al., 16 Jun 2026, Sayana et al., 14 May 2026, Cai et al., 13 Feb 2026). Demo2Tutorial, EDV, Steve-Evolving, Decocted Experience, Mem$^2$Evolve, and EE-MCP instead store distilled artifacts as external structures—tutorials, memories, skills, guardrails, lessons, or rule banks—then retrieve them at inference time (Bai et al., 2 Jun 2026, Zhu et al., 23 Jun 2026, Xie et al., 13 Mar 2026, Shen et al., 6 Apr 2026, Cheng et al., 13 Apr 2026, He et al., 10 Apr 2026).

2. Representational forms

The representational choice determines what is retained from experience and what can later be acted upon. Some systems retain procedural abstractions, others retain decision constraints, and others encode distillation targets directly as distributions.

The following forms recur across the literature (Chen et al., 20 May 2026, Bai et al., 2 Jun 2026, Zhang et al., 16 Jun 2026, Wu et al., 17 Oct 2025, Xie et al., 13 Mar 2026, Cheng et al., 13 Apr 2026, Shen et al., 6 Apr 2026):

Work	Distilled unit	Structure
GenEvolve	Visual experience bundle	search, know, ref, prompt, fail
Demo2Tutorial	Task graph and tutorial	goal, chapter, step; image–text steps
OPD-Evolver	Multi-tier memory	traj, tip, skill, tool
EvolveR	Principle	description, triples, type, score
Steve-Evolving	Skill / guardrail	preconditions and verification / triggers and forbidden actions
Mem$^2$Evolve	Experience Memory item	title, description, use cases, core knowledge
Decocted Experience	Lesson	task description, strategy or workflow, pitfalls

These forms are not interchangeable. GenEvolve’s bundle is explicitly tool-oriented and references prompt wording, ordinal reference binding, and skills such as spatial_layout or text_rendering (Chen et al., 20 May 2026). Demo2Tutorial’s representation is pedagogical: the task graph $G=(V,E)$ encodes prerequisite structure, while the tutorial $T$ interleaves step text with selected frames, OCR-grounded annotations, and visual highlights (Bai et al., 2 Jun 2026). EvolveR’s principle is textual-symbolic, combining a single-sentence strategic description with structured triples and dynamic metadata such as usage count and success score (Wu et al., 17 Oct 2025). Steve-Evolving splits successful experience into reusable skills and failures into executable guardrails, so positive and negative knowledge are represented by different schemas (Xie et al., 13 Mar 2026).

Older distillation work used narrower representations. PPO distillation stores observations and teacher action probabilities in a replay buffer and trains the student with policy KL (Green et al., 2019). FRD and Proxy Experience Replay compress distributed RL experience into proxy state clusters and time-averaged policies (Cha et al., 2019, Cha et al., 2020). BCMER retains only boundary experiences near action decision boundaries, then uses nearest-neighbor rules for interpretable policy distillation (Liu et al., 2022). EEKD stores uniformly spaced intermediate teacher checkpoints and learns sample-dependent attention weights over their softened predictions (Wang et al., 2022). The shift from these earlier representations to modern memory artifacts marks a broadening from policy imitation toward strategic abstraction.

3. Distillation mechanisms and learning objectives

One major family of methods distills experience by comparing trajectories and turning their differences into denser supervision than scalar reward alone. GenEvolve samples $K$ trajectories for the same request, scores them with a mixed reward $$R=(1-\alpha)R_{\mathrm{img}}+\alpha R_{\mathrm{text}}$$ with $\alpha=0.5$, identifies $\mathcal{X}$0 and $\mathcal{X}$1, and summarizes best–worst differences into a five-slot bundle when $\mathcal{X}$2 with $\mathcal{X}$3 (Chen et al., 20 May 2026). The teacher sees $\mathcal{X}$4, the student sees only the plain context, and self-distillation uses an importance-weighted sampled-token reverse-KL on the top 10% disagreement tokens, combined with GRPO through $\mathcal{X}$5 and $\mathcal{X}$6.

A second family distills across the full experience lifecycle rather than only execution. OPD-Evolver runs a fast loop over a four-level repository and a slow loop that computes outcome-calibrated memory attribution

$\mathcal{X}$7

then defines $\mathcal{X}$8 and uses token-level KL on student-visited prefixes to distill selection, execution grounding, writing, and maintenance simultaneously (Zhang et al., 16 Jun 2026). Here, the privileged teacher sees hindsight about which memories were actually valuable, while deployment uses only the student-facing inputs.

A third family distills without parametric student updates by turning raw experience into externally consumable structure. Demo2Tutorial first aligns visual and interaction streams with a temporal pivot, uses a VLM-based Action Parser to reconstruct observation, difference, action, and intent, then groups atomic actions into steps and chapters through a Step Planner with actor–critic refinement, and finally composes multimodal tutorials through a key-frame objective

$\mathcal{X}$9

with SAM2 and RapidOCR-based highlighting (Bai et al., 2 Jun 2026). EDV similarly externalizes experience, but does so through role separation: heterogeneous executors generate trajectories, a third-party distiller selects the single best trajectory under the priority order Compliance $\mathcal{X}$0 Accuracy $\mathcal{X}$1 Efficiency, and verifiers apply a default-reject consensus gate before insertion into shared or private memory (Zhu et al., 23 Jun 2026).

A fourth family treats experience distillation as prompt-policy or teacher-environment learning. The prompting-policy framework trains a small Prompter to internalize multi-turn corrective prompt refinement using a contrastive buffer of prompts, rewards, and dense critiques; the worker model remains frozen, and the buffer conditions the next prompt without any explicit InfoNCE objective (Sayana et al., 14 May 2026). $\mathcal{X}$2-KD augments sequence-level, forward-KL, or JS-based KD with an experiential regularizer derived from AVRIL, so that conventional KD becomes

$\mathcal{X}$3

where $\mathcal{X}$4 enforces consistency between the student’s TD errors and an inferred reward posterior for the teacher’s original environment (Cai et al., 13 Feb 2026).

4. Memory, retrieval, and verification

Because distilled experience is usually stored outside the immediate trajectory, retrieval policy becomes part of the method itself. GenEvolve attaches each experience bundle to its source prompt with Qwen3-Embedding-0.6B keys, retrieves the nearest source prompt by cosine similarity, gates insertion into teacher context at similarity $\mathcal{X}$5, and keeps a buffer of 500 bundles with FIFO plus reward-gap eviction (Chen et al., 20 May 2026). EvolveR builds a repository $\mathcal{X}$6 of principles, indexes them with BGE-M3, merges semantically equivalent entries using embedding prefiltering with $\mathcal{X}$7 plus an LLM equivalence judge, tracks utility by

$\mathcal{X}$8

retrieves top-$\mathcal{X}$9 principles at inference, and prunes entries with $^2$0 where $^2$1 (Wu et al., 17 Oct 2025).

Several systems organize experience hierarchically. Decocted Experience stores per-problem lessons in flat memory, then consolidates them by k-means over joint embeddings of problem statements and lessons, and finally builds a hierarchical concept tree that supports multi-leaf retrieval and LLM re-ranking (Shen et al., 6 Apr 2026). Steve-Evolving stores anchored experiences in a three-tier space of document, index, and summary, with indices

$^2$2

and rolling summarization for auditable recall (Xie et al., 13 Mar 2026). OPD-Evolver similarly separates memory by reuse granularity—traj, tip, skill, tool—and performs retrieval top-$^2$3 per tier with $^2$4, periodic maintenance every $^2$5 tasks, and teacher contexts restricted to at most 20 memories (Zhang et al., 16 Jun 2026).

Verification policies vary sharply. EDV routes unanimously approved items into shared memory and partially approved items into private memory, then retrieves from shared memory at similarity threshold $^2$6 and from private memory at threshold $^2$7 (Zhu et al., 23 Jun 2026). EE-MCP avoids free-form memory growth by storing experience-bank rules under skill categories $^2$8 and filtering them by application type before prompt augmentation (He et al., 10 Apr 2026). Mem$^2$9Evolve splits long-term state into Asset Memory and Experience Memory, so distilled experience can guide both future execution and future asset creation rather than only retrieval-time planning (Cheng et al., 13 Apr 2026). This suggests that memory design is not ancillary: in many systems, it is the operational substrate through which experience becomes reusable.

5. Empirical domains and reported effects

Experience distillation has been evaluated across image generation, software tutorials, lifelong memory agents, reasoning, tool use, embodied control, classical RL distillation, and distributed RL. In open-ended image generation, GenEvolve with Qwen-Image-Edit reaches KScore 0.3663 on GenEvolve-Bench versus 0.3493 for Gen-Searcher 8B, and with Nano Banana Pro reaches 0.5739 versus 0.5298 for raw Nano Banana Pro and 0.5481 for Gen-Searcher+Nano; its WISE score of 0.82 exceeds GPT-4o at 0.80 and agentic baselines including GenAgent 0.72 and Gen-Searcher-8B 0.77 (Chen et al., 20 May 2026). In multimodal tutorial generation, Demo2Tutorial attains overall 86.2 on TutorialBench versus 79.1 for human-authored GT and 64.3–70.3 for baselines, reduces task completion time from 147.1s for demos to 131.6s for tutorials, and improves OSWorld planning in Agent-S3, for example from 52.9% to 70.6% in Chrome with GPT-5 (Bai et al., 2 Jun 2026).

In lifelong or memory-centric agents, OPD-Evolver is reported as the top memory system on 10/10 subsets for both 4B and 9B backbones, surpassing memory systems by up to 11.5% and training-based methods by about 5.8%; OPD-Evolver-9B beats Step-3.5-Flash on 9/10 subsets and exceeds Qwen3.5-397B-A17B on 6/10 subsets (Zhang et al., 16 Jun 2026). EvolveR reaches the best average Exact Match of 0.382 at 3B across NQ, HotpotQA, TriviaQA, PopQA, 2Wiki, MuSiQue, and Bamboogle, and removing experience retrieval drops the 3B average from 0.382 to 0.340 (Wu et al., 17 Oct 2025). EDV attains 86.6 average Pass@1 on $^2$0-bench as an ensemble, ahead of Router at 83.5 and Judge at 81.5, while human audit scores its memories higher than ReasoningBank on groundedness/correctness, actionability, and specificity and lower on noise/hallucination and potential harm (Zhu et al., 23 Jun 2026). Steve-Evolving raises Overall SR on Minecraft MCU, for example to 52.52 with Qwen3.5-plus versus 47.42 for Optimus-1 and 42.59 for Jarvis-1, with especially large gains in Iron, Diamond, and Armor (Xie et al., 13 Mar 2026).

In reasoning and prompt optimization, decocted experience improves both effectiveness and efficiency relative to vanilla context construction; relevant lessons yield higher information gain than random context, with mean 172.2 versus 97.2 and wins on 76% of cases, while conditional entropy and output length show Pearson $^2$1 (Shen et al., 6 Apr 2026). The prompting-policy framework improves logic-intensive reasoning from about 55% to about 90% and tool-use from about 74% to about 91%, with buffer-based distillation reducing Dyck convergence steps from 195 to 102 and DQA convergence steps from 180 to 75 (Sayana et al., 14 May 2026). EE-MCP shows application-aware asymmetry: on Chrome, distillation plus experience reaches 77.8% pass rate, up 17.8 percentage points from baseline 60.0%, whereas on VS Code the experience bank alone reaches 63.3%, up 10.0 percentage points over 53.3%, and distillation hurts to 43.3% (He et al., 10 Apr 2026). Mem$^2$2Evolve reports average Pass@1 of 70.24%, improving by 18.53% over naive LLMs, 11.80% over experience-centric evolution, and 6.46% over capability-centric evolution (Cheng et al., 13 Apr 2026).

Classical distillation and RL work show earlier forms of the same idea. EEKD improves CIFAR-100 and ImageNet students by ensembling uniformly spaced teacher checkpoints, and its central empirical claim is that strong ensemble teachers do not necessarily produce strong students (Wang et al., 2022). PPO distillation trains smaller students from teacher action probabilities and reports that a medium-capacity distilled student fine-tuned for 20$^2$3 timesteps reaches 100% of teacher and 169% of DQN, while a low-capacity distilled and tuned student reaches 151% of DQN (Green et al., 2019). BCMER retains only 1.4%–19.1% of a 10k experience pool while maintaining high interpretable policy-distillation performance (Liu et al., 2022). FRD and Proxy Experience Replay show that proxy states and time-averaged policies can reduce communication and privacy exposure in distributed RL, with MixFRD reported to reduce variance of mission completion time and communication cost relative to FRD, FRL, and PD in Cartpole (Cha et al., 2019, Cha et al., 2020).

6. Limitations, controversies, and open directions

The dominant limitation is experience quality. GenEvolve depends on reliable search, image retrieval, and judge diagnostics; noisy search or misleading references can yield misleading experience, and only cases with reward gap $^2$4 contribute memory (Chen et al., 20 May 2026). Demo2Tutorial inherits parser misgrounding, step-boundary ambiguity, OCR and segmentation sensitivity, and privacy concerns from raw screen recordings (Bai et al., 2 Jun 2026). Decocted Experience shows that performance is non-monotonic with memory size: too small loses essential information, too large reintroduces redundancy and reduces diversity (Shen et al., 6 Apr 2026). EvolveR notes that noisy or spurious principles can bias later reasoning, so deduplication, scoring, and pruning are required (Wu et al., 17 Oct 2025).

A second limitation is attribution reliability. OPD-Evolver’s memory value estimates depend on scalar returns and candidate-controlled comparisons, which become noisy under sparse rewards or short horizons (Zhang et al., 16 Jun 2026). EDV mitigates the Self-Confirmation Trap by decoupling execution, distillation, and verification, yet it explicitly acknowledges consensus bias if heterogeneous agents share the same failure mode, and interference from low-quality agents during verification (Zhu et al., 23 Jun 2026). Steve-Evolving relies on rule-based diagnosis and an enumerated failure taxonomy, so portability requires environment-specific signal engineering (Xie et al., 13 Mar 2026). EE-MCP shows that the same mechanism can help in one application and hurt in another, which argues against a single default choice between distillation and retrieval-time experience augmentation (He et al., 10 Apr 2026).

A third issue is computational and systems overhead. Long trajectories, multi-turn tool calls, memory summarization, embedding retrieval, or offline construction pipelines add cost even when deployment remains parameter-free. GenEvolve uses responses up to 30,000 tokens with tool budgets of 11 per rollout (Chen et al., 20 May 2026). Demo2Tutorial reports approximately 259k input tokens, approximately 3k output tokens, approximately 368 s, and approximately \$1.36 API cost per tutorial (Bai et al., 2 Jun 2026). OPD-Evolver and EDV both shift substantial work into offline or slow-loop construction (Zhang et al., 16 Jun 2026, Zhu et al., 23 Jun 2026). In distributed RL, FRD reduces communication by proxy compression, but its effectiveness depends on cluster design and exchange interval (Cha et al., 2019, Cha et al., 2020).

Open directions in the literature remain broad but technically coherent. Several papers suggest transferring the same idea to other modalities or agentic substrates: GenEvolve proposes non-visual extensions such as code or audio generation (Chen et al., 20 May 2026); Demo2Tutorial proposes vision-only inference from arbitrary screen videos and broader platform coverage (Bai et al., 2 Jun 2026); Mem$^2$5Evolve calls for stronger memory pruning and capability governance (Cheng et al., 13 Apr 2026); $^2$6-KD suggests that experiential regularization can remain within ordinary supervised distillation pipelines while modeling the teacher’s original reward environment (Cai et al., 13 Feb 2026). Taken together, these works indicate that experience distillation is evolving from a narrow compression metaphor into a general design principle for converting interaction history into reusable strategic structure.