Experience-Based Skill Distillation

• Experience-based skill distillation is a machine learning paradigm that extracts transferable skills from trajectories and intermediate model snapshots.
• It leverages diverse sources of experiential data, including teacher snapshots and demonstration trajectories, to capture richer behavioral insights.
• Empirical implementations have shown improvements in vision tasks, reinforcement learning, and multi-agent systems with measurable performance gains.

Experience-based skill distillation refers to a class of machine learning techniques that extract, represent, and transfer actionable knowledge—“skills”—from historical trajectories, intermediate models, or expert demonstrations, emphasizing the learning process rather than merely the final model parameters. Unlike classical knowledge distillation, which typically leverages only the static output of a fully trained teacher model, experience-based approaches capture richer behavioral information accumulated throughout training or interaction episodes. This paradigm encompasses methods in supervised learning, reinforcement learning, continual and interpretable policy distillation, and LLM self-improvement, unified by their reliance on leveraging trajectories or process-derived experience to drive efficient, robust skill transfer and accumulation.

1. Methodological Foundations

Experience-based skill distillation mechanisms capitalize on the diversity and structure present in agent experience, whether in the form of teacher model snapshots, successful/failed episodes, or expert-annotated demonstrations. Techniques vary in the manner experience is extracted, distilled, and operationalized for student training:

• Ensemble of Intermediate Models: Experience Ensemble Knowledge Distillation (EEKD) constructs a virtual teacher by uniformly selecting $M$ intermediate snapshots $\{\theta_i^t\}_{i=1}^M$ from the teacher's training trajectory. Instead of using only the converged teacher, EEKD ensembles the softened predictions $p_i = \mathrm{softmax}(z_i/T)$ of each snapshot, where $z_i$ denotes logits and $T > 1$ is a temperature. Aggregation weights $\alpha_i$ are computed dynamically per input via a self-attention mechanism, yielding a composite soft target $p_{\mathrm{ensemble}} = \sum_{i=1}^M \alpha_i p_i$ for student distillation (Wang et al., 2022).
• Skill Extraction from Trajectories: In frameworks such as SkillRL, successful and failed trajectories under a base policy are processed by a fixed high-capacity teacher model to extract succinct “skill” snippets characterizing actionable high-level knowledge. Distilled skills are clustered into general and task-specific sets, forming a hierarchical SkillBank used to guide further learning and recursive refinement (Xia et al., 9 Feb 2026).
• Boundary Experience Retention: BCMER discards the vast majority of internal experiences and retains only those lying near decision boundaries, using a Multidimensional Hyperspheres Intersection (MHI) algorithm. This process ensures that the distilled set maximally preserves policy interpretability and fidelity despite aggressive pruning, focusing on trajectories most informative for skill delineation (Liu et al., 2022).
• Latent Skill Priors: SPiRL learns a variational latent skill space and trains a state-conditional skill prior from agent trajectories. In downstream tasks, a skill-prior-regularized policy exploits this knowledge, biasing exploration toward empirically grounded skills (Pertsch et al., 2020).
• Self-Distillation from Demonstration Experience: Continuous acquisition and transfer of behavioral priors is achieved via self-distillation from demonstration-derived experience, leveraging a demonstration-conditioned teacher (typically an EMA of the current model) to guide on-policy rollouts, thereby mitigating distributional shift in sequential or continual learning settings (Shenfeld et al., 27 Jan 2026).

2. Algorithmic Components and Loss Structures

Experience-based skill distillation algorithms are characterized by distinctive loss functions and training regimes that incorporate experience trajectory data:

• Weighted Ensemble Distillation Loss (EEKD):

$$L_\mathrm{total} = (1 - \lambda)L_\mathrm{CE} + \lambda T^2 KL(p_\mathrm{ensemble} \| \mathrm{softmax}(z^s / T))$$

where $p_\mathrm{ensemble}$ is the adaptive attention-weighted ensemble of intermediate teacher snapshots. The attention weights $\alpha_i$ are derived from compatibility between projected feature vectors of teacher and student activations.

• DKL-Regularized Policy Optimization (SPiRL):

$$J(\theta) = \mathbb{E}_\pi \left[ \sum_{t} \gamma^t ( \tilde{r}(s_t, z_t) - \alpha KL[\pi(z_t | s_t) \| p_p(z_t | s_t)] ) \right]$$

where $\tilde{r}$ denotes horizon-aggregated rewards and $p_p(z|s)$ the learned skill prior (Pertsch et al., 2020).

• Skill-based Data Selection and Curriculum: In data-efficient reasoning distillation, student skill weaknesses $w_k$ (inverse of per-skill accuracies $a_k$) define a sampling distribution for assembling fine-tuning datasets tailored to the student’s deficiency profile (Zhang et al., 15 Jan 2026).
• Self-Distillation with Reverse-KL (SDFT):

$$L_{\mathrm{SDFT}}(\theta) = \alpha \mathbb{E}_{(x,c)\sim \mathcal{D}_i}[\ell_{\mathrm{new}}(\theta; x, c)] + (1-\alpha) \mathbb{E}_{(x,c), y\sim\pi_\theta} [D_{KL}(\pi_\theta(\cdot|x) \| \pi_\phi(\cdot|x, c))]$$

integrating off-policy imitation and on-policy self-distillation to balance new skill acquisition and preservation of prior capabilities (Shenfeld et al., 27 Jan 2026).

• Minimal Experience Retention: BCMER’s rule induction and policy distillation leverage only the experience points at the boundaries of the teacher’s decision surface, reducing the dataset size by factors of $5-50$ with negligible accuracy loss (Liu et al., 2022).

3. Skill Representation and Abstraction

Skill distillation from experience necessitates the choice of skill representations. These span:

• Latent Codes in a Variational Space: SPiRL encodes skill segments as continuous latent vectors, ensuring that skills capture temporally extended behavioral motifs while supporting probabilistic inference and flexible prior regularization (Pertsch et al., 2020).
• Normalized Programs with Interfaces: Audited Skill-Graph Self-Improvement (ASG-SI) formalizes each skill as a program $P: I \rightarrow O$, with explicit input/output schemas and logical pre- and post-conditions, verified under deterministic replay. This enables compositionality, interface safety, and auditability (Huang et al., 28 Dec 2025).
• Semantic Skill Snippets and Hierarchical SkillBanks: SkillRL stores extracted skills as text-based records partitioned into general-purpose and category-specific layers. Each record includes metadata (title, principle, applicability) and is dynamically retrieved during inference (Xia et al., 9 Feb 2026).
• Boundary Experience Points and Interpretable Rule Lists: BCMER reduces the skill characterization task to a set of nearest-neighbor-based rules or geometric regions, with each retained trajectory directly inducing a local policy rule (Liu et al., 2022).

4. Practical Implementations and Empirical Validation

Multiple empirical studies demonstrate the impact and efficiency of experience-based skill distillation:

• Vision Classification: EEKD, when applied to CIFAR-100 and ImageNet-1K, consistently outperforms classical knowledge distillation and state-of-the-art variants, with 1–2% gains across student-teacher pairs, occasionally enabling the student to surpass the teacher’s own accuracy. Notably, using intermediate teacher models with adaptive weighting enables more effective student distillation than constructing higher-accuracy ensembles from independent teachers, at reduced training cost (Wang et al., 2022).
• Reinforcement Learning and Embodied Control: SPiRL achieves efficient transfer on navigation/manipulation tasks with skill horizons $H=8$–$12$, tolerating substantial demonstration noise and enabling re-use of priors across related goals (Pertsch et al., 2020). In articulated control, PLAiD integrates expert policies sequentially into a single student via DAGGER-style replay, preserving performance across skills while facilitating incremental generalization and avoiding catastrophic forgetting (Berseth et al., 2018).
• LLM Continual Learning: SDFT allows sequence models to acquire new skills from demonstration corpora without catastrophic forgetting. In multi-skill sequences, only on-policy self-distillation with demonstration conditioning maintains high accuracy across all tasks, outperforming supervised fine-tuning and off-policy variants in both skill learning and knowledge acquisition regimes (Shenfeld et al., 27 Jan 2026).
• Skill Data Selection for Reasoning: An experience-based selection curriculum—sampling more from skills where the student is weakest—improves average accuracy by 1.4–1.6% over random fine-tuning on multiple mathematical reasoning benchmarks, concentrating gains on previously poorly mastered skills (Zhang et al., 15 Jan 2026).
• Skill Library Construction for Multi-agent Systems: In SkillRL, recursive distillation and evolution of a structured SkillBank yield compressed (10–20× smaller) experience footprints and 12.3% higher success rates on ALFWorld relative to baselines. Adaptive skill retrieval and continual incorporation of new failure-driven skills robustify generalization and maintain compact prompts (Xia et al., 9 Feb 2026).

5. Security, Auditability, and Operational Guarantees

Experience-based distillation, particularly in agentic LLMs and safety-critical applications, raises issues of verifiability and governance:

• Skill Graph Verification and Audit: ASG-SI integrates a formal trajectory logging, skill extraction, contract-checking harness, and cryptographically signed promotion pipeline. Every skill is validated via deterministic replay under held-out test suites, and promotion decisions, rewards, and state updates are reconstructible and tamper-evident. This operationalizes reproducibility, interpretable credit assignment, and security boundary enforcement (Huang et al., 28 Dec 2025).
• Memory Control and Continual Performance: Long-horizon agents undergoing experience-based distillation must manage bounded context. ASG-SI specifies continual memory control regimes, with explicit update and retrieval protocols, and constraints are stress-tested via synthesis of adversarial tasks to ensure temporal credit assignment is robust to context loss (Huang et al., 28 Dec 2025).

6. Limitations and Extensions

Experience-based skill distillation presents unique challenges and open directions:

• Model Capacity and In-Context Learning: Self-distillation methods such as SDFT require sufficient model capacity and strong in-context learning abilities; smaller models do not reliably self-distill (Shenfeld et al., 27 Jan 2026).
• Combinatorial Complexity in Boundary Experience: BCMER’s experience pruning scales poorly in very high-dimensional spaces; approximate nearest-neighbor methods or embedding-based action distances are necessary for scalability (Liu et al., 2022).
• Computational Cost: Generating and storing experience (particularly with skill attribution) can be computationally intensive, but is amortized across many student models and seeds. SkillRL and SDFT both report 2–2.5× higher training FLOPs versus standard SFT, reflecting the added complexity of on-policy or meta-distillation steps (Xia et al., 9 Feb 2026, Shenfeld et al., 27 Jan 2026).
• Potential Research Extensions: Hybridization with reward-based RL, noisy demonstration learning, dynamic curriculum adaptation, and richer task/skill interface representations are all identified as promising for further advances (Shenfeld et al., 27 Jan 2026, Huang et al., 28 Dec 2025).

7. Representative Methods and Experimental Results

Method / Reference	Experience Source	Skill Representation	Select Results / Findings
EEKD (Wang et al., 2022)	Teacher model training path	Attention ensemble logits	1–2% accuracy gains on vision tasks
SkillRL (Xia et al., 9 Feb 2026)	Trajectory→SkillBank	Text skill/heuristic	12.3% higher ALFWorld success, 10% context reduction
SPiRL (Pertsch et al., 2020)	Offline RL trajectories	Latent skill prior	Boosted transfer/success in navigation/manip
BCMER (Liu et al., 2022)	Experience pool, policy actions	Boundary points/rules	80–98% data reduction, $<$5% reward loss
SDFT (Shenfeld et al., 27 Jan 2026)	Demonstration context, on-policy	Model-generated trajectories	Preserves old skills, +3.5–38% accuracy gains
Skill-Aware Distillation (Zhang et al., 15 Jan 2026)	Student eval + demo pool	Weak-skill sampling	+1.6% reasoning accuracy on Qwen3-4B/8B

These and related methods show that diverse experience representations—if carefully structured, curated, and distilled—can drive compact, robust, and verifiable skill acquisition across a spectrum of learning systems.