SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

Abstract: LLMs are increasingly trained as interactive agents for long-horizon tasks involving multi-turn interaction, tool use, and environment feedback. Outcome-based reinforcement learning (RL) provides a practical optimization paradigm, but its sparse trajectory-level rewards offer limited guidance on intermediate decisions, leaving a supervision gap between episode-level outcomes and token-level policy learning. We propose SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills and distills their behavioral effect back into the policy model. SEED first fine-tunes the policy to analyze completed trajectories and generate natural-language skills that capture reusable workflows, decisive observations, or failure-avoidance rules. During RL, the current policy both collects trajectories and serves as the analyzer that extracts hindsight skills from them. Policy updates therefore improve subsequent decision making and skill analysis together, allowing hindsight supervision to evolve with the policy. SEED then re-scores the sampled actions under ordinary and skill-augmented contexts, converting the skill-induced probability shift into a dense token-level on-policy distillation signal. This signal is jointly optimized with outcome-based RL, keeping the auxiliary supervision aligned with the current trajectory distribution. Extensive experiments on text-based and vision-based agentic tasks show that SEED consistently improves performance and sample efficiency, exhibiting robust generalization to unseen scenarios. Our code is available at https://github.com/jinyangwu/SEED.

• The paper introduces a self-evolving distillation method that transforms on-policy trajectories into dense token-level skill-based supervision.
• It leverages synchronized policy-analyzer duality to ensure adaptive credit assignment and significantly boost performance and sample efficiency.
• Experimental results show notable gains across ALFWorld, WebShop, and search-based QA tasks, with improved generalization in data-sparse regimes.

SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

Introduction and Motivation

The shift of LLMs toward agentic, multi-turn tasks introduces unique credit assignment and policy optimization issues under sparse, delayed, trajectory-level RL rewards. While outcome-based RL is the de facto optimization paradigm for such agents, its supervision is too coarse for high-fidelity, token-level policy improvement. Research has shown that completed interactions often encapsulate rich hindsight that can inform subgoal decomposition, strategy extraction, and error analysis, but prior approaches predominantly utilize static memory or prompt-based skills, resulting in stale or non-adaptive supervision.

Framework Overview

SEED (Self-Evolving On-Policy Distillation) presents a parametric alternative: it converts completed on-policy trajectories into natural-language "hindsight skills" and distills their decision-level impact back into the policy itself. This is achieved by allowing the current RL policy to double as an analyzer, synchronously generating skills from completed trajectories. SEED then contrasts sampled actions under standard and skill-augmented contexts, using log-probability shifts as a dense, token-level self-distillation signal. This auxiliary signal is optimized jointly with the outcome-driven RL loss, ensuring that both trajectory induction and skill generation co-evolve.

Technical Contributions

SEED introduces several key innovations:

• Synchronized Policy-Analyzer Duality: The core policy model acts as both the actor (trajectory sampler) and analyzer (skill generator). Every RL update synchronizes both roles, mitigating the trajectory-skill distribution mismatch.
• Hindsight Skill Extraction and Distillation: Completed trajectories are summarized into transferable skills—guiding rules, workflows, or failure avoidance principles—by the policy-analyzer. In training, the policy's sampled tokens are re-scored under contexts augmented with these skills; the induced probability shift forms a dense, per-token distillation target.
• Occupancy-Matched, Self-Evolving Supervision: Unlike prior work depending on offline teachers or static datasets, SEED ensures the skill signal is always on-policy. The auxiliary distillation signal adapts alongside the agent, yielding dense, context-specific credit assignment even under tied or uninformative terminal rewards.
• Token-Level Gradient Gating: The magnitude of auxiliary supervision for each token is adaptively scaled based on teacher-student logit differences and a sigmoid gate, with positive skill support amplifying the learning, while weak or negative support attenuates it.

Experimental Results

SEED is evaluated on text-based embodied environments (ALFWorld), web navigation (WebShop), and search-augmented QA spanning multiple open-domain datasets. Strong numerical results are established:

• Performance: On ALFWorld, SEED achieves macro-averaged success rates up to 91.8% (Qwen2.5-3B), surpassing prior outcome-only RL (GRPO) by 14.9-45.9 points and all static distillation baselines. Gains of 8.7-19.8 points in WebShop normalized scores and up to 11.6 points in search-based QA accuracy are reported.
• Sample Efficiency: SEED matches or exceeds GRPO accuracy with as little as 40-60% of the training data, indicating substantially higher reward per sample due to auxiliary skill-based supervision.
• Generalization: In out-of-domain splits (ALFWorld Unseen), SEED improves success by +15.3 macro points over GRPO, with the largest gains in data-sparse families, establishing that internalized skill distillation leads to more robust, transferable policy representations compared to prompt-based or static methods.
• Ablations: Each main SEED component is critical—removing synchronized OPD, skill SFT, or on-policy skill generation significantly degrades performance (e.g., −7.4 macro points for offline skill replay).
• Multimodal Extension: Applying SEED in visual domains (e.g., Sokoban with vision-language backbones) yields absolute gains of up to 14.9 points in success rate, confirming the generality of self-evolving hindsight distillation.

Theoretical Properties

Three main properties are formally established:

1. Occupancy-matched target: On-policy collection ensures the auxiliary gradient aligns with the actual policy distribution, mitigating mismatch and staleness.
2. Dense, decision-specific signal: The OPD term remains informative even in the absence of trajectory-level reward differences (reward ties), supporting fine-grained credit assignment.
3. Staleness control via synchronization: Refreshing the analyzer every policy update bounds the divergence between skill-induced gradients and current policy behavior, preventing accumulation of outdated supervision.

Practical and Theoretical Implications

The SEED framework delivers a mechanism for continuous, self-adaptive shaping of agentic policies, internalizing strategy abstractions or error-corrective rules at a behaviorally meaningful granularity—without requiring skills at inference or relying on external evaluators. This decouples trajectory-level reward sparsity from learning efficiency and reduces deployment overhead. The self-evolving OPD paradigm enhances addressability of local credit assignment, crucial for compositional, long-horizon reasoning, and manipulation tasks.

However, reliance on self-generated skills introduces risks of systematic reinforcement of flawed rules (self-preference), particularly if model errors are consistently interpreted as successful strategies. Future directions include integrating verifiable state-change analysis, uncertainty quantification for hindsight gating, and cross-policy or cross-snapshot skill validation.

In terms of broad impact, the method offers a scalable avenue for policy optimization in open-ended environments, enables rapid adaptation to novel states, and can be extended to temporally hierarchical learning or multi-agent policy distillation regimes.

Conclusion

SEED establishes a unified, parametric framework for unifying sparse reward optimization with dense, hindsight-grounded supervision in agentic RL. By synchronizing policy and analysis roles, dynamically generating and distilling skills, and supporting dense, token-level credit signals, SEED demonstrates major empirical gains in sample efficiency, robustness, and generalization on diverse language- and vision-centric long-horizon tasks. The approach delineates an effective path toward scalable, self-improving agents capable of sophisticated, transferable reasoning without reliance on hand-engineered credit assignment or deployment-time skill provisioning (2607.14777).