OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning

Abstract: Outcome-based reinforcement learning provides a stable optimization backbone for language agents, but its sparse trajectory-level rewards provide little guidance on which intermediate decisions should be reinforced or suppressed. On-policy self-distillation offers dense token-level supervision, yet existing skill-conditioned variants often rely on external skill memories or retrieved privileged context, which are costly to maintain and can be mismatched with the state distribution induced by the current policy in multi-turn interaction. We propose \textbf{OPID} (\textbf{O}n-\textbf{P}olicy Sk\textbf{i}ll \textbf{D}istillation), a framework that extracts skill supervision directly from completed on-policy trajectories. OPID represents trajectory hindsight as hierarchical skills: episode-level skills capture global workflows or failure-avoidance rules, while step-level skills capture local decision knowledge at critical timesteps. A critical-first routing mechanism uses step-level skills when critical decisions are identified and falls back to episode-level skills as default guidance otherwise. The selected skill is injected into the interaction history, allowing the old policy to re-score the same sampled response under both original and skill-augmented contexts. The resulting log-probability shift yields a token-level self-distillation advantage, which is combined with the outcome advantage for policy optimization. OPID thus preserves RL as the primary training objective while introducing dense, distribution-matched hindsight supervision. Experiments on ALFWorld, WebShop and Search-based QA demonstrate that OPID generally improves agent performance, sample efficiency, and robustness over outcome-only RL and existing skill-distillation baselines. Our code is available at https://github.com/jinyangwu/OPID/tree/main.

• The paper presents a unified on-policy skill distillation framework that extracts hierarchical skills from trajectories to deliver dense credit assignment in agentic RL.
• The methodology employs a critical-first routing strategy to integrate episode- and step-level skills into a clipped PPO objective for effective policy updates.
• Empirical results show up to 13-point gains over baselines, with improved sample efficiency and generalizability across various domains.

OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning

Introduction and Motivation

Long-horizon agentic RL for LLM-based agents poses a fundamental challenge: how to deliver dense, state-matched credit assignment signals under sparse, delayed trajectory-level rewards. Token-level supervision via self-distillation is known to significantly improve sample efficiency and local decision optimality, but most prior skill-conditioned approaches require external skill libraries, adding both operational overhead and distribution mismatch against the current policy’s state occupancy. "OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning" (2606.26790) proposes a unified on-policy distillation framework that systematically extracts hierarchical skills from completed on-policy trajectories and re-injects them as dense hindsight supervision during policy optimization. This method delivers credit assignment that is both contextually aligned and incrementally adaptive, without test-time dependency on external or privileged information.

Methodology

OPID runs in three stages per roll-out group. First, after sampling a batch of trajectories on policy, an LLM-based analyzer parses each completed trajectory into an episode-level skill (i.e., workflow or risk-avoidance summary) and a sparse set of state-dependent step-level skills reflecting critical decisions. The analyzer is only invoked at training time, ensuring full on-policy context alignment.

Second, a critical-first routing strategy selects between the global episode-level skill (default guidance) and precise step-level skills (for states identified as pivotal), ensuring appropriate granularity and mitigating spurious or mismatched guidance. The selected skill is injected into the original interaction history, and the policy re-scores its own sampled actions under both standard and skill-augmented contexts. The difference in log-probabilities yields a local skill advantage, forming a dense, token-level credit assignment signal.

Finally, this skill advantage is linearly combined with the outcome-level group-relative advantage (as in GRPO), and the policy is updated using a clipped PPO objective. This design preserves the primary RL objective while tightly integrating dense on-policy skill distillation, yielding both stable reward-driven optimization and strong local signal. Notably, OPID requires no skill extraction, retrieval, or privileged context at inference.

Empirical Results

Comprehensive evaluation on ALFWorld (embodied household manipulation), WebShop (web-based product search and purchase), and Search-based QA (search-augmented QA) demonstrates three principal findings:

1. Consistent improvements over outcome-only RL baselines: On various Qwen series backbones, OPID achieves up to ~13 point absolute gains in ALFWorld (e.g., 84.3 vs 75.0 on Qwen2.5-3B) and strong gains in WebShop and QA. Notably, gains are robust under data ablation—OPID with 60% data nearly matches or surpasses the baseline RL model at 100%.
2. Competitiveness against strong distillation and hybrid baselines: OPID often matches or outperforms methods that rely on auxiliary skill-conditioned signals, full vocabulary distillation, or hybrid token-level shaping methods, while being strictly on-policy and training-time-only in its use of skills.
3. No inference-time dependence on skills: Methods like Skill-GRPO exhibit severe performance drops when skill prompts are ablated at test time, while OPID internalizes skills, significantly outperforming these baselines in skill-less inference.

Ablations confirm that hierarchical skills (both episode/step) and proper critical-first routing are necessary for peak performance; removing either skill level or indiscriminately combining skills leads to substantial degradation. Cross-domain transfer further shows that OPID’s internalized skills are generalizable, yielding systematic gains on unseen ALFWorld tasks.

Theoretical Analysis

The paper provides a rigorous analysis of OPID’s distillation mechanism. The unclipped skill-distillation objective is shown to be an exact relative-KL loss, locally (i.e., around the current policy) equivalent to scaling a sampled-token reverse-KL distillation update. Critically, because both autoregressive contexts and hindsight skills are extracted on-policy, OPID avoids classical pitfalls of distributional drift seen in off-policy or retrieval-based skill conditioning. The critical-first routing strategy is justified under a specialization assumption—when the analyzer effectively distinguishes critical steps, routing provably recovers the best candidate teacher among hierarchical skills.

Implications and Future Directions

OPID advances the agentic RL landscape by providing a scalable, skill-internalizing training protocol that solely leverages on-policy interactions, sidestepping the maintenance, retrieval, and test-time mismatch issues inherent to external skill libraries. The training pipeline remains fully self-contained: no privileged signals are required past training, greatly simplifying LLM-based agent deployment in practical, long-horizon environments.

The framework’s reliance on trajectory-derived, distribution-matched skills offers two main theoretical and practical advantages:

• Easier credit assignment: Tokens involved in critical, outcome-relevant steps receive dense signals, addressing credit diffusion under sparse RL rewards.
• Internalized procedural knowledge: Skills become part of the policy parameters, not merely in-context hints, allowing for robust, context-independent behavior adaptation at test time.

Future research could probe more compositional skill structures, adaptive routing mechanisms, and expand benchmarking into multi-modal or more open-ended interactive settings. Extensions toward continual learning, agent exploration, and policy-aware reasoning abstractions are natural directions facilitated by OPID’s hindsight-focused design.

Conclusion

OPID provides a principled, effective protocol for hierarchical skill distillation in agentic RL, addressing fundamental barriers in training long-horizon LLM-based agents. By extracting and distilling hindsight skills on-policy, OPID achieves superior local and global policy refinement, enhanced sample efficiency, and operational simplicity—offering a compelling foundation for robust, scalable deployment of RL agents in complex, interactive domains (2606.26790).