- The paper introduces a novel UCOB framework that uses local, credit-aware bidirectional self-distillation to select the higher-return skill context for policy improvement.
- It employs dual-level skill memory with UCB-based retrieval to dynamically update both task-level and state-level strategies, enhancing sample efficiency.
- Empirical results across benchmarks demonstrate significant performance gains and validate the approach's robustness against misleading skill inputs.
UCOB: Credit-Aware On-Policy Bidirectional Self-Distillation for Skill Utilization and Evolution in Agentic RL
Motivation and Problem Analysis
UCOB addresses the core problem of utilizing and evolving textual skill memories for agentic reinforcement learning (RL) in multiturn language agents, particularly in environments with sparse feedback and long horizons. While prior skill-augmented RL and self-distillation methods typically treat skill-conditioned behaviors as privileged sources of supervision, this paradigm is empirically fragile. The skill-conditioned prompt may provide superior guidance in some contexts, but retrieved skills can be locally irrelevant, stale, or misleading, and the no-skill view may outperform the skill-conditioned view in a substantial fraction of states.
Empirical diagnostics in the paper demonstrate that even with on-policy skill-conditioned rollouts, there is no guarantee that skill-prompted behavior is consistently beneficial. Both SDAR and dual-rollout SDAR regimes confirm that skill-to-no-skill distillation can propagate locally non-optimal behavior. Hence, fixed-direction skill-conditioned distillation can degrade policy performance in states where the skill context is not actually helpful, violating the assumptions underpinning most existing self-distillation architectures.
Approach: Credit-Aware Bidirectional Self-Distillation
UCOB reframes the teacher-student paradigm into a context-sensitive, local, and bidirectional interaction. For each anchor state within a sampled task group, UCOB compares returns from trajectories generated under both the skill-conditioned and the no-skill contexts. The prompt that yields higher return for a given anchor state is dynamically selected as the teacher for self-distillation, enabling internalization of truly beneficial skill-conditioned behaviors and correction of misleading skill usage. This mechanism is formalized as Credit-Aware Bidirectional Self-Distillation (CBSD).
CBSD operates at per-anchor-state granularity rather than entire trajectories, leveraging step-level credit assignment. Returned-based comparison ensures only locally validated skill-conditioned behaviors are distilled, which theoretically guarantees positive-surrogate policy improvement under trust-region updates. The bidirectional mechanism prevents negative updates inherent to fixed-direction distillation when the supposed “teacher” branch underperforms.
System Architecture: Skill Memory, Retrieval, and Self-Evolution
- Dual-level skill memory: UCOB maintains task-level and state-level textual skill pools. Task-level skills encode episode-wide strategies, while state-level skills capture local decision rules, both continuously updated based on observed utility.
- Utility-aware retrieval: Skill candidates are ranked and selected for prompting using a UCB-based score, balancing relevance (embedding similarity) and empirical utility, which is updated online via EMA based on actual return improvements.
- Mixed-context rollouts: Both skill-conditioned and no-skill rollouts are sampled under the current policy. No branch is permanently assigned as teacher or student. Teacher direction is decided post hoc using local return gaps.
- Reflection-based skill writer: The system self-trains its skill writer using reflection on successful behaviors and same-state contrasts, reinforcing high-utility skills and demoting spurious ones in the memory banks.
- Joint objective: The training loss incorporates on-policy RL, CBSD, reflection self-training, and KL regularization.
Theoretical Justification
A local policy-improvement analysis shows that, for each anchor state, moving the lower-returning branch toward the higher-returning branch via KL distillation provably improves local expected advantage. This policy update contributes positively to the global expected return when rollouts are kept within an appropriate trust region. Crucially, this avoids negative expected updates, which are inherent in fixed-direction, privilege-based self-distillation when the teacher branch is suboptimal.
Empirical Results
UCOB demonstrates significant empirical improvements on ALFWORLD, WEBSHOP, and SEARCH-QA benchmarks, spanning navigation, web interaction, and search-based QA tasks across multiple LLM backbones (Qwen2.5-7B/3B, Qwen3-1.7B). Strong numerical highlights include:
| Backbone |
ALFWORLD Succ. (UCOB) |
ALFWORLD Δ vs SOTA |
WEBSHOP Succ. (UCOB) |
WEBSHOP Δ vs SOTA |
| Qwen2.5-7B-Instruct |
93.0 |
+0.8 |
85.9 |
+0.8 |
| Qwen2.5-3B-Instruct |
92.2 |
+1.6 |
78.1 |
+3.1 |
| Qwen3-1.7B |
89.1 |
+23.5 |
79.7 |
+18.0 |
These improvements are especially pronounced on smaller backbones, and UCOB is competitive or top-performing in search-centric QA tasks as well.
Ablation studies confirm that anchor-state-level credit assignment, bidirectional self-distillation, and state-level memory are all crucial for maximal gains. Removing bidirectionality or credit-aware routing aligns UCOB with prior baselines, leading to substantial degradation in both success rate and sample efficiency. Reflection self-training further improves skill memory quality and retrieval utility.
Additional analyses show that neither skill-conditioned nor no-skill contexts can be assumed universally superior—their relative policy values switch frequently at local states, exactly justifying the bidirectional, credit-aware distillation design.
Practical and Theoretical Implications
Practically, UCOB enables more robust, sample-efficient training for LLM-based agents operating in complex, memory-augmented RL environments. By avoiding overcommitment to potentially misleading skills and continuously evolving both the skill memory and the policy, UCOB offers state-aware skill internalization and correction during online learning. Its modular structure makes it extensible to any retrieval-augmented agentic RL setting.
Theoretically, UCOB recasts skill utilization as a local credit assignment problem, tightly integrating reinforcement signals not only for action learning but also for context view selection and skill memory evolution. The bidirectional design represents a corrective mechanism for the weaknesses in privileged-teacher self-distillation under imperfect memory, a general concern in skill-based RL for LLM agents.
Speculation on Future Directions
- Generalization across domains: The local credit-aware routing approach can be further generalized to other forms of context augmentation, including multi-agent self-play and tool-augmented LMs, where competing context sources must be arbitrated.
- Hierarchical skill evolution: UCOB’s dual-level skill memory provides a foundation for hierarchical skill learning and transfer, where higher-order skills could be constructed from compositions of locally validated primitives.
- Policy-skill joint optimization: Future work may integrate tighter co-adaptation between policy optimization and memory management, potentially using meta-learning to dynamically adjust UCB retrieval or skill-writing strategies.
- Sample efficiency and scaling: Further architectural and algorithmic refinements can aim to reduce rollout and training cost, especially as transformer backbones scale upward.
Conclusion
UCOB establishes a new paradigm in agentic RL with skill memories by making teacher selection for self-distillation locally value-based and bidirectional. This design corrects the asymmetric and sometimes deleterious inductive biases of prior fixed-direction distillation architectures. Experimentally, UCOB yields strong improvements on challenging multiturn LLM agentic tasks, efficiently leveraging and evolving skill memory to produce robust policies and versatile, context-sensitive skill writers. The framework extends our understanding of skill-conditioned RL training, emphasizing the centrality of local, credit-aware arbitration in memory-augmented language agents (2606.29502).
