Co-Evolving Policy Distillation

Abstract: RLVR and OPD have become standard paradigms for post-training. We provide a unified analysis of these two paradigms in consolidating multiple expert capabilities into a single model, identifying capability loss in different ways: mixed RLVR suffers from inter-capability divergence cost, while the pipeline of first training experts and then performing OPD, though avoiding divergence, fails to fully absorb teacher capabilities due to large behavioral pattern gaps between teacher and student. We propose Co-Evolving Policy Distillation (CoPD), which encourages parallel training of experts and introduces OPD during each expert's ongoing RLVR training rather than after complete expert training, with experts serving as mutual teachers (making OPD bidirectional) to co-evolve. This enables more consistent behavioral patterns among experts while maintaining sufficient complementary knowledge throughout. Experiments validate that CoPD achieves all-in-one integration of text, image, and video reasoning capabilities, significantly outperforming strong baselines such as mixed RLVR and MOPD, and even surpassing domain-specific experts. The model parallel training pattern offered by CoPD may inspire a novel training scaling paradigm.

• The paper introduces a unified framework that alternates parallel RLVR and mutual OPD to maintain behavioral coupling and enhance knowledge absorption.
• The method enables each domain-specific branch to specialize while effectively transferring expertise through on-policy distillation.
• Experimental results on text, image, and video reasoning show state-of-the-art performance, outperforming traditional static and multi-teacher approaches.

Co-Evolving Policy Distillation: A Unified Framework for Cross-Domain Capability Consolidation

Introduction and Motivation

Recent work on post-training large language and multimodal models has converged on two main paradigms for capability integration: reinforcement learning with verifiable rewards (RLVR), often applied on mixed data, and on-policy distillation (OPD) pipelines which first produce domain-specific experts and then consolidate them. However, both approaches systematically fail to absorb the full potential of the expert ensemble: mixed-data RLVR suffers from gradient conflict causing capability divergence, while post-hoc (static) OPD is limited by behavioral drift between teachers and the student, resulting in low absorption efficiency.

The key insight motivating Co-Evolving Policy Distillation (CoPD) is that absorbing cross-domain knowledge requires teachers and students to stay within an "absorbable" behavioral distance throughout training—empirically measured by top-$k$ token overlap. Pure RLVR and static OPD pipelines violate this: RLVR creates gradient interference, and static OPD allows experts to drift too far apart (Figures 1–2).

Figure 1: CoPD addresses the limitations of mixed-data RLVR (a) and static OPD (b) by letting two branches co-evolve as teachers and students across domains (c), achieving the best overall performance.

Figure 2: Behavioral pilot analysis—OPD gain increases linearly with teacher–student top-$k$ overlap, but RLVR training causes this overlap to deteriorate, confirming that static OPD operates in a regime of low absorption efficiency.

Methodology: Co-Evolving Policy Distillation

CoPD alternates parallel RLVR and cross-branch mutual OPD among multiple expert branches, with all branches co-evolving as both teachers and students. Each branch maintains its own RLVR-driven specialization on a distinct domain, but periodically performs OPD against the other branches, always on-policy. This interleaving is essential: RLVR builds domain expertise (increasing knowledge gap), mutual OPD aligns behavioral patterns (increasing absorbability), and parameterization remains tightly coupled throughout.

The procedure (Figure 3) is as follows: In phase I, each branch executes RLVR; in phase II, branches perform mutual OPD by generating on-policy rollouts on the other’s task distribution, and receive token-level KL supervision. Both RLVR and OPD contributions are combined in the parameter update, ensuring both domain-specific exploration and efficient knowledge transfer.

Figure 3: Graphical overview of the CoPD framework, where parallel RLVR and mutual OPD cycles maintain both domain-specific advancement and effective cross-branch transfer with absorbable behavioral distance.

CoPD generalizes trivially to $K$ branches using a hub-and-spoke topology, where one branch (typically the LLM) acts as a hub exchanging OPD with each domain-specific branch (text, image, video, etc.).

Empirical Analysis

Pilot Study and Behavioral Metrics

A systematic pilot study demonstrates that the effectiveness of OPD, $\eta(\mathcal{O}_k)$, rises monotonically with behavioral overlap until teacher and student are indistinguishable (at which point the supervision signal loses informativeness), and that independently trained RLVR experts quickly reach the regime of low overlap with the base or consolidating student. Thus, static OPD pipelines are fundamentally limited by operating away from the optimal range for knowledge absorption (Figure 2).

Training Dynamics

Extensive experiments on Qwen3-VL-4B-Instruct confirm the theoretical analysis. In both two- and three-branch consolidation tasks (text/image and text/image/video), CoPD outperforms all baselines, including strong RLVR, static OPD, and multi-teacher OPD (MOPD). Notably, CoPD achieves simultaneous improvement on all domains and surpasses each domain’s specialized expert, breaking the conventional student-expert ceiling.

Figure 4: Training dynamics and design analysis of CoPD. (a) Top-$k$ token overlap is actively maintained via alternation, (b) symmetric KL divergence remains low, and (c) tuning $S_{\mathrm{RL}}/S_{\mathrm{OPD}}$ allows a trade-off between domain exploration and consolidation, with 1.5:1 optimal.

Ablation studies confirm that bidirectional mutual OPD is essential for domain transfer; unidirectional OPD or removal of the merge operation degrades both image and text reasoning. Each branch individually achieves well-rounded competencies, and parameter merging further consolidates complementary strengths.

Quantitatively, CoPD sets new state-of-the-art results across all evaluation suites for text, image, and video reasoning, surpassing both static and MOPD approaches. Performance is robust to the RLVR/OPD steps hyperparameter, with the best setting encouraging sufficient domain exploration before consolidation.

Implications, Theoretical Perspective, and Future Directions

CoPD demonstrates that parallel, tightly coupled mutual distillation during training is crucial for effective consolidation and transfer of capabilities among large-scale models. The CoPD framework integrates capability exploration and consolidation with no trade-off, leveraging ongoing bidirectional teacher–student interactions. This is theoretically significant as it provides a recipe for avoiding both gradient divergence and absorption inefficiency, the central obstacles in previous works.

Practically, CoPD opens a new direction for post-training paradigms: parallel and collaborative model specialization with tightly interleaved knowledge sharing, rather than costly end-to-end retraining or brittle final-stage distillation. This approach generalizes naturally to an arbitrary number of branches, and its model-parallel structure is particularly suited to distributed training and future scaling regimes.

The results further suggest that parameter merging in the regime of tight behavioral coupling does not incur significant loss—contrary to prior beliefs about domain drift and consolidation. There is a clear path for future research to scale this pattern to a larger number of capabilities and modalities (e.g., multi-lingual, agentic, or long-context tasks), and to explore finer dynamic scheduling policies for phase alternation.

Conclusion

Co-Evolving Policy Distillation offers a unified, robust solution to consolidating the expertise of multiple RLVR-trained models into a single policy without sacrificing individual capabilities or overall performance. Through alternating parallel RLVR and mutual OPD phases, CoPD maintains tight behavioral coupling and leverages ongoing bidirectional knowledge transfer. Extensive empirical evaluation confirms simultaneous, cross-domain consolidation with no performance trade-offs, consistently outperforming both mixed RLVR and static OPD paradigms—including surpassing all domain-specific experts and previous MOPD methods. This co-evolutionary approach reconfigures the landscape of post-training strategies for multimodal and agentic models, with significant implications for both theoretical and scalable practical development in AI (2604.27083).