Be My Tutor: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedback

Abstract: We study multi-domain LLM training in which two models, each stronger in a different domain, co-evolve by tutoring each other through on-policy feedback. Unlike one-way distillation or single-model fine-tuning, our goal is mutual Pareto improvement: each model improves across domains without losing its original strength. To this end, we propose On-Policy Co-Distillation (OPCoD), where each student's self-distillation is conditioned on its own correct rollout and feedback from its peer. To make feedback exchange effective, OPCoD uses cognizance-based gating to decide when to give feedback and feedback anchoring to ground feedback in the problem. On Science Q&A tasks, OPCoD consistently outperforms baselines and achieves Pareto improvement across all evaluated domain pairs and students.

• The paper introduces On-Policy Co-Distillation (OPCoD), which achieves mutual Pareto improvement by enabling bidirectional peer feedback between domain-specialized LLMs.
• OPCoD employs dynamic cognizance-based gating and a two-step feedback anchoring process to filter and integrate specialist insights effectively.
• Experimental results in chemistry, physics, and materials science show that OPCoD boosts cross-domain performance while preserving native expertise and outperforming baseline methods.

On-Policy Co-Distillation for Mutual LLM Enhancement via Peer Feedback

Motivation and Problem Statement

Multi-domain capability in LLMs is often hampered by negative transfer when naively merging specialist finetuning datasets across domains. Existing techniques—single-model on-policy distillation, self-distillation, and one-way multi-teacher aggregation—have failed to guarantee robust cross-domain generalization without jeopardizing native-domain performance. This paper introduces On-Policy Co-Distillation (OPCoD), a framework for mutual LLM improvement by means of bidirectional, on-policy peer feedback. The central objective is mutual Pareto improvement: both models, initially specialized in distinct domains, should advance in overall performance across both domains while preserving or enhancing their original strengths.

Figure 1: Conceptual illustration of OPCoD’s dual-agent peer tutoring dynamic in a multi-domain setting.

Framework: On-Policy Co-Distillation (OPCoD)

OPCoD operates with two student LLMs, each originally stronger in a different domain. Training proceeds in rounds; in each round, the models alternate as tutee (student updated) and tutor (feedback provider), thereby coupling their learning via feedback exchange.

Figure 2: OPCoD overview for a prompt $x$—the tutee samples multiple on-policy rollouts, and each receives tutor-generated feedback if the cognizance-gate is passed.

The central optimization involves self-distillation, but the self-teacher is conditioned not only on a student’s own correct rollout (as in SDPO/OPSD) but also on peer-generated feedback. Tutor feedback is admitted only when passing a cognizance-based gate, which is determined dynamically each round based on relative domain competence. Bidirectional feedback creates coupled optimization dynamics, in contrast to previous one-way approaches.

Figure 3: Mutual tutoring (left): Students update via bidirectional feedback. Within-round (right): For each prompt, the tutee samples multiple rollouts, and the tutor provides feedback when permitted by cognizance-based gating.

Formalization

In each direction, student policy $\pi^i$ minimizes divergence to a self-teacher conditioned on correct self-rollout $s$ and peer feedback $f$:

$$\mathcal{L}^i(\pi_S^i; \pi^{-i}) = \mathbb{E}_{x, y}\left[\frac{1}{|y|}\sum_{t=1}^{|y|} D\left(\pi_S^i(\cdot \mid x, y_{<t}) \big\| \pi_T^i(\cdot \mid x, s, f, y_{<t})\right)\right]$$

where $D$ is e.g. Jensen–Shannon divergence.

Cognizance-Based Gating

To mitigate risky or unreliable peer feedback, a tutor's eligibility each round is established via dynamic relative domain validation scores. Specifically, a model $\pi^i$ is cognizant if for all domains, its validation score is within a preset threshold $\tau$ of the best-performer. Otherwise, feedback is silenced for that round and the tutee falls back to domain self-distillation.

Figure 4: An incognizant tutor’s feedback can corrupt a correct solution—highlighting the need for dynamic gating.

Feedback Anchoring

To suppress feedback hallucination (problem-irrelevant, ungrounded suggestions), feedback is anchored via a two-step prompt: the tutor must extract and reference a core technical concept from the actual question (validated automatically), and provide concise critique without explicit answer reveals. Feedback failing the anchoring or containing generic/conceptually irrelevant tags is sanitized and filtered out.

(Figures 5 & 6)

Figure 5: Without anchoring, tutor hallucinates irrelevant feedback (left); with anchoring, feedback is properly problem-grounded (right).

Empirical Results

Experiments use Qwen3-8B-based students, each specialized for chemistry, physics, or materials science, evaluated on SciKnowEval split multi-domain Science QA. OPCoD is compared to SDPO and GRPO, both strong self-distillation baselines.

Key numerical findings:

• Mutual Pareto improvement is achieved in all tested pairs: Both students in each pair improve cross-domain scores without native-domain degradation (in fact, each agent consistently improves on its specialty), while baseline methods (SDPO, GRPO) often suffer collapse or side-domain improvement at the expense of native skill.
• Across all evaluated domain pairs (chemistry-materials, materials-physics, physics-chemistry), OPCoD yields the highest scores for each model in all cases (“OPCoD Pareto-dominates the baselines”).
• Gating is critical: Incognizant tutors break previously correct rollouts at 2.4× higher rate than cognizant tutors. Even when ablation admits incognizant feedback only within the tutor’s specialty, breakdown rate remains much higher than with strict gating.

  Figure 6: Feedback gating ablation—cognizance-based gating achieves robust cross-domain improvement; other strategies lead to negative transfer.

Feedback Anchoring Analysis

Over 70% of tutor feedback passes the anchoring filter (concept tag matches problem), with only ∼2.5% of feedback considered ungrounded or irrelevant, indicating high precision and sufficiency of anchoring for reliable signal.

Figure 7: Feedback filtering statistics throughout training show anchoring is both conservative and preserves valuable feedback volume.

Round-Step Ablation

Pareto improvement is robust to round/step allocation, but multi-round schedules with sufficient per-round steps (e.g., 2×50) yield optimal and stable results, reflecting the need for sufficient adaptation time to incoming peer distributions each round.

Figure 8: Round-step ablation shows stable trajectories for reasonable round/step splits.

Qualitative Analysis

Case studies reveal that peer feedback enables genuine cross-domain error correction beyond single-model limitations. Notably, the physics-stronger student fails a heat-capacity question due to improper handling of molecular vibration, but recovers the right answer only after chemistry-strong tutor feedback exposes the missing domain insight—thus combining modalities of expertise.

Figure 9: Peer feedback supplies missing physical-chemistry reasoning, enabling the tutee to revise and solve the problem correctly.

(Figures 15 & 16)

Figure 10: Chemistry misconception corrected by properly targeted feedback.

Figure 11: Physics question’s missing equilibrium interpretation unlocked by chemistry-domain peer feedback.

Implications, Limitations, and Future Directions

OPCoD advances multi-domain LLM co-training by operationalizing mutual, controllable on-policy feedback without external teachers. This enables safe cross-specialty generalization and robust maintenance of original expertise. Key practical implications include:

• Negative transfer is actively mitigated: Gating and anchoring policies allow only beneficial peer tuning; the framework is robust against regression on native domains.
• Generalizes beyond knowledge merging: By relying on dynamic peer feedback rather than static distillation, OPCoD is inherently suitable for settings where single-model multitask finetuning is unstable.

Limitations include restriction to pairwise co-distillation (generalization to $n$-way or graph-structured co-training remains unexplored), reliance on effective prompt design for feedback extraction/anchoring, and demonstration on scientific QA domains—transfer to more distant or underspecified domains is untested.

Conclusion

OPCoD demonstrates that peer-feedback-driven, on-policy co-distillation enables stable, bidirectional LLM improvement in multi-domain settings, overcoming negative transfer observed in previous approaches. The results concretely show that Pareto improvement is feasible with strict gating and feedback anchoring, and cross-domain reasoning can be acquired without sacrificing original domain strengths. Extending this paradigm to larger groups, non-scientific domains, and more elaborate feedback selection or validation mechanisms constitutes a promising vein for future research in collaborative continual learning for foundation models.

---

Reference: "Be My Tutor: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedback" (2606.14368)