Collaborative Distillation & Self-Learning

• The paper introduces collaborative distillation and self-learning as a design pattern that integrates peer-based supervision and adaptive self-feedback to enhance model performance.
• It details diverse implementations ranging from peer ensemble methods and cross-modal supervision to reinforcement-learning driven agent guidance, each balancing teacher quality with student adaptability.
• Applications span online knowledge distillation, continual learning, and autonomous driving, with empirical results demonstrating significant gains in accuracy, stability, and robustness.

Collaborative distillation and self-learning denotes a class of training frameworks in which supervision is produced not only by fixed labels or a separately pre-trained teacher, but also by peers, ensembles, frozen counterparts, auxiliary modalities, or communities of models, while each learner receives an additional self-improvement signal such as temporal self-distillation, consistency on challenging samples, contrastive pretext learning, intrinsic rewards, or iterative re-collaboration (Wu et al., 2020, Wang et al., 2 Sep 2025, Jo et al., 13 Feb 2026, Feng et al., 5 Feb 2026). In a narrow sense, the explicit phrase appears in the reinforcement-learning-driven infrared-visible image fusion framework of AgentFuse, where teacher guidance and self-learning on degraded samples are jointly scheduled by an agent (Wang et al., 2 Sep 2025). In a broader sense, closely related designs recur across online knowledge distillation, multimodal self-supervision, privacy-preserving collaborative learning, continual learning, multi-agent reinforcement learning, dense prediction, and autonomous driving (Berdoz et al., 2022, Ji et al., 2021, Zhou et al., 5 Feb 2025, Cao et al., 10 Mar 2026).

1. Conceptual scope and taxonomy

The literature uses collaborative distillation and self-learning in multiple technically distinct ways. Some methods remove the fixed external teacher entirely and construct supervision online from peers or ensembles. Peer Collaborative Learning (PCL) builds a multi-branch network in which each branch is a peer, forms a learned peer ensemble teacher from concatenated peer features, and simultaneously uses temporal mean models as peer mean teachers (Wu et al., 2020). Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (CTSL-MKT) likewise dispenses with a separately pre-trained teacher after a self-initialization stage and combines online mutual learning with self-distillation (Sun et al., 2021). The Single-Multi Evolution Loop moves further toward ecosystem-level collaboration: multiple LLMs collaborate to produce synthetic outputs, each model distills from the collaborative outputs, and the stronger post-distillation models collaborate again (Feng et al., 5 Feb 2026).

Other methods define collaboration across modalities, agents, or nodes rather than among homogeneous peers. LiDAR-Anchored Collaborative Distillation aligns a LiDAR encoder to a reliable clear-day 2D feature space and then freezes the LiDAR branch to supervise the 2D encoder under adverse weather (Jo et al., 13 Feb 2026). Prediction Consensus lets agents exchange predictions on shared unlabeled auxiliary data and compute trust-weighted pseudo-labels until consensus is approached (Fan et al., 2023). LENC organizes deployed DNN nodes into a node community in which any node may become either student or teacher depending on its OOD-based self-assessment (Kaimakamidis et al., 2024).

Framework	Collaborative signal	Self-learning signal
PCL	Learned peer ensemble teacher	Peer mean teacher
LiDAR-Anchored CD	Frozen cross-modal LiDAR anchor	Stage-swapped self-supervision
AgentFuse	Teacher fusion guidance	Challenging-sample consistency
Single-Multi Evolution Loop	Multi-LM collaborative outputs	Iterative re-distillation
Prediction Consensus	Trust-weighted peer pseudo-labels	Local update on auxiliary data
LENC	Dynamic node education	OOD-triggered autonomous role change

This variety suggests that “collaborative distillation and self-learning” is best understood as a design pattern rather than a single canonical architecture. The recurrent principle is that collaboration supplies externalized structure, while self-learning supplies stabilizing or difficulty-adaptive feedback.

2. Architectural realizations

In online classification settings, collaboration is often realized by parallel branches or experts. PCL starts from an off-the-shelf convolutional backbone, splits it into shared low-level layers and peer-specific high-level layers, attaches $M$ heads, and feeds each peer an independently augmented version of the same image. The peer features are concatenated into $h(x)\in\mathbb{R}^{M\cdot d}$ and passed through an additional classifier $g(\cdot;\phi)$ to form ensemble logits $z^t$ (Wu et al., 2020). ECL similarly trains $K$ experts in parallel, but each expert contains two classifier heads—a reference head supervised only by the class-balanced loss and a collaborative head supervised by class-balanced loss plus online KD—and also includes a MoCo-style siamese branch for a contrastive proxy task (Xu et al., 2023). CTSL-MKT uses peer networks that expose both logits and intermediate features so that response-based and relation-based knowledge can both be transferred (Sun et al., 2021).

Cross-modal and planning-oriented systems instantiate the same pattern with asymmetric teachers. LiDAR-Anchored Collaborative Distillation is explicitly two-stage: Stage 1 freezes $E_{2D}$ and distills pixel-wise 2D features into a trainable $E_{3D}$; Stage 2 freezes the aligned $E_{3D}$ and pulls degraded 2D features toward the LiDAR anchor space (Jo et al., 13 Feb 2026). EvoDriveVLA adds two frozen teachers to a Vision-Language-Action backbone: a self-anchor teacher, defined as a frozen copy of the student’s visual encoder before fine-tuning, and an oracle teacher with privileged future images and ego states. An AnchorFormer module computes token-level visual anchoring weights, while the oracle teacher supplies refined trajectory candidates for distillation (Cao et al., 10 Mar 2026).

Reinforcement-learning and decentralized settings use explicit controller or communication modules. AgentFuse employs a pretrained high-capacity teacher model, LDFusion, and a lightweight dual-branch encoder-decoder student obtained by pruning approximately $88\%$ of LDFusion’s parameters. An RL agent observes both the student’s current fusion quality and the teacher-student performance gap, then outputs $\alpha_t$, $h(x)\in\mathbb{R}^{M\cdot d}$0, and $h(x)\in\mathbb{R}^{M\cdot d}$1, which respectively control teacher guidance, self-learning weight, and degradation parameters for challenging samples (Wang et al., 2 Sep 2025). DDN augments CTDE with a Global Guiding Network, a Local Policy Network, and an Internal Distillation Module that maps global state novelty into intrinsic rewards (Zhou et al., 5 Feb 2025).

Communication-efficient collaborative systems replace parameter exchange with latent or predictive summaries. Representation-sharing collaborative learning has clients upload averaged last hidden-layer activations by class to a relay server, download global prototypes and sampled class representations, and distill these via feature and contrastive objectives (Berdoz et al., 2022). Prediction Consensus broadcasts only predictions on shared unlabeled data and derives pseudo-labels through a trust matrix built from entropy-weighted cosine similarity (Fan et al., 2023). LENC further equips each node with decision heads and per-task OOD detectors, so that unknown streams trigger an education cycle rather than ordinary inference (Kaimakamidis et al., 2024).

3. Forms of self-learning

Self-learning is not uniform across this literature. In PCL, self-learning is embodied by a temporal mean teacher for each peer and for the shared stem. At global step $h(x)\in\mathbb{R}^{M\cdot d}$2, the averaged parameters are updated as

$h(x)\in\mathbb{R}^{M\cdot d}$3

with $h(x)\in\mathbb{R}^{M\cdot d}$4 and $h(x)\in\mathbb{R}^{M\cdot d}$5. The temporal model yields smoother targets and more stable mutual distillation among peers (Wu et al., 2020).

In AgentFuse, self-learning is generated by challenging samples rather than by temporal averaging. The RL agent increases sample difficulty through Gaussian blur, JPEG compression, brightness shift, contrast adjustment, and additive Gaussian noise, and the student is trained to keep its degraded-input fusion $h(x)\in\mathbb{R}^{M\cdot d}$6 consistent with its own fusion $h(x)\in\mathbb{R}^{M\cdot d}$7 on the original inputs (Wang et al., 2 Sep 2025). DistillFlow uses a related but denser version of this idea for optical flow: random cropping, superpixel noise injection, affine transforms, and color transforms create hallucinated occlusions and less confident predictions, after which confident teacher predictions supervise the student on those harder pixels (Liu et al., 2021).

In continual and long-tailed learning, self-learning is coupled to representation shaping. CoCa addresses feature deviation through pretext tasks and supervised contrastive learning, defining $h(x)\in\mathbb{R}^{M\cdot d}$8 so that features become both complete and discriminative (Ji et al., 2021). ECL adds a contrastive proxy task branch per expert, with a momentum encoder and InfoNCE-style loss, and treats this feature regularization as complementary to logit- and feature-level collaborative distillation (Xu et al., 2023). CTSL-MKT uses a peer’s own Stage 1 converged network as a frozen teacher during Stage 2, which is a direct self-distillation mechanism (Sun et al., 2021).

At the system level, self-learning can mean recursive collaboration. The Single-Multi Evolution Loop lets a collaboration strategy $h(x)\in\mathbb{R}^{M\cdot d}$9 produce a dataset $g(\cdot;\phi)$0, after which each model updates on $g(\cdot;\phi)$1 and the improved models form the next collaboration pool $g(\cdot;\phi)$2 (Feng et al., 5 Feb 2026). LENC uses OOD-based self-assessment to trigger autonomous learning episodes (Kaimakamidis et al., 2024), while DDN’s Internal Distillation Module converts predictor-target mismatch on global states into intrinsic reward, thereby making exploration itself a self-learning signal (Zhou et al., 5 Feb 2025).

A common misconception is that self-learning in these frameworks always means pseudo-labeling. The published methods show a broader space: temporal ensembling, self-distillation, challenge-driven consistency, pretext-task supervision, supervised contrastive learning, intrinsic-reward shaping, and iterative self-improvement all fall under the same general label.

4. Objectives and optimization principles

The optimization layer typically combines supervised terms, collaborative transfer terms, and self-learning terms. PCL is exemplary in its explicit decomposition: $g(\cdot;\phi)$3 where $g(\cdot;\phi)$4 is peer cross-entropy, $g(\cdot;\phi)$5 supervises the ensemble teacher, $g(\cdot;\phi)$6 distills from the ensemble teacher to each peer, and $g(\cdot;\phi)$7 aligns each peer to the mean teachers of the other peers. The distillation terms are temperature-scaled and ramped up through $g(\cdot;\phi)$8, so early training is not dominated by noisy KD gradients (Wu et al., 2020).

Cross-modal collaborative distillation often uses a stop-gradient asymmetry rather than a simultaneous mutual objective. LiDAR-Anchored Collaborative Distillation uses the same $g(\cdot;\phi)$9 matching form in both stages,

$z^t$0

but reverses gradient flow between stages. No teacher-student moving average or momentum encoder is used; the teacher is simply the frozen counterpart from the other modality (Jo et al., 13 Feb 2026).

AgentFuse makes the balance between collaboration and autonomy explicit: $z^t$1 with $z^t$2 enforced by softmax and policy-gradient updates

$z^t$3

Here the state $z^t$4 concatenates normalized student metrics and the teacher-student gap, and the action $z^t$5 controls guidance strength, self-learning weight, and degradation severity (Wang et al., 2 Sep 2025).

Several frameworks enrich KD with additional structural objectives. CTSL-MKT defines

$z^t$6

where $z^t$7 contains both response-based KL transfer and relation-based distance- and angle-wise transfer, and $z^t$8 is self-distillation from the peer’s own Stage 1 model (Sun et al., 2021). Representation-sharing collaborative learning combines local cross-entropy with a feature KD term and a discriminator loss $z^t$9 that lower-bounds mutual information through $K$0 (Berdoz et al., 2022). Prediction Consensus forms pseudo-labels as

$K$1

with $K$2 derived from entropy-weighted cosine agreement, and then optimizes a local supervised-plus-distillation objective on labeled data and shared unlabeled auxiliary data (Fan et al., 2023).

These objectives reveal a persistent design choice: collaborative distillation is rarely a single KL term. In most mature formulations it is embedded in a composite loss that also includes supervised calibration, stability regularization, feature matching, relation preservation, or policy optimization.

5. Application domains and empirical profile

The empirical record spans classification, representation learning, model collaboration, continual learning, dense prediction, reinforcement learning, autonomous driving, and node communities. The reported gains are not confined to a single modality or task family.

Domain	Representative quantitative result	Citation
Online KD	CIFAR-100, ResNet-110: baseline $K$3, PCL $K$4, PCL-E $K$5	(Wu et al., 2020)
Robust 2D SSL	Night segmentation mIoU $K$6; VPQ $K$7	(Jo et al., 13 Feb 2026)
Infrared-visible fusion	AG on M3FD $K$8; [email protected] $K$9	(Wang et al., 2 Sep 2025)
LM collaboration	$E_{2D}$0 average single-model gain; $E_{2D}$1 evolved multi-LM gain	(Feng et al., 5 Feb 2026)
Privacy-preserving learning	MNIST, $E_{2D}$2: IL $E_{2D}$3, FL $E_{2D}$4, FD $E_{2D}$5, Ours $E_{2D}$6	(Berdoz et al., 2022)
Continual learning	CIFAR-100 average accuracy, buffer $E_{2D}$7: CoCa $E_{2D}$8, DER $E_{2D}$9, ER $E_{3D}$0	(Ji et al., 2021)
Optical flow	KITTI 2015 unsupervised test Fl-all $E_{3D}$1; fine-tuned $E_{3D}$2	(Liu et al., 2021)
Multi-agent RL	VDN on 3s_vs_5z $E_{3D}$3; MMM2 $E_{3D}$4	(Zhou et al., 5 Feb 2025)
Node-community CKD	CIFAR-10, ResNet-18$E_{3D}$5ResNet-18, $E_{3D}$6: LENC $E_{3D}$7 vs KDCL $E_{3D}$8	(Kaimakamidis et al., 2024)

Additional results reinforce the same pattern. In EvoDriveVLA, open-loop ST-P3 average $E_{3D}$9 decreases from $E_{3D}$0 for the best previous distillation baseline to $E_{3D}$1, collision rate drops from $E_{3D}$2 to $E_{3D}$3, and closed-loop NAVSIM PDMS reaches $E_{3D}$4 for EvoDriveVLA-3B versus $E_{3D}$5 for UniAD (3B) and $E_{3D}$6 for Qwen2.5-VL 8B (Cao et al., 10 Mar 2026). In DDN, end-to-end training yields up to $E_{3D}$7 win-rate on 3s5z and $E_{3D}$8 on MMM2 for GGN/LPN, outperforming PTDE and CTDS in both teacher and student stages (Zhou et al., 5 Feb 2025). In the Single-Multi Evolution Loop, reasoning tasks improve by $E_{3D}$9, knowledge by $88\%$0, and QA by $88\%$1, while on-policy KD adds another $88\%$2 over multi-student KD (Feng et al., 5 Feb 2026).

The breadth of these results suggests that collaborative distillation is especially effective when the teacher signal can be made either richer than any individual model—through ensembles, debates, or privileged inputs—or more stable than instantaneous predictions—through temporal averaging, frozen anchors, or averaged representations.

6. Limitations, failure modes, and recurrent technical questions

The literature also identifies several boundary conditions. PCL is motivated by specific failures of prior online KD: collaborative learning and mutual learning fail to construct an online high-capacity teacher, while online ensembling based on logit summation impedes further optimization of the ensemble teacher (Wu et al., 2020). LiDAR-Anchored Collaborative Distillation reports that collapsing or unstable behavior appears when the two stages are collapsed into one joint stage, with performance dropping by more than $88\%$3 mIoU, and that using adverse-weather pairs in Stage 1 hurts alignment (Jo et al., 13 Feb 2026). DistillFlow notes that student quality is upper-bounded by the teacher ensemble and that the two-stage procedure doubles compute cost (Liu et al., 2021).

Communication-efficient collaboration creates a different set of trade-offs. Prediction Consensus reduces traffic relative to model exchange, but performance gains from additional rounds come at the cost of potentially more information leakage (Fan et al., 2023). Representation-sharing systems are scalable and communication efficient, but empirical stability depends on proper $88\%$4 and $88\%$5 tuning, and no further convergence guarantee beyond well-posedness of the contrastive objective is given (Berdoz et al., 2022). LENC shows that using stored test accuracy as the teacher-selection metric gives the highest peak but blocks students from ever teaching, while adding more nodes requires proportionally more education cycles (Kaimakamidis et al., 2024).

Iterative self-improvement frameworks remain compute-constrained. The Single-Multi Evolution Loop evaluates only $88\%$6 iterations and pools of $88\%$7 models, explicitly leaving scaling to dozens of models and broader collaboration strategies as future work (Feng et al., 5 Feb 2026). DDN depends on centralized state information during training and on careful tuning of intrinsic-reward masking, with $88\%$8 reported as optimal in ablations (Zhou et al., 5 Feb 2025). LiDAR-Anchored Collaborative Distillation continues to help under moderate LiDAR corruption, but extreme LiDAR degradation reduces gains (Jo et al., 13 Feb 2026).

Several misconceptions are corrected by these findings. Collaborative distillation does not require parameter sharing, since some methods communicate only logits, predictions, or averaged representations (Berdoz et al., 2022, Fan et al., 2023). It does not require a fixed external teacher, because teachers may be online ensembles, temporal means, frozen copies of the student, oracle planners with privileged information, or a consensus of peers (Wu et al., 2020, Sun et al., 2021, Cao et al., 10 Mar 2026). Self-learning does not necessarily mean autonomous pseudo-labeling; it may instead refer to challenge generation, self-distillation, self-supervision, intrinsic reward, or repeated evolution through collaboration (Wang et al., 2 Sep 2025, Ji et al., 2021, Feng et al., 5 Feb 2026). This suggests that the central unresolved question is not whether to collaborate or self-learn, but how to couple the two signals so that teacher quality, student plasticity, communication cost, and stability improve simultaneously rather than trading off against one another.