Collaborative Distillation Overview

• Collaborative Distillation is a family of methods where multiple interacting models generate supervisory signals instead of a single static teacher.
• It spans applications from low-level vision to continual, federated, and multimodal learning, using structured compatibility constraints.
• The approach optimizes training via encoder-decoder collaborations, ensemble teacher constructions, and modality integrations while avoiding extra inference cost.

Searching arXiv for recent and foundational papers on collaborative distillation to ground the article in the literature. Collaborative distillation is a family of knowledge-transfer methods in which the supervisory signal is produced by interacting models, modules, samples, modalities, or nodes rather than by a single static teacher. In the literature, “collaboration” has referred to the exclusive encoder–decoder pairing of low-level vision networks, co-trained peer branches and temporal mean teachers in online distillation, hybrid targets that combine replayed teacher outputs with current-model relations in continual learning, importance-weighted aggregation of multiple quantized teachers, cross-modal transfer from fused or LiDAR-aligned representations, and server–client synthesis of global distilled data in distributed settings (Wang et al., 2020, Yang et al., 2020, Wu et al., 2020, Ji et al., 2021, Pham et al., 2022, Kim et al., 2023, Gong et al., 2023, Holland et al., 2024, Jo et al., 13 Feb 2026). The term therefore denotes a research family rather than a single canonical algorithm.

1. Terminology, scope, and historical spread

By 2019, the phrase had already appeared in distinct technical senses. In top-$N$ recommendation, Collaborative Distillation reformulated KD for one-class collaborative filtering through a selective CF loss, probabilistic rank-aware sampling, and teacher- or student-guided feedback selection (Lee et al., 2019). In the same period, Collaborative Teaching Knowledge Distillation introduced a two-teacher scheme in which a scratch teacher provided step-by-step logits and an expert teacher supplied attention-based hints (Zhao et al., 2019).

During 2020, the term expanded rapidly across domains. It was used for ultra-resolution universal style transfer and facial makeup transfer via encoder–decoder collaboration (Wang et al., 2020, Yang et al., 2020), for online knowledge distillation through peer branches and online ensembling (Wu et al., 2020), and for video action recognition through collaboration in parameter and spectrum domains (Su et al., 2020). Subsequent work extended collaborative distillation to continual learning (Ji et al., 2021), low bit-width quantization with multiple teachers (Pham et al., 2022), federated learning (Gong et al., 2023), one-shot collaborative data distillation (Holland et al., 2024), semi-supervised sequence generation from LLM teachers (Zhao et al., 2023), community-based task-agnostic continual learning (Kaimakamidis et al., 2024), multi-modal domain generalization (Wang et al., 25 Nov 2025), LiDAR-anchored robustness transfer (Jo et al., 13 Feb 2026), and distributed communications and sensing (Nguyen et al., 17 Mar 2026).

The diversity of these formulations suggests a useful operational interpretation: collaborative distillation transfers not only predictions, but also compatibility constraints among cooperating entities. Depending on the task, those entities may be encoder–decoder pairs, peer students, teacher communities, replay buffers, fused and uni-modal branches, or server–client components.

Collaborative unit	Representative mechanism	Representative papers
Paired modules	Exclusive encoder–decoder relationship; linear embedding	(Wang et al., 2020, Yang et al., 2020)
Peer learners	Ensemble teacher, mean teacher, auxiliary learners	(Zhao et al., 2019, Wu et al., 2020, Liu et al., 2021)
Multiple teachers	Importance-aware shared knowledge	(Pham et al., 2022)
Continual or distributed nodes	Hybrid targets, local-to-central distillation, global synthetic data	(Ji et al., 2021, Gong et al., 2023, Holland et al., 2024, Kaimakamidis et al., 2024)
Modalities or samples	SVGD particles, EMA fused teacher, LiDAR anchor	(Kim et al., 2023, Wang et al., 25 Nov 2025, Jo et al., 13 Feb 2026)

2. Encoder–decoder collaboration in low-level vision

A foundational low-level vision line argued that encoder–decoder pairs form an “exclusive collaborative relationship,” meaning that the decoder is tuned to the statistics, feature spacings, and dynamic ranges of its matched encoder. In “Collaborative Distillation for Ultra-Resolution Universal Style Transfer” (Wang et al., 2020), this exclusivity was treated as knowledge to be distilled into a narrower student encoder. Because the student feature tensor has fewer channels, the method introduced a linear embedding loss

$\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$

and optimized

$$\mathcal{L}_{total}=\beta\sum_{i=1}^k \mathcal{L}_{embed}^{(i)}+\mathcal{L}_{collab}.$$

The student preserved stylization quality even when the model size was reduced by $15.5$ times, and the compressed WCT model achieved universal style transfer over $40$ megapixels on a $12$GB GPU for the first time (Wang et al., 2020).

“Compressing Facial Makeup Transfer Networks by Collaborative Distillation and Kernel Decomposition” (Yang et al., 2020) imported the same principle into BeautyGAN. Let $E(\cdot)$ be the teacher encoder, $E'(\cdot)$ the student encoder, $F_i$ and $F_i'$ their feature maps, and $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$0 a $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$1 linear mapping. The paper defined

$\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$2

and trained the student with

$\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$3

with $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$4, $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$5, $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$6, and $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$7. The student encoder kept the teacher’s topological layout but used fewer filters, the six $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$8 residual blocks were replaced by nine depth-wise + point-wise blocks, and the decoder remained identical to the teacher’s (Yang et al., 2020).

The empirical compression figures in BeautyGAN were explicit: parameter count $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$9M $$\mathcal{L}_{total}=\beta\sum_{i=1}^k \mathcal{L}_{embed}^{(i)}+\mathcal{L}_{collab}.$$0M, MACs $$\mathcal{L}_{total}=\beta\sum_{i=1}^k \mathcal{L}_{embed}^{(i)}+\mathcal{L}_{collab}.$$1G $$\mathcal{L}_{total}=\beta\sum_{i=1}^k \mathcal{L}_{embed}^{(i)}+\mathcal{L}_{collab}.$$2G, and inference time $$\mathcal{L}_{total}=\beta\sum_{i=1}^k \mathcal{L}_{embed}^{(i)}+\mathcal{L}_{collab}.$$3s $$\mathcal{L}_{total}=\beta\sum_{i=1}^k \mathcal{L}_{embed}^{(i)}+\mathcal{L}_{collab}.$$4s on Xeon E5-2620 v4. Makeup distance changed from $$\mathcal{L}_{total}=\beta\sum_{i=1}^k \mathcal{L}_{embed}^{(i)}+\mathcal{L}_{collab}.$$5 to $$\mathcal{L}_{total}=\beta\sum_{i=1}^k \mathcal{L}_{embed}^{(i)}+\mathcal{L}_{collab}.$$6, face distance from $$\mathcal{L}_{total}=\beta\sum_{i=1}^k \mathcal{L}_{embed}^{(i)}+\mathcal{L}_{collab}.$$7 to $$\mathcal{L}_{total}=\beta\sum_{i=1}^k \mathcal{L}_{embed}^{(i)}+\mathcal{L}_{collab}.$$8, and in a user study with $$\mathcal{L}_{total}=\beta\sum_{i=1}^k \mathcal{L}_{embed}^{(i)}+\mathcal{L}_{collab}.$$9 participants and $15.5$0 votes the student was preferred $15.5$1 of the time versus $15.5$2 for BeautyGAN (Yang et al., 2020).

These formulations depart sharply from logit-centric KD. In low-level vision, the distilled object is the dense feature subspace that a decoder can invert or stylize, not a class-probability simplex. This suggests that “collaboration” here names a compatibility structure internal to the network.

3. Collaborative teacher construction in supervised distillation

In classification-oriented settings, collaborative distillation often constructs a stronger or more informative teacher during training rather than assuming a single frozen expert. CTKD used two teachers with distinct functions: a scratch teacher $15.5$3, trained jointly with the student, and a pre-trained expert teacher $15.5$4, which provided attention maps. Its total objective was

$15.5$5

with $15.5$6 and an attention-transfer term built from normalized attention maps (Zhao et al., 2019). On CIFAR-10 with WRN-16-1, accuracy improved from $15.5$7 to $15.5$8; on Tiny-ImageNet, from $15.5$9 to $40$0 (Zhao et al., 2019).

Peer Collaborative Learning replaced the external teacher with a multi-branch training architecture. Each peer processed a different random augmentation of the same image, the high-level features were concatenated and passed through an extra classifier to form a peer ensemble teacher, and each peer also had a temporal mean teacher maintained by exponential moving average. The overall loss was

$40$1

On CIFAR-100 with ResNet110, the baseline error $40$2 dropped to $40$3, and the full PCL ensemble reached $40$4 (Wu et al., 2020).

Adversarial Collaborative Knowledge Distillation widened the student during training by attaching auxiliary learners at different depths, then fused auxiliary features through an Adversarial Collaborative Module with attention and adversarial diversity learning. Its total loss was

$40$5

with $40$6 in practice (Liu et al., 2021). The method explicitly retained only the original student at inference time. Across the reported classification benchmarks, it outperformed vanilla KD by about $40$7–$40$8 absolute and proxy-based ESKD or TAKD by about $40$9–$12$0 (Liu et al., 2021).

Collaborative Multi-Teacher KD for low bit-width networks used multiple quantized teachers $12$1. At layer $12$2, the teachers produced feature maps $12$3, which were aggregated into importance-aware shared knowledge

$12$4

This shared representation was both fed back into the next layer of each teacher and used as an intermediate target for the student (Pham et al., 2022). On CIFAR-100 with a $12$5-bit ResNet18, top-1 accuracy improved from $12$6 for a single $12$7-bit model to $12$8 for the full pipeline, surpassing the full-precision baseline of $12$9 (Pham et al., 2022).

In video action recognition, FPCD argued that small students should explicitly mimic temporal feature spectra and parameter frequency distributions. Its composite objective,

$E(\cdot)$0

combined spectrum loss, parameter-distribution KL, and an epoch-dependent collaborative weighting schedule (Su et al., 2020). On Kinetics-400 with STM-ResNet-18, top-1 accuracy improved from $E(\cdot)$1 to $E(\cdot)$2, outperforming simple KD and correlation-based KD (Su et al., 2020).

4. Continual, federated, distributed, and recommendation-oriented formulations

In continual learning, collaboration often addresses the mismatch between old and new knowledge sources. Complementary Calibration defined “relation deviation” as the deficiency of soft targets when the old model has never seen the new classes. Its collaborative distillation formed a hybrid teacher

$E(\cdot)$3

where $E(\cdot)$4 arose from label propagation on a feature-similarity graph (Ji et al., 2021). On CIFAR-100 with $E(\cdot)$5 tasks and buffer $E(\cdot)$6, ER alone achieved about $E(\cdot)$7 overall accuracy, while ER + CKD reached about $E(\cdot)$8 (Ji et al., 2021).

In top-$E(\cdot)$9 recommendation, collaborative distillation was adapted to the one-class collaborative filtering regime. The student minimized

$E'(\cdot)$0

with a selective CF loss over positives only and a KD loss defined on a sampled subset of missing items chosen by probabilistic rank-aware sampling (Lee et al., 2019). Reported gains over the state of the art were $E'(\cdot)$1–$E'(\cdot)$2 in HR and $E'(\cdot)$3–$E'(\cdot)$4 in NDCG, and the student achieved performance comparable to the teacher (Lee et al., 2019). Continual Collaborative Distillation later extended teacher–student recommendation KD to non-stationary data streams by alternating list-wise distillation, proxy-guided replay, and teacher updates, achieving the highest $E'(\cdot)$5-mean for teacher and student across blocks on Gowalla and Yelp (Lee et al., 2024).

Federated and distributed variants reinterpret collaboration at the system level. FedIOD replaced recursive parameter sharing with one-way, data-free distillation from local teachers to a central student by jointly optimizing input-space and output-space objectives. Its formulation combined confidence, uniqueness, mimic, and GAN realism terms in a min–max game over student, generator, and local discriminators (Gong et al., 2023). On CIFAR-10 with $E'(\cdot)$6 non-IID splits, it achieved $E'(\cdot)$7 and $E'(\cdot)$8 for Dirichlet $E'(\cdot)$9 and $F_i$0, respectively (Gong et al., 2023).

CollabDM addressed distributed dataset condensation rather than model-parameter transfer. It minimized a distribution-matching objective over a global synthetic set and required exactly one round of communication between clients and server (Holland et al., 2024). Clients sent back local synthetic sets and random-encoder embedding means; the server refined the union by gradient descent. Under Dirichlet-skewed partitions with IPC $F_i$1, CollabDM on MNIST achieved $F_i$2, $F_i$3, and $F_i$4 for $F_i$5, while CollabDM-pae reached $F_i$6, $F_i$7, and $F_i$8 (Holland et al., 2024).

LENC generalized collaborative distillation to a community of deployed nodes with dynamic teacher–student roles, OOD-based self-assessment, and multiple transfer policies ranging from data transfer to logit distillation, feature+logit distillation, and model copy (Kaimakamidis et al., 2024). This framework explicitly targeted task-agnostic continual learning without task-boundary information. A broader systems view was later provided for distributed communications and sensing, where collaborative KD was organized into one-to-many, many-to-one, and peer-to-peer architectures for resource-constrained 6G nodes (Nguyen et al., 17 Mar 2026).

5. Multimodal, generative, and language-centered variants

Several recent formulations shift the collaborative object from teachers to samples, modalities, or iterative relabeling loops. Collaborative Score Distillation treated multiple images as SVGD particles. For particles $F_i$9, the update was

$F_i'$0

with the diffusion-model score replacing $F_i'$1 (Kim et al., 2023). On DAVIS-2017 video editing, CSD-Edit obtained CLIP directional similarity $F_i'$2, CLIP consistency $F_i'$3, and LPIPS $F_i'$4, compared with $F_i'$5 for FateZero and $F_i'$6 for Pix2Video (Kim et al., 2023).

In multi-modal domain generalization, MBCD used a student multi-branch model and a weight-averaged teacher with parameters

$F_i'$7

setting $F_i'$8 in practice (Wang et al., 25 Nov 2025). Adaptive modality dropout, a gradient consistency constraint, and KL-based cross-modal distillation from the teacher’s fused logits jointly countered early dominance of strong modalities. On EPIC-Kitchens with all three modalities, MBCD achieved $F_i'$9 average Top-1 accuracy versus $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$00 for CMRF and below $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$01 for non-WA methods (Wang et al., 25 Nov 2025).

LiDAR-Anchored Collaborative Distillation used two asymmetric stages. Stage 1 aligned LiDAR features to clear-day 2D features with

$\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$02

and Stage 2 used the aligned LiDAR encoder as a fixed anchor to update the 2D encoder with

$\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$03

(Jo et al., 13 Feb 2026). For ViT-B/14, nuScenes semantic linear probing improved from $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$04 to $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$05 mIoU, KITTI depth linear probing from $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$06 to $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$07 RMSE, and night DVPS improved from VPQ $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$08 to $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$09 while depth RMSE improved from $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$10 to $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$11 (Jo et al., 13 Feb 2026).

In sequence generation, MCKD first prompted GPT-3.5 Turbo with $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$12 in-context examples to pseudolabel unlabeled data, then repeatedly trained two students on disjoint partitions and used each to relabel the opposite partition (Zhao et al., 2023). On CRAFT with $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$13 labeled examples, $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$14-stage MCKD achieved $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$15 parsing F1, exceeding GPT-3.5 at $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$16 and vanilla KD at $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$17, and matching supervised finetuning with $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$18 labeled examples (Zhao et al., 2023).

Other language- and fusion-oriented work used multi-teacher output fusion and self-learning. A parameter-efficient LM deployment framework fused teacher probabilities as

$\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$19

with entropy-driven dynamic teacher weights and a feature-alignment loss (Meng et al., 21 Jul 2025). On C4, its student reported perplexity $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$20, KL loss $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$21, and BLEU $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$22, outperforming TinyBERT, MobileBERT, MiniLM, and DKD (Meng et al., 21 Jul 2025). In infrared-visible image fusion, AgentFuse combined teacher guidance and self-learning under an RL agent that adjusted $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$23, $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$24, and degradation parameters; on M3FD it achieved AG $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$25 versus $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$26 for the teacher LDFusion (Wang et al., 2 Sep 2025).

6. Empirical patterns, misconceptions, and open issues

A recurrent misconception is that collaborative distillation denotes a symmetric peer-learning protocol. The literature does not support that restriction. Encoder–decoder collaboration in style transfer and makeup transfer is asymmetric (Wang et al., 2020, Yang et al., 2020); CTKD combines a moving scratch teacher with a fixed expert teacher (Zhao et al., 2019); FedIOD is a local-to-central one-way distillation framework (Gong et al., 2023); LiDAR-anchored training alternates which encoder receives gradient across two stages (Jo et al., 13 Feb 2026). “Collaboration” therefore refers to the structure of knowledge exchange, not to role symmetry.

A second misconception is that collaborative distillation always requires additional inference cost. Several methods attach training-only structures and remove them at test time. ACKD strips away auxiliary learners and the ACM during inference, keeping only the original student (Liu et al., 2021). CTKD retains the student after the two-teacher training phase (Zhao et al., 2019). This suggests that collaboration is frequently a training-time mechanism for reshaping the student’s hypothesis space.

Empirically, collaborative distillation tends to be introduced when single-teacher KD leaves some structure unmodeled: dense feature compatibility in low-level vision, inter-class relations in continual learning, modality imbalance in multimodal DG, temporal or spectral structure in video, or client heterogeneity in distributed settings (Yang et al., 2020, Ji et al., 2021, Wang et al., 25 Nov 2025, Su et al., 2020, Holland et al., 2024). The distributed communications and sensing perspective makes this especially explicit by treating response-based, feature-based, and relation-based distillation as interchangeable building blocks within one-to-many, many-to-one, and peer-to-peer deployment architectures (Nguyen et al., 17 Mar 2026).

The open problems recorded in the literature are correspondingly heterogeneous. CollabDM identifies a bandwidth–fidelity trade-off because increasing the number of random seeds $\mathcal{L}_{embed}=\|F-Q\cdot F'\|_2^2,$27 raises fidelity at the cost of bandwidth, and notes that privacy guarantees and dynamic client availability remain open (Holland et al., 2024). LENC reports that OOD-module reliability depends on sufficient data for KSA training and that communication overhead grows with community size (Kaimakamidis et al., 2024). CSD notes limitations inherited from the underlying diffusion prior, including possible language misunderstanding, societal bias, flicker, and viewpoint-editing limits (Kim et al., 2023). The LM multi-teacher framework points to unresolved teacher-conflict handling and cross-lingual or cross-modal extensions (Meng et al., 21 Jul 2025).

Taken together, the literature presents collaborative distillation as a unifying response to a common systems problem: the useful knowledge to be transferred is often distributed across interacting components rather than localized in one pretrained teacher. The technical forms vary widely, but the central premise is consistent: when the target behavior depends on relations among modules, classes, modalities, samples, or nodes, distillation must preserve those relations explicitly rather than treating them as incidental by-products of logits alone.