Cross Knowledge Distillation (CKD)

• Cross Knowledge Distillation (CKD) is a research pattern that transfers learned knowledge across heterogeneous sources such as modalities, domains, or model architectures.
• CKD encompasses various implementations—including contrastive, collaborative, and cascaded approaches—that address alignment challenges between diverse teacher and student setups.
• Empirical studies demonstrate that CKD improves performance in tasks like biometric recognition, recommendation systems, and RF sensing by achieving notable gains in accuracy and efficiency.

Searching arXiv for recent and foundational uses of “Cross Knowledge Distillation (CKD)” to ground the article in the literature. Cross Knowledge Distillation (CKD) denotes a family of knowledge-distillation formulations in which the transfer pathway is explicitly “cross”: across modalities, domains, model families, languages, or other heterogeneous sources. In the surveyed arXiv literature, the term is not standardized. It appears as Consistent Knowledge Distillation for face-to-periocular transfer (Jung et al., 2020), Cross Knowledge Distillation for multi-domain sequential recommendation (Sun et al., 2024) and for ANN-to-SNN transfer (Ye et al., 12 Jul 2025), Correlated Knowledge Distillation for vision-to-RF pose estimation (Pokhrel et al., 2023), and, in adjacent usages of the same acronym, as Collaborative, Contrastive, and Cascaded Knowledge Distillation (Kaimakamidis et al., 2024, Gao et al., 2021, Zhu et al., 2024, Gkalelis et al., 11 May 2026). This suggests that CKD is best understood not as a single canonical algorithm but as a research pattern centered on heterogeneous knowledge transfer.

1. Terminology and definitional scope

The acronym CKD is paper-specific. In some works it names a particular mechanism for cross-modal transfer; in others it names a broader training framework or a distinct distillation objective.

Paper	Meaning of CKD	Setting
(Jung et al., 2020)	Consistent Knowledge Distillation	Face teacher, periocular student
(Sun et al., 2024)	Cross Knowledge Distillation	Multiple pre-trained recommendation teachers, compact student
(Pokhrel et al., 2023)	Correlated Knowledge Distillation	Camera-fed teacher, RF student
(Ye et al., 12 Jul 2025)	Cross Knowledge Distillation	ANN on RGB, SNN on DVS
(Kaimakamidis et al., 2024)	Collaborative Knowledge Distillation	Peer-node online unlabeled distillation
(Gkalelis et al., 11 May 2026)	Cascaded Knowledge Distillation	Bottom-up staged TA-to-student, then teacher-to-student

The ambiguity is explicit in the literature. DistilCSE uses CKD to mean Contrastive Knowledge Distillation, with InfoNCE replacing MSE during sentence-embedding distillation (Gao et al., 2021). A separate vision paper also uses CKD for Contrastive Knowledge Distillation from a sample-wise perspective, where teacher–student logits of the same sample are the positive pair and cross-sample logits are negatives (Zhu et al., 2024). In consequence, “CKD” by itself does not identify a unique method; the expansion and operational meaning must be read from the specific paper.

2. Canonical teacher–student settings

A dominant CKD pattern is cross-modal teacher guidance with student-only deployment. In periocular recognition, the training set is paired as $\{(x_i, x_i^F, y_i)\}_{i=1}^N$, where $x_i$ is a periocular crop and $x_i^F$ is the corresponding face image, and only the periocular branch is retained at test time (Jung et al., 2020). In RF sensing, synchronized pairs $(I_t, C_t)$ connect webcam frames and WiFi CSI; the student is trained from RF while being supervised by a camera-fed teacher, and inference uses only RF signals (Pokhrel et al., 2023). In ANN-to-SNN transfer, training uses RGB and DVS counterparts, but testing uses DVS only (Ye et al., 12 Jul 2025). Cross-modal action recognition follows the same deployment logic: a teacher trained on RGB or optical flow supervises students operating on 3D skeleton poses, using paired but unlabeled samples of both modalities (Thoker et al., 2019).

A second pattern is cross-domain or cross-source distillation from multiple heterogeneous teachers. In multi-domain sequential recommendation, CKD-MDSR is instantiated by PRM-KD, which distills from UniSRec, Recformer, and UniM$^2$Rec into a compact student such as SASRec. Distillation is performed on in-batch item-score distributions, and teacher outputs are fused through per-user confidence/consistency gating (Sun et al., 2024). In cross-lingual temporal knowledge-graph reasoning, MP-KD trains a teacher on a source TKG and a student on a target TKG, with an alignment module, pseudo alignment generation, and temporal event transfer serving as the cross-lingual bridge (Wang et al., 2023).

A third pattern is staged or peer-to-peer transfer rather than fixed one-way supervision. LLaVA-CKD uses a bottom-up cascade in which a teacher assistant first distills to the student and the strongest teacher distills to the improved student in a second stage (Gkalelis et al., 11 May 2026). LENC defines CKD as an online collaborative process among deployed peer nodes, where a node can dynamically act as student or teacher, teacher selection is policy-driven, and knowledge exchange is triggered by unlabeled streams encountered at deployment time (Kaimakamidis et al., 2024).

3. Objective design and theoretical interpretations

There is no single CKD loss. Different papers instantiate CKD with markedly different objectives, reflecting the heterogeneity of the transfer setting.

In face-to-periocular transfer, the full objective is

$L_{full}=L_{id}+L_{pred},$

where $L_{id}$ is cross-entropy over both branches and $L_{pred}$ is a symmetric KL divergence between temperature-smoothed face and periocular posteriors. In that formulation, feature-layer consistency is not imposed by an explicit distance term; it is realized structurally by shared weights and shared batch-normalization statistics (Jung et al., 2020). The same paper proves that CKD is equivalent, up to scale, to learned label smoothing with smoothed targets

$$\tilde y=\frac{y+\tau p_\tau}{1+\tau},\qquad \tilde y^F=\frac{y+\tau p_\tau^F}{1+\tau},$$

plus a sparsity-oriented regularizer

$$R(z)=-\log\left[\frac{\sum_k e^{z_k}}{\left(\sum_k e^{z_k/\tau}\right)^\tau}\right].$$

This theoretical result is central because it recasts prediction-layer consistency as adaptive smoothing rather than fixed uniform smoothing (Jung et al., 2020).

In recommendation, the distilled signal is a fused multi-teacher in-batch distribution

$x_i$0

and the student is trained with its own recommendation loss plus

$x_i$1

The teachers are selectively integrated through

$x_i$2

and the least consistent teacher can be dropped for a user when disagreement exceeds $x_i$3 (Sun et al., 2024).

In RF sensing, CKD explicitly decomposes logit distillation into teacher-class and student-class components:

$x_i$4

where $x_i$5 distills the binary relevant-versus-non-relevant prediction and $x_i$6 distills the normalized distribution over non-relevant classes. Both are implemented with normalized squared error on log-soft probabilities (Pokhrel et al., 2023). This correlated-logit formulation differs from standard softened-logit KD by weighting the non-relevant-class term with the teacher’s relevance probability.

In ANN-to-SNN transfer, the objective couples classification, temporal domain alignment, and phased KD:

$x_i$7

Here $x_i$8 is a CKA-based loss between static and dynamic SNN features, $x_i$9 is a sigmoid phase-switching function that gradually turns off KD, and the static stream undergoes “sliding replacement” by DVS inputs according to

$x_i^F$0

The method therefore treats cross-modality and cross-architecture mismatch separately: semantic similarity and sliding replacement address RGB–DVS mismatch, while phased KD addresses ANN–SNN mismatch (Ye et al., 12 Jul 2025).

Contrastive variants replace KL-style matching with InfoNCE. DistilCSE uses student embeddings as anchors and teacher embeddings as positives and negatives, with optional memory-bank negatives:

$x_i^F$1

The rationale is objective consistency: teacher training, distillation, and student fine-tuning all use InfoNCE (Gao et al., 2021). A separate sample-wise logit formulation defines teacher–student logits of the same sample as positives and cross-sample student logits as negatives:

$x_i^F$2

which is then added to the supervised task loss as $x_i^F$3 (Zhu et al., 2024).

4. Architectural strategies and training protocols

CKD methods frequently encode “cross” structure not only in the loss but also in the architecture. In the periocular model, a single backbone $x_i^F$4 with shared weights and shared batch-normalization statistics processes both face and periocular inputs, while separate heads $x_i^F$5 and $x_i^F$6 specialize for each modality (Jung et al., 2020). This arrangement makes feature consistency structural rather than explicitly penalized.

RF sensing adopts a dual multi-branch design. The teacher branch is AlphaPose with YOLOv3 detector and ResNet-50 backbone; it outputs 18 keypoints that are converted into a pose adjacency matrix $x_i^F$7. The student branch receives CSI from an SDR pipeline, upsamples it with eight stacked transposed convolution layers, extracts features with a revised ResNet built from custom ResidualBlock modules, and regresses a predicted pose adjacency matrix $x_i^F$8 while also producing CKD logits (Pokhrel et al., 2023).

ANN-to-SNN CKD uses a shared SNN with a static stream and a dynamic stream. RGB images are converted to HSV, the Value channel is replicated across timesteps to produce a static input, and DVS event frames drive the dynamic input. The hybrid stream is gradually transformed from RGB-dominant to DVS-dominant by Bernoulli replacement governed by $x_i^F$9, while CKA aligns static and dynamic features across timesteps (Ye et al., 12 Jul 2025).

In recommendation, architecture heterogeneity is deliberately tolerated rather than suppressed. UniSRec, Recformer, and UniM$(I_t, C_t)$0Rec can all produce user–item scores for in-batch items, and this commonality is sufficient for unified distillation. The student may be SASRec, FM, DeepFM, or LightGCN, since the distilled object is the in-batch score distribution rather than attention maps or hidden states (Sun et al., 2024).

Collaborative and cascaded variants expand the architectural repertoire further. LENC nodes comprise a shared feature module, task-specific decision heads, Knowledge Self-Assessment modules for OOD detection, and interaction rules that govern teacher selection and transfer policy (Kaimakamidis et al., 2024). LLaVA-CKD keeps a common LLaVA-style decomposition—SigLIP vision encoder, MLP2x_GELU connector, Qwen2.5 backbone—but stages student training through a teacher assistant before the strongest teacher is applied (Gkalelis et al., 11 May 2026).

5. Application domains and empirical findings

CKD has been applied to biometric recognition, recommendation, RF sensing, neuromorphic vision, language representation, multimodal sentiment analysis, object detection, and temporal knowledge-graph reasoning.

In periocular recognition, CKD achieves an average identification rank-1 of 88.96% across six datasets, compared with 85.49% for the periocular cross-entropy baseline, and an average EER of 7.11% versus 9.55%. The paper reports relative gains of +49% for identification and +54% for verification when normalized against a face upper bound (Jung et al., 2020). In RF sensing, CKD improves Top-1 accuracy from 70.23% for standard KD to 76.46%, while keeping “same parameters” and reporting 11 ms training time per unit versus 10 ms for vanilla KD (Pokhrel et al., 2023).

In multi-domain sequential recommendation, PRM-KD improves SASRec across all five Amazon subsets reported. On Instruments, it raises NDCG@10 from 0.0635 to 0.0757 and Recall@10 from 0.0979 to 0.1089; on Arts, Recall@10 increases from 0.1108 to 0.1272 and NDCG@10 from 0.0587 to 0.0695 (Sun et al., 2024). On Arts, the method preserves the student’s online cost while reporting normalized training speed 1.2x, inference memory 1.0x, and inference speed 1.0x, compared with much higher inference memory for the teachers (Sun et al., 2024).

In ANN-to-SNN transfer, CKD reports 96.71 ± 0.30% on N-Caltech101 with best 97.13%, and 38.80 ± 1.23% on CEP-DVS with best 40.20% (Ye et al., 12 Jul 2025). In sentence embeddings, DistilCSE’s 110M CKD student reaches average Spearman 85.04 on seven STS tasks, exceeding Sentence-T5 11B at 84.94, while using 1% parameters and 0.25% unlabeled data (Gao et al., 2021). In sample-wise visual CKD, ResNet34$(I_t, C_t)$1ResNet18 on ImageNet-1K improves from 70.66/89.88 with Vanilla KD to 72.24/90.81, and ResNet50$(I_t, C_t)$2MobileNetV2 improves from 68.58/88.98 to 72.97/91.36 (Zhu et al., 2024).

In multimodal sentiment analysis under incomplete modalities, MM-CKD achieves competitive results without imputation and reports 5.2×10^9 test FLOPs, versus 17.4×10^9 for DicMOR and IMDer on the reported cost comparison (Sang et al., 2024). In cross-lingual temporal knowledge-graph reasoning, MP-KD reports on average +33% MRR and +30% H@10 over the best no-transfer baseline, with roughly 14–16% relative improvements over the strongest transfer baselines and markedly better robustness to noisy seed alignments (Wang et al., 2023). In online unlabeled collaborative CKD, LENC reports state-of-the-art results on CIFAR-10 and CIFAR-100 settings; for example, on CIFAR-10 with stream size 5000 and ResNet-18 students, LENC reaches 86.31±0.32 versus 85.76±0.07 for KDCL, and on CIFAR-100 with stream size 5000 and ResNet-18 students it reaches 65.02±0.13 versus 58.70±0.09 for KDCL (Kaimakamidis et al., 2024).

6. Ambiguities, limitations, and related directions

A common misconception is that CKD necessarily means a softened-logit KL term between a single teacher and a single student. The literature does not support that view. CKD may use symmetric KL with shared batch statistics (Jung et al., 2020), KL to a fused multi-teacher target (Sun et al., 2024), normalized squared error on correlated logits (Pokhrel et al., 2023), CKA-based domain alignment plus phased KD (Ye et al., 12 Jul 2025), InfoNCE on embeddings (Gao et al., 2021), sample-wise InfoNCE on logits (Zhu et al., 2024), or peer-to-peer online transfer among deployed models (Kaimakamidis et al., 2024).

Another misconception is that CKD is intrinsically a multimodal method. Several influential formulations are indeed multimodal or cross-architecture, but CKD also appears in multi-domain recommendation (Sun et al., 2024), cross-lingual TKG reasoning (Wang et al., 2023), and bottom-up VLM distillation where the acronym means Cascaded rather than Cross Knowledge Distillation (Gkalelis et al., 11 May 2026). The term therefore describes a transfer principle more than a fixed input configuration.

The limitations are correspondingly diverse. Many cross-modal formulations require paired or synchronized data during training, as in face–periocular pairing, synchronized image–CSI pairs, RGB–DVS correspondence, or RGB–skeleton pairing (Jung et al., 2020, Pokhrel et al., 2023, Ye et al., 12 Jul 2025, Thoker et al., 2019). Multi-teacher recommendation depends on teacher calibration, domain priors $(I_t, C_t)$3, and a threshold $(I_t, C_t)$4 that can over-drop useful teachers or underweight valuable signals (Sun et al., 2024). Cross-lingual TKG transfer is constrained by scarce or noisy alignments, even though MP-KD adds pseudo alignment generation and temporal attention to mitigate the problem (Wang et al., 2023). Collaborative CKD depends on reliable OOD-based Knowledge Self-Assessment and incurs communication overhead that scales with community size (Kaimakamidis et al., 2024). Multimodal sentiment CKD remains weaker when lexical input is absent, especially on audio-only, video-only, and audio-video settings (Sang et al., 2024). Bottom-up cascaded VLM distillation approximately doubles training time relative to single-stage KD, since one teacher-assistant stage is added (Gkalelis et al., 11 May 2026).

The surveyed papers point to several convergent directions. One is adaptive transfer strength, seen in temporal cross-lingual attention, per-user consistency gating, and phased KD scheduling (Wang et al., 2023, Sun et al., 2024, Ye et al., 12 Jul 2025). A second is structural rather than purely loss-based alignment, such as shared backbones, shared statistics, or modality-sorted multimodal pipelines (Jung et al., 2020, Gkalelis et al., 11 May 2026). A third is contrastive reformulation, where preserving relational geometry across samples or views replaces direct mimicry of logits or features (Gao et al., 2021, Zhu et al., 2024, Sang et al., 2024). Taken together, these developments indicate that CKD research has shifted from simple teacher-to-student imitation toward more selective, geometry-aware, and deployment-conscious transfer mechanisms.