Cross-Domain Distillation (CDD)

• Cross-Domain Distillation (CDD) is a transfer-learning framework that distills supervision from one representation regime to another to handle distribution shifts.
• CDD employs techniques like teacher–student transfer, self-distillation, and relation-preserving mechanisms to align domain-invariant and domain-aware features.
• Empirical studies across vision, language, and scientific signals show that CDD effectively improves performance by maintaining useful invariances while suppressing domain-specific artifacts.

Cross-Domain Distillation (CDD) denotes a family of transfer-learning procedures that transfer task-relevant knowledge across domains by distilling supervision from one representation regime into another under distribution shift. In the literature, the term covers classical teacher–student transfer from a source-trained model to target-domain data, self-distillation across different views of the same target sample, feature-level and relation-level distillation, and broader distillation-by-disentanglement or correlation-preserving formulations rather than only logit matching (Islam et al., 2021, Feng et al., 2023, Hu et al., 2019, Gao et al., 2022, Lee et al., 2022). The underlying problem is always the same: a model that is reliable in one domain is biased, miscalibrated, or structurally mismatched in another, so transfer must preserve useful invariances while suppressing domain-specific artifacts (Lee et al., 17 Mar 2026, gu et al., 2024, Chen et al., 4 Jul 2026, Zhang et al., 19 Mar 2026).

1. Definition and semantic scope

The meaning of CDD is broader than fixed-teacher pseudo-labeling. In cross-domain few-shot recognition and unsupervised domain adaptation, CDD often means that a source-trained teacher produces supervisory signals on target-domain inputs, and a student is optimized to match them under augmentations, masking, or momentum updates (Islam et al., 2021, Feng et al., 2023). In cross-dataset text-to-image person retrieval, the same idea appears as momentum teacher–student contrastive distillation over unlabeled target data, coupled with graph propagation across source and target memories (Luo et al., 25 Jan 2025).

Other papers use the term “distillation” in a less classical sense. In aspect-level sentiment transfer, domain-invariant sentiment features are “distilled” by separating them from domain-dependent aspect features through Context Allocation, Aspect Detection, and adversarial alignment, without a teacher–student KL term (Hu et al., 2019). In nighttime semantic segmentation, “distillation” refers to enforcing invariance of content and style correlations across domain pairs that isolate illumination-only and dataset-only shifts, again without a standard teacher–student architecture (Gao et al., 2022). In domain generalization, XDED constructs a cross-domain ensemble of same-label logits inside a mini-batch and distills each sample toward that ensemble, so the teacher is an on-the-fly domain-agnostic soft target rather than a separate network (Lee et al., 2022).

A second boundary concerns invariance. Some CDD methods explicitly seek domain-invariant representations, but others preserve or exploit domain-aware cues when they are predictive in the target domain. TAMEPT argues that previous work overemphasized domain-invariant or task-agnostic features and used self-supervised distillation specifically to amplify domain-aware target tokens (Feng et al., 2023). DIFD, by contrast, treats domain-dependent aspect information as an orthogonal signal whose explicit modeling helps purify domain-invariant sentiment representations (Hu et al., 2019). CDD is therefore not a single algorithmic template; it is a class of knowledge-transfer mechanisms defined by cross-domain supervision.

2. Problem settings and sources of domain shift

CDD appears in several distinct supervision regimes. A large group of methods assumes access to unlabeled target-domain data during training. This includes cross-domain few-shot recognition with unlabeled target images (Islam et al., 2021, Zheng et al., 2023), unsupervised domain adaptation for text classification (Feng et al., 2023), cross-dataset text-to-image person retrieval (Luo et al., 25 Jan 2025), and cross-domain object detection with unlabeled target images and labeled source or target-like data (He et al., 2022). In these settings, the central problem is that source-domain decision boundaries, class geometry, or feature statistics do not transfer directly to the target.

A second group operates without target-domain data at training time. CD-FKD studies single-domain generalization in object detection, where the teacher sees clean source images and the student sees diversified source images produced by downscaling and ImageNet-C corruptions; “cross-domain” here refers to clean versus corrupted/low-resolution domains rather than source versus external target data (Lee et al., 17 Mar 2026). XDED likewise belongs to domain generalization: only multiple source domains are available, and the model must generalize to unseen targets by distilling same-label consensus across source domains (Lee et al., 2022).

A third regime involves structured temporal or continual shifts. ContiStain addresses continual multi-domain virtual IHC staining, where biomarker domains arrive sequentially and previous-domain data are unavailable; CDD preserves cross-domain token-level relations to mitigate forgetting (Chen et al., 4 Jul 2026). FUAD CDD addresses unlabeled training sets contaminated by anomalies, constructing multiple lower-contamination “domains” and aggregating pseudo-normal signals across them (Liu et al., 25 Aug 2025). STEP studies scientific time series with heterogeneous sequence lengths, channel counts, and numeric scales, distilling from foundation models pretrained on audio, general time series, and neural signals into a unified scientific encoder (Zhang et al., 19 Mar 2026).

The sources of shift are correspondingly varied: natural images versus medical or satellite imagery in few-shot recognition (Islam et al., 2021), source and target review domains in sentiment classification (Feng et al., 2023), weather, lighting, corruption, and scale changes in detection (Lee et al., 17 Mar 2026), resolution mismatch in human pose estimation (gu et al., 2024), modality gaps such as image–speech or audio–video (Zhao et al., 22 Jul 2025, Sarkar et al., 2022), and sequential biomarker drift in pathology (Chen et al., 4 Jul 2026).

3. Distillation operators and optimization patterns

A canonical CDD pattern is momentum-teacher consistency. In Dynamic Distillation Network, the teacher is updated as an exponential moving average of the student,

$$\theta_t \leftarrow m \cdot \theta_t + (1-m)\cdot \theta_s,$$

and the student optimizes a supervised loss on labeled base data together with a consistency term on unlabeled images,

$$L = L_{\mathrm{sup}} + \lambda L_{\mathrm{cons}}, \qquad L_{\mathrm{cons}} = E_{x \sim \mathcal U}\!\left[ KL\!\big(p_t(y|a_w(x)) \,\|\, p_s(y|a_s(x))\big)\right].$$

The weakly augmented teacher view provides a smoother target than hard pseudo-labels under large domain shift (Islam et al., 2021).

A second pattern is self-distillation without a separate teacher. TAMEPT uses the model’s own prediction on the original target prompt as a stop-gradient teacher and distills it into the masked version of the same input:

$$L_{\mathrm{ssd}}(D_t;\theta)=E_{x\in D_t} KL\!\big(p_\theta(y|x_{pm}) \,\|\, p_\theta(y|x_p)\big).$$

There is no EMA teacher and no target labels; the masking operation forces the model to rely on remaining target-domain cues that are predictive for the downstream task (Feng et al., 2023).

Feature-level CDD is prominent in dense prediction. CD-FKD distills a frozen clean teacher into a student trained on downscaled and corrupted inputs, using cosine similarity both globally on the final backbone map and locally on RoI-aligned instance features:

$$L_{\mathrm{total}} = L_{\mathrm{det}}^s + \alpha L_{\mathrm{global}} + \beta L_{\mathrm{inst}}.$$

No logit distillation is used; the method focuses on feature invariance under appearance degradation (Lee et al., 17 Mar 2026). Cross-Level Distillation for cross-domain few-shot classification transfers deeper teacher information into shallower student blocks, with an “old student” gradually replacing the teacher to reduce source bias (Zheng et al., 2023). In low-resolution pose estimation, CDKD introduces a Scale-Adaptive Projector Ensemble to align student and teacher feature maps of different spatial sizes and a Cross-Class Alignment module to merge teacher SimCC bins when class counts differ across resolutions (gu et al., 2024).

Relation-preserving variants replace pointwise matching with structural constraints. ContiStain computes cross-domain token-level cosine similarity matrices for old biomarker domains and preserves them across continual updates by minimizing

$$L_{\mathrm{rel}} = \frac{1}{|\mathcal P||\mathcal S|}\sum_{(i,j)\in\mathcal P}\sum_{\ell\in\mathcal S} \|R^{(t)}_\ell(i,j)-R^{(t-1)}_\ell(i,j)\|_1,$$

thereby constraining cross-domain geometry rather than only outputs (Chen et al., 4 Jul 2026). CCDistill similarly distills invariance of content and style correlations across illumination and dataset differences (Gao et al., 2022). XDED forms an ensemble logit target by averaging same-label logits from different source domains and penalizes the KL divergence between each prediction and that ensemble, which is a self-distillation variant designed to increase posterior entropy and favor flatter minima (Lee et al., 2022). In widely differing modalities, hard L2 alignment is reported to overfit, and soft margin-based feature distillation plus a shared classifier are used instead (Zhao et al., 22 Jul 2025).

4. Representative instantiations across tasks

In cross-domain few-shot vision, Dynamic Distillation Network replaces STARTUP’s fixed teacher with an EMA teacher updated on unlabeled target images (Islam et al., 2021). ProLAD addresses batch-statistic mismatch by training two adapters—one without normalization and one with Standard Normalization plus GroupConv—and uses adaptive episode-wise distillation to decide how much the normalization-aware branch should influence adaptation (Yang et al., 2023). Cross-Level Distillation and Feature Denoising complements this line by supervising shallower student layers with deeper teacher features and removing low-activation feature components at evaluation time to reduce few-shot overfitting (Zheng et al., 2023).

In language and retrieval, TAMEPT uses masked language modeling and self-supervised distillation to adapt prompt-based sentiment classifiers across review domains (Feng et al., 2023). DIFD uses aspect detection as an orthogonal, domain-dependent task to siphon aspect-specific information away from domain-invariant sentiment features (Hu et al., 2019). GCKD extends CDD to cross-modal retrieval: graph-based multi-modal propagation links source and target embeddings, while a momentum teacher supplies soft contrastive targets and confident cross-domain matches for unlabeled target data (Luo et al., 25 Jan 2025).

In dense prediction and structured vision, TDD introduces a Target Proposal Perceiver that injects target-like proposal context into a source-adaptive branch through geometry-aware cross-attention, followed by dual-branch self-distillation on pseudo labels (He et al., 2022). CD-FKD applies clean-to-corrupted feature distillation for robust single-domain generalization in detection (Lee et al., 17 Mar 2026). CCDistill performs one-stage nighttime semantic segmentation by preserving content and style correlation invariances across paired day/night and cross-dataset samples (Gao et al., 2022). CDKD handles high-resolution to low-resolution pose transfer by explicitly solving feature-size mismatch and output-class mismatch (gu et al., 2024).

In medical imaging, anomaly detection, video, and scientific signals, the same design space reappears with domain-specific constraints. The unified medical framework of (Ciprian-Mihai et al., 2 May 2026) fuses multiple teachers from heterogeneous datasets into a joint teacher and distills multi-level features, logits, attention maps, and ROI/query representations into compact students for segmentation, classification, and detection. ContiStain performs continual virtual staining with domain-aware MoE token spaces and relation-preserving distillation (Chen et al., 4 Jul 2026). FUAD CDD creates multiple lower-anomaly domains and aggregates pseudo-normal features across domain-specific students to protect global anomaly detectors from contaminated training data (Liu et al., 25 Aug 2025). XKD combines masked reconstruction with cross-modal KD and MMD-based domain alignment in audio–visual video representation learning (Sarkar et al., 2022). DMSD-CDFSAR uses a mixed-source teacher branch, temporal cycle consistency, and dual distillation for cross-domain few-shot action recognition (Guo et al., 2024). STEP distills simultaneously from audio, general time-series, and neural-signal foundation models into a unified scientific encoder with adaptive patching and statistics compensation (Zhang et al., 19 Mar 2026).

5. Empirical evidence

The reported gains are task-specific and metric-specific, so they are not directly comparable across domains. Even so, the literature shows consistent improvements whenever the distillation mechanism is matched to the type of domain shift.

Method	Setting	Reported outcome
Dynamic Distillation Network	BSCD-FSL	Outperforms STARTUP by 4.4% for 1-shot and 3.6% for 5-shot classification (Islam et al., 2021)
Cross-Level Distillation + Feature Denoising	BSCD-FSL	Surpasses Dynamic-Distillation by 5.44% on 1-shot and 1.37% on 5-shot on average (Zheng et al., 2023)
TAMEPT	Amazon Reviews UDA	94.17% ±0.40 single-source and 95.09% ±0.19 multi-source, with gains of 1.03% and 1.34% over AdSPT (Feng et al., 2023)
CD-FKD	Single-domain generalization in detection	Average target mAP@0.5 of 38.3 versus 27.2 for Faster R-CNN, a gain of 11.1 points (Lee et al., 17 Mar 2026)
GCKD	ICFG $\rightarrow$ RSTPReid	Rank-1 59.95 and mAP 49.68 versus RaSa 55.00 and 46.18 (Luo et al., 25 Jan 2025)
XDED + UniStyle	PACS domain generalization	Average 86.4 versus 79.5 for ERM (Lee et al., 2022)
FUAD CDD	MVTec AD-noise-0.1, Overlap	RD 0.708/0.818/0.901 versus CDD 0.971/0.973/0.921 for I-AUC/P-AUC/PRO (Liu et al., 25 Aug 2025)
ContiStain	Sequential virtual IHC staining	Reduces FID and ConchFID by 11.1 and 60.9 compared to sequential fine-tuning (Chen et al., 4 Jul 2026)

Additional reports extend this pattern. XKD improves video action classification by 8% to 14% on UCF101, HMDB51, and Kinetics400, improves multimodal action classification by 5.5% on Kinetics-Sound, and reaches 96.5% top-1 on ESC50 (Sarkar et al., 2022). TDD improves source-only detection baselines across adverse weather, synthetic-to-real, and cross-camera benchmarks, for example reaching 63.3 AP on Sim10k $\rightarrow$ Cityscapes with ResNet-50 versus 42.8 for the source-only baseline (He et al., 2022). In scientific time series, STEP ranks first on GWOSC, LEAVES, MarmAudio, and RadSeg among the reported baselines and remains competitive on SleepEDF and WBCIC, which suggests that multi-teacher cross-domain distillation can be effective even when the target domain is highly heterogeneous (Zhang et al., 19 Mar 2026).

6. Limitations, misconceptions, and open questions

One persistent misconception is that CDD always means enforcing stronger invariance by harder matching. The surveyed literature shows the opposite in several regimes. TAMEPT explicitly seeks domain-aware target features rather than only domain-invariant ones (Feng et al., 2023). DIFD improves invariant sentiment transfer by explicitly modeling domain-dependent aspect information (Hu et al., 2019). In widely differing modalities, hard-constrained L2 alignment is reported to overfit, and soft feature margins plus classifier-level sharing are preferred (Zhao et al., 22 Jul 2025). More rigid matching is therefore not uniformly better.

Another misconception is that CDD always requires target-domain data or a separate teacher network. CD-FKD performs clean-to-corrupted distillation with no target data at train time (Lee et al., 17 Mar 2026), and XDED uses no external teacher at all, only same-label ensembles across source domains (Lee et al., 2022). Conversely, some CDD settings rely heavily on unlabeled target data and can fail when the teacher’s target predictions are poor, the unlabeled target set is scarce or imbalanced, or the augmentation and masking policy is ill-suited to the domain (Islam et al., 2021, Feng et al., 2023). STEP reports that teacher mismatch remains a real limitation when none of the available foundation models fit the scientific signal regime particularly well (Zhang et al., 19 Mar 2026).

The main technical failure modes recur across tasks. Domain construction can be brittle in contaminated anomaly detection if similar anomalies concentrate in one domain (Liu et al., 25 Aug 2025). Corruption mismatch limits clean-to-corrupted detection transfer when the chosen corruptions do not reflect deployment shifts (Lee et al., 17 Mar 2026). Batch-statistic noise is central in ProLAD, where normalization helps on dissimilar domains but can be harmful on similar domains with very small support sets (Yang et al., 2023). In continual pathology, overly large relational weights preserve old geometry at the expense of plasticity on the newest biomarker domain (Chen et al., 4 Jul 2026). XKD makes the complementary point that domain alignment is not optional in strongly heterogeneous modalities: without alignment, cross-modal KD can collapse to masked autoencoding behavior (Sarkar et al., 2022).

Open directions are already visible inside the current literature. Multi-source teacher fusion and multi-level distillation are being pushed beyond single-task settings in medical imaging and scientific time series (Ciprian-Mihai et al., 2 May 2026, Zhang et al., 19 Mar 2026). Episode-wise or domain-wise adaptive weighting, as in ProLAD, points toward CDD systems that modulate distillation strength according to measured domain similarity rather than using fixed coefficients (Yang et al., 2023). Relation-preserving formulations such as ContiStain and correlation-based formulations such as CCDistill suggest that future CDD systems may rely less on pointwise imitation and more on preserving inter-domain structure (Chen et al., 4 Jul 2026, Gao et al., 2022). Across these variants, the stable conclusion is that effective cross-domain transfer depends less on the presence of a teacher than on how the transferred knowledge is filtered, aligned, and regularized before it reaches the student.