Multi-Dataset Cross-Domain Knowledge Distillation for Unified Medical Image Segmentation, Classification, and Detection

Abstract: We propose a unified cross-domain transfer learning framework that leverages knowledge from multiple heterogeneous medical imaging datasets to improve performance across segmentation, classification, and object detection tasks. Our approach employs a teacher-student paradigm in which a joint teacher model aggregates domain-invariant representations learned from diverse source datasets, while a task-specific student model is trained via multi-level knowledge distillation. Originally developed for medical image segmentation, the framework is extended to support image-level classification and object-level detection, enabling a general multi-task formulation for medical image analysis. We evaluate our method on a broad suite of datasets, including six segmentation benchmarks, BrainMetShare, ISLES, BraTS (MRI) and Lung MSD, LiTS, KiTS (CT), as well as multiple classification datasets for pulmonary disease and dementia, and detection datasets with native bounding-box annotations. Across all tasks and modalities, the proposed approach yields consistent improvements over strong dataset-specific and multi-head baselines, demonstrating enhanced robustness to distributional shifts and superior generalization. These findings highlight the potential of multi-dataset knowledge distillation as a scalable and task-agnostic approach for enhancing segmentation, classification, and object detection performance across heterogeneous medical imaging domains.

• The paper introduces a unified framework that employs domain-adversarial teacher training and multi-level fusion to improve segmentation, classification, and detection across diverse medical imaging datasets.
• The method leverages cross-attention modules and curriculum-driven distillation to effectively merge source-target features, achieving higher IoU, DSC, and mAP metrics.
• Evaluation on multiple MRI and CT datasets demonstrates that the student model consistently outperforms dataset-specific baselines, showcasing enhanced robustness and efficiency.

Multi-Dataset Cross-Domain Knowledge Distillation for Unified Medical Image Segmentation, Classification, and Detection

Framework Architecture and Methodology

The paper introduces a unified cross-domain teacher-student knowledge distillation framework addressing segmentation, classification, and object detection tasks across heterogeneous medical imaging datasets. The pipeline comprises three main stages: domain-adversarial teacher training, multi-level feature fusion via a joint teacher, and curriculum-driven multi-level distillation to a compact student model.

In Stage 1, both source and target teacher models are trained with domain-adversarial objectives to enforce domain-invariant features, leveraging target supervision where available to improve feature alignment and boundary accuracy (Figure 1).

Figure 1: Overview of the pipeline highlighting teacher training, multi-level fusion, and knowledge distillation.

Source teachers employ an encoder, decoder, and domain discriminator architecture, and optimization uses combined segmentation and binary cross-entropy domain losses. The adversarial schedule ensures the encoder learns indistinguishable source-target features. The same domain-adaptation mechanism is applied in segmentation, classification, and detection tasks.

Figure 2: Source teacher model training via domain-adversarial feature alignment.

Stage 2 constructs a joint teacher via cross-attention fusion of multi-level features extracted from source and target teachers at corresponding abstraction levels. Frozen teacher encoders and bottlenecks are harmonized using cross-attention modules, and a task-specific head (segmentation decoder, classifier, or detection head) is trained exclusively on the target domain. This design aggregates complementary domain representations without loss of dataset-specific information.

Stage 3 distills multi-source fused representations from the joint teacher into a task-specific student using curriculum-driven objectives. Distillation combines supervised task losses (Dice+BCE for segmentation, cross-entropy for classification, detection loss for object detection) with contrastive, feature alignment, and cosine similarity terms, guided by a curriculum factor to balance early task supervision against late-stage distillation. The framework supports heterogeneous teacher-student architectures and is directly extensible across tasks and modalities.

Segmentation Evaluation

The framework is evaluated on six MRI and CT datasets spanning brain and organ tumor segmentation. Quantitative results show that student models consistently outperform both dataset-specific and multi-head baselines across all datasets and metrics, with strongest gains in CT-based tasks (see Table below for summary).

Model	MRI IoU	CT IoU	MRI DSC	CT DSC	MRI HD95	CT HD95
Multi-head	76.9	79.9	80.5	82.6	13.5	12.5
Dataset-specific	83.9	86.8	86.9	89.4	10.4	9.8
Joint teacher	84.0	87.1	87.0	89.6	10.2	9.4
Student	85.3	89.3	88.1	91.9	9.2	6.1

Per-dataset analysis reveals particularly pronounced IoU and HD95 gains for BrainMetShare (MRI) and LiTS (CT), indicating improved small-lesion recovery and sharper boundaries, respectively. Ablation studies confirm benefits from increased source dataset diversity and multi-level distillation (encoder+bottleneck features). Supervised target adaptation primarily improves boundary accuracy but is not critical for overlap-based metrics.

Figure 3: Qualitative segmentation results across MRI and CT datasets comparing baseline and student outputs.

Figure 4: Attention map visualization illustrating improved localization and discriminative power in distilled student models.

Classification and Detection Results

On pulmonary (COVID-19, pneumonia) and cerebral (Alzheimer’s, dementia) classification, the framework achieves substantial accuracy improvements. For example, student models distilled with late-stage feature alignment outperform dataset-specific baselines by +3–5% across all groups, with further gains observed with joint teacher models.

Model	COVIDx-CXR	RT-PCR Covid19	COVID-QU-Ex	OASIS MRI	ADNI
Multi-head	73.4	90.1	89.7	70.1	84.7
Dataset-specific	78.5	93.8	91.5	71.9	88.7
Joint teacher	83.6	97.4	95.8	76.9	90.3
Student L3	82.0	94.8	93.4	77.8	90.7

Late-stage distillation universally provides superior feature alignment and accuracy over early-stage strategies. Cross-architecture teacher ensembles for cerebral tasks (MedViT/EfficientNet) validate effective transfer even in heterogeneous scenarios.

Object detection evaluation on lung CT datasets (DeepLesion, LungCT, LungPet) with two detector paradigms (Faster R-CNN, RF-DETR) shows joint teacher and distilled student models achieve significant mAP improvements. For example, in DeepLesion, joint teacher improves [email protected]–0.95 by +8.7 points over dataset-specific baseline, with student retaining ~4 points of this gain while offering efficiency.

Figure 5: Best accuracy achieved for each method across the pulmonary disease datasets.

Figure 6: Best accuracy achieved for each method across the cerebral degenerative disease datasets.

Figure 7: Qualitative object detection results on lung CT datasets. Student detectors exhibit improved localization and reduced false positives, particularly for small or low-contrast lesions.

Representational Analysis and Robustness

t-SNE feature visualizations across segmentation (pixel bottlenecks), classification (image-level features), and detection (region embeddings) show distilled student models produce more compact and better-separated clusters than dataset-specific baselines, indicating improved discriminative representations and cross-domain robustness.

Figure 8: t-SNE visualizations across tasks. Distilled students form compact, discriminative clusters indicating enhanced representation structure.

Ablation studies further confirm model robustness to the absence of supervised target adaptation, and performance scaling with the number and diversity of source datasets. Multi-level distillation (encoder+bottleneck) consistently outperforms single-level configurations.

Practical and Theoretical Implications

The results demonstrate that structured multi-dataset knowledge distillation via joint teacher fusion and curriculum-driven multi-level feature alignment provides consistent improvements across all major medical imaging tasks—segmentation, classification, and object detection. The architecture-agnostic, task-agnostic framework is inherently robust to domain shifts, annotation scarcity, and institutional bias.

Practically, the method enables deployment-efficient student models for real-time inference with direct applicability in heterogeneous clinical contexts. Theoretically, the approach substantiates the advantage of cross-domain representation aggregation, curriculum distillation, and multi-level supervisory signals in medical imaging pipelines.

Future Directions

Potential extensions include federated and privacy-preserving training, self-supervised pretraining within the joint teacher, adaptive fusion and uncertainty-aware distillation mechanisms, and unification with multi-label diagnosis and survival prediction scenarios. Scalability to other modalities and real-world deployment with limited supervision are immediate avenues for research.

Conclusion

This paper presents a rigorous, universal approach for multi-dataset cross-domain knowledge distillation in medical image analysis, demonstrating robust, transferable gains in segmentation, classification, and detection. The framework sets a strong methodological precedent for scalable, task-agnostic transfer learning in heterogeneous clinical applications, with implications for both foundational research and translational deployment in AI-driven medicine (2605.01563).