Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm

Abstract: Cross-modal knowledge distillation (CMKD) studies how a (large) teacher model trained on one type of data (e.g., images) can guide a (smaller) student model building on another type of data (e.g., text/audio). Existing CMKD methods often require paired multi-modal data with aligned semantics, but obtaining such paired data are often costly and impractical. To mitigate this limitation, we develop a new CMKD framework for the more challenging setting where paired data are unavailable. In particular, we establish a cross-modal distributional relationship between teacher and student models, which reveals two fundamental quantities governing effective distillation: feature alignment and label alignment. These quantities characterize semantic discrepancy between modalities at the levels of representation and prediction distributions, respectively. Motivated by this insight, we propose a principled framework, with theoretical guarantees, that enables effective cross-modal knowledge distillation by aligning distributions rather than individual samples. Extensive experiments across a wide range of multimodal benchmarks show that our framework is highly effective in both unpaired and paired data settings, improving significantly over prior work.

• The paper introduces UCMKD to enable cross-modal distillation without paired data using a novel bi-level optimization framework.
• It derives theoretical generalization bounds that combine teacher error, feature alignment via Wasserstein distance, and label alignment.
• Empirical results on multiple multimodal benchmarks confirm significant accuracy gains and increased robustness over traditional methods.

Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm

Introduction

The paper "Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm" (2606.10504) presents a principled framework, UCMKD, for knowledge distillation (KD) in cross-modal settings where paired multi-modal data is unavailable. This addresses a key limitation of most CMKD approaches, which rely on explicit sample-level correspondence between modalities to enable transfer from a teacher network trained on one modality (e.g., vision) to a student network targeting another (e.g., audio or text). The proposed method formulates a theoretical foundation for unpaired cross-modal knowledge transfer, identifying and operationalizing two crucial alignment criteria—feature alignment (FA) and label alignment (LA)—by minimizing distributional discrepancies in shared latent and prediction spaces.

Theoretical Contributions

A major contribution of the work is the derivation of generalization bounds for the student model under both infinite- and finite-sample regimes. Specifically, the paper shows that the expected student error in the unpaired CMKD setting is effectively upper bounded by the sum of three terms: the teacher error (fixed overhead), a distribution-level feature alignment term (FA), and a distribution-level label alignment term (LA). Formally, for student encoder $\phi$ and teacher encoder $\theta$, and prediction heads $p_S$, $p_T$,

$$\mathrm{err}_S(\phi) \leq \mathrm{err}_T(\theta) + \mathrm{FA}(\phi, \theta) + \mathrm{LA}(p_S, p_T)$$

where $\mathrm{FA}$ is given by the Wasserstein-1 distance between the induced latent distributions, and $\mathrm{LA}$ is a log-likelihood ratio reflecting the prediction distribution mismatch.

Crucially, the bound reveals that minimizing representation alignment alone (i.e., distribution matching in latent space) is insufficient for effective distillation in cross-modal, unpaired data regimes. Excessive emphasis on feature space matching can actually amplify semantic mismatches, leading to generalization degradation if label-level alignment is ignored. The tightness of the bounds is supported by empirical analysis, with the gap consistently narrowing as dataset size increases.

Algorithmic Framework (UCMKD)

UCMKD realizes the theoretical analysis by formulating distillation as a bi-level optimization problem with distributional matching objectives. The outer optimization minimizes the student loss, while the inner loop is split into two stages: aligning the feature distributions (FA) of teacher and student modality encoders in a shared latent space via entropic-regularized optimal transport, and then aligning their conditional label distributions (LA) using an empirical label transport kernel. This kernel adaptively weighs the distillation signal by the semantic agreement between teacher and student predictions, mitigating the risk of negative transfer from unreliable teacher outputs.

Concretely, UCMKD alternates between minimizing a Sinkhorn-regularized feature alignment loss and a selective, reweighted label alignment loss on mini-batches of unpaired student and teacher samples. The framework does not require explicit or weakly paired multimodal training data, a marked departure from previous approaches. The algorithm's scalability is empirically validated on both ResNet-based and ViT-based architectures.

Empirical Results

Extensive experimentation across four multimodal benchmarks (AVE, CREMA-D, RAVDESS, and VGGSound) demonstrates that UCMKD consistently surpasses both unpaired and paired baselines. In the challenging unpaired setting, average student accuracy improvements reach up to 14.3% compared to cross-entropy and 7.5% over strong feature distillation baselines. Notably, UCMKD outperforms Vanilla KD (which uses paired data) in the majority of transfer directions.

In the paired setting, UCMKD remains competitive with recent state-of-the-art distillation approaches including C2KD, DKD, and ReviewKD, achieving the highest performance on most reporting tasks. Ablation studies confirm the necessity and complementarity of both feature and label alignment losses. Additional robustness and scaling analyses on hard unpaired benchmarks (e.g., with domain shift, marginal mismatch, and label imbalance) further highlight the resilience and generalizability of the method, which degrades more gracefully than alternatives as the unpaired setting becomes more realistic.

Implications and Future Directions

Practically, the framework provides an effective and modular solution for transferring knowledge in multi-modal regimes without incurring the prohibitive cost of paired data collection. This enables deployment in real-world applications where modalities are only loosely correlated in time or may be acquired independently (e.g., asynchronous sensor fusion, vision-to-text transfer in large archives, or privacy-constrained multimodal training).

Theoretically, the bound establishes that tight control over both distributional feature and semantic label mismatches is necessary for generalization in cross-modal distillation. The meta-optimization approach and the use of Wasserstein distances facilitate extension to more expressive, learned transport geometries and highlight promising directions for integrating adaptive metric learning.

Possible extensions include generalizing the framework to generative multimodal modeling, leveraging learned optimal transport cost metrics, and increasing scalability via implicit differentiation or reduced unrolling in the bi-level optimization. The results also open avenues for foundation model adaptation across modalities and unsupervised multi-modal data augmentation in the absence of aligned supervision.

Conclusion

This work sets a rigorous foundation for cross-modal knowledge distillation in the unpaired data regime, offering both theoretical justification and a practical learning algorithm that advances empirical state-of-the-art across diverse multimodal tasks (2606.10504). By uncovering and operationalizing key distributional alignment factors, UCMKD enables robust, scalable knowledge transfer in settings where sample-level correspondence between modalities is unattainable, thus broadening the applicability of KD in heterogeneous, real-world environments.