CAB Distillation Bridge

• CAB Distillation Bridge is a mechanism enabling cross-architecture knowledge transfer by explicitly aligning intermediate representations between heterogeneous neural models.
• It employs techniques like multi-head cross-attention, token projection, and logits-space projection to overcome feature misalignment and weak gradients.
• Empirical results indicate improvements in metrics like c-index, ImageNet accuracy, and CIFAR-100 performance, demonstrating its effectiveness in low-data and multimodal scenarios.

A Cross-Architecture Bridge (CAB) for distillation—termed here CAB Distillation Bridge (Editor's term)—is a mechanism or module designed to facilitate effective knowledge transfer between heterogeneous neural architectures during training. In the context of knowledge distillation, CAB acts as an explicit translation or alignment module, mediating the transfer of complex inductive biases, intermediate representations, and output structures between architectures with fundamentally different parametrizations, operational modalities, or feature spaces. CAB frameworks have been adapted for vision, language, and multimodal tasks, particularly where direct feature matching or output-level distillation is inadequate due to mismatched latent representations or lack of cross-modal supervision (Wang et al., 2024, Wang et al., 22 Oct 2025, Hao et al., 2023).

1. Rationale for Cross-Architecture Distillation and the Need for CAB

Traditional knowledge distillation methods train a student model to mimic the output (typically logits) of a teacher model, usually within the same architectural family (e.g., Transformer-to-Transformer). When applied to cross-family scenarios—such as Transformer-to-State Space Model (SSM), CNN-to-MLP, or multimodal uni-modal transfer—output-level distillation alone does not address the following challenges:

• Lack of inductive bias transfer: Architecturally unique capacities (e.g., self-attention, parameterized recurrence) are not conveyed.
• Feature misalignment: Features from CNNs, ViTs, MLPs, and SSMs inhabit distinct latent spaces, as demonstrated via Centered Kernel Alignment (CKA) analyses (Hao et al., 2023).
• Weak or vanishing gradients: Deep students may fail to receive strong intermediate supervision if learning signals are only backpropagated from the final task loss.
• Absence of cross-modal semantics: For vision–genomics tasks, critical cross-modal associations between observable phenotypes and latent genotypes are not instantiated in the student model.

A CAB Distillation Bridge directly confronts these issues by introducing explicit intermediate matching, attention-alignment, or cross-modal associative objectives during training.

2. CAB Mechanisms in Multimodal and Heterogeneous Architectures

CAB modules have been instantiated in several modalities, each adapting the bridging mechanism to architectural idiosyncrasies:

a. Histo-Genomic Cross-Modal CAB (Wang et al., 2024)

In G-HANet, the CAB receives patch-level histopathology features

$$F^p = \{ f_j = E(x_j^p) \in \mathbb{R}^d \}_{j=1}^{N_p},$$

and a set of learnable genomic-function tokens

$T = \{ t_i \in \mathbb{R}^{d'} \}_{i=1}^{N_g}.$

It leverages two rounds of multi-head cross-attention (MHCA+FFN modules, parameter sharing between rounds) to produce functional features $f_{F2}$, governed by association matrices $M^{am}$. These distilled features then serve as inputs to self-normalizing networks that reconstruct gene-expression profiles. The reconstruction losses jointly supervise the CAB and upstream WSI encoder, compelling image-based features to internalize regions that are predictive of genomic status.

b. Attention Bridge for Transformer–Mamba Distillation (Wang et al., 22 Oct 2025)

For Transformer-to-SSM transfer, CAB bridges token-level projections between the teacher’s Query/Key representations and the Mamba student’s $B_t$/$C_t$ projections. Two-layer MLPs ($\phi_B$, $\phi_C$) map the student’s $d_s$-dimensional state projections to the teacher’s $d_t$-dimensional attention space: $$\phi_B: \mathbb{R}^{d_s} \to \mathbb{R}^{d_t},\quad \phi_C: \mathbb{R}^{d_s} \to \mathbb{R}^{d_t}.$$ Layer-wise alignment is achieved using a proportional mapping function $g(l) = \lfloor \frac{l}{L} T \rfloor$ to associate each student layer with the corresponding teacher layer. Direct alignment losses on the projected token representations provide the student with explicit attention-based supervision even in the absence of an attention mechanism in the student architecture.

c. Logits Space Projection for OFA-KD (Hao et al., 2023)

In OFA-KD, CAB materializes as exit-branch classifiers and feature projectors inserted at multiple depths in the student, projecting intermediate features into the shared logits space $\mathbb{R}^C$ via

$$\phi_i(\mathbf{F}^S_i) = \mathrm{softmax}(h_i(g_i(\mathbf{F}^S_i))).$$

This discards architecture- and modality-specific structural detail, focusing the distillation loss entirely on class-probability alignment and mitigating failures of naive feature-matching suggested by CKA analyses.

3. Mathematical Formulation and Training Objectives

The loss functions governing CAB-based distillation unify intermediate-level supervision with classic output-based losses. Prototypical objective formulations include:

• Cross-modal supervision and reconstruction: For multimodal CABs,

$$\mathcal{L}_\mathrm{CAB} = \lambda_\mathrm{rec}\,(\mathcal{L}_\mathrm{MSE} + \mathcal{L}_\mathrm{SCE}) + \lambda_\mathrm{cls} \mathcal{L}_\mathrm{cls} + \lambda_\mathrm{att} \mathcal{L}_\mathrm{att},$$

where $\mathcal{L}_\mathrm{MSE}$ and $\mathcal{L}_\mathrm{SCE}$ encourage morphologically encoded features to reconstruct genomic signals (Wang et al., 2024).

• Attention bridge supervision: For cross-architecture attention alignment,

$$\mathcal{L}_\mathrm{attn} = \frac{1}{L} \sum_{l=1}^{L} \| \phi_B(B^{(l)}) - K^{(g(l))} \|^2_2 + \| \phi_C(C^{(l)}) - Q^{(g(l))} \|^2_2.$$

This is combined with a KL divergence loss on the model outputs: $$\mathcal{L}_\mathrm{total} = \mathcal{L}_\mathrm{attn} + \lambda \mathcal{L}_\mathrm{KL},$$ with $\lambda$ tuned by validation (Wang et al., 22 Oct 2025).

• Logit-space adaptive loss: In OFA-KD,

$$\mathcal{L}_\mathrm{OFA} = - (1 + p^t_{\hat{c}})^\gamma \log p^s_{\hat{c}} - \sum_{c \neq \hat{c}} p^t_c \log p^s_c,$$

with $\gamma$ controlling the adaptivity of target enhancement (Hao et al., 2023). The total loss sums over all exit branches.

4. Empirical Results and Comparative Performance

CAB mechanisms have consistently demonstrated substantial performance gains across heterogeneous settings:

• G-HANet with CAB achieves c-index improvements of ~1–2% over single-stage attention in histopathological cancer prognosis and shows Spearman gene rank correlations of 0.2–0.3 between real and reconstructed expressions, indicating successful histo-genomic knowledge internalization (Wang et al., 2024).
• CAB for Transformer-to-Mamba distillation yields top-1 ImageNet accuracy improvements of up to +7.2% over previous state-of-the-art cross-architecture distillation methods under 10% data settings and closes much of the perplexity gap to Transformer teachers in language modeling (Wang et al., 22 Oct 2025). CAB is also 2–4× faster and more memory-efficient than full-matrix alignment.
• OFA-KD’s Bridge delivers up to +8.0% accuracy increase for cross-architecture pairings on CIFAR-100 and 0.2–0.7% on ImageNet-1K, confirming the superiority of projection-based cross-architecture bridges in heterogeneous settings (Hao et al., 2023).

Table: Summary of CAB Distillation Bridge Mechanisms

Paper (arXiv ID)	Bridge Mechanism	Loss Type / Supervision
(Wang et al., 2024)	Two-stage MHCA (CAB)	MSE + SCE on genomics (cross-modal)
(Wang et al., 22 Oct 2025)	Token proj. MLP bridge	Token attn. align + output KL
(Hao et al., 2023)	Logit projection branch	Logit-space CE/OFA adaptive loss

5. Implementation Considerations and Hyperparameters

CAB implementation in heterogeneous distillation demands careful architectural and training choices:

• Projection MLPs for attention and feature bridging commonly use two-layer SiLU activations with hidden sizes proportional to the feature dimension (e.g., $h \approx 4d_s$).
• Layer alignment strategies such as the proportional mapping $g(l)$ are critical when the teacher and student differ in depth.
• Loss balancing hyperparameters (e.g., $\alpha$ in (Wang et al., 2024), $\lambda$ in (Wang et al., 22 Oct 2025)) are optimized via grid search on validation folds.
• Regime scheduling: Early-stopping attention supervision (e.g., first 35–50 epochs) can mitigate over-constrained students in full-data regimes (Wang et al., 22 Oct 2025).
• Branch insertion (e.g., 4 exit branches per student for OFA-KD) enables multi-stage supervision (Hao et al., 2023).

Practical experiments confirm CAB’s robustness under aggressive low-data regimes and architectural variation but highlight the risk of over-regularization if auxiliary supervision is sustained throughout full-length training.

6. Interpretation, Limitations, and Outlook

CAB distillation bridges—whether realized as cross-modal attention, MLP-based token alignment, or logits-space projection—enable robust, efficient, and architecture-agnostic transfer of high-level inductive biases and fine-grained information across modality and model divides. They provide actionable gradients and interpretable mappings in settings where naive feature matching or output-only KD fail.

Nevertheless, CAB performance can degrade under prolonged attention supervision in full-data regimes, an effect remediable by early stopping (Wang et al., 22 Oct 2025). Current bridges focus on Transformer, SSM, and DNN families, but extensions to other SSM variants (e.g., S4, RWKV), hybrid models, or additional modalities remain open directions. Adaptive weighting of attention- versus output-level losses, or exploration of richer alignment strategies between latent spaces, constitutes an active frontier.

A plausible implication is that future CAB variants, possibly leveraging dynamic alignment rules or learned bridge functions, could further generalize distillation across emerging neural architectures and multimodal domains.