Dual-Aligned Prototype Distillation (DAPD)

• The paper introduces DAPD, which enforces dual prototype alignment to robustly handle modality gaps and label noise in cross-modal and continual learning scenarios.
• It leverages local-to-global and local-to-old-local alignment objectives to maintain semantic consistency and reduce the effects of catastrophic forgetting.
• Empirical evaluations show improved performance in EEG–vision emotion recognition and incremental medical image segmentation relative to standard baselines.

Dual-Aligned Prototype Distillation (DAPD) is a supervised representation alignment framework designed for cross-modal knowledge distillation and continual learning scenarios requiring robust class- or semantic-level alignment under challenging conditions such as modality gaps, label inconsistencies, and data stream non-stationarity. The method enforces feature-space compactness and semantic consistency by introducing dual alignment objectives that leverage prototype representations—typically meaning class centroids or semantic anchors—across distinct models, learning steps, or data modalities. DAPD has been applied in both cross-modal EEG–vision distillation for emotion recognition (Jang et al., 17 Jul 2025) and class-incremental medical image segmentation (Zhu et al., 11 Nov 2025), motivating a general formulation encompassing cross-modal and continual settings.

1. Foundational Motivation and Scope

DAPD is a response to two recurrent challenges—modality gap and soft-label/feature-space misalignment in knowledge distillation, and catastrophic forgetting in continual and class-incremental learning:

• Modality gap in cross-modal distillation (e.g., EEG×vision) derives from differences in feature statistics and semantic structures across modalities. Standard distillation losses cannot sufficiently align these heterogeneous representations.
• Label or prototype misalignment is aggravated by label noise or the continual introduction of new classes, compromising feature robustness and retention of old knowledge.
• Prototype replay or anchoring methods alleviate forgetting by maintaining class-specific feature centroids (prototypes), but are limited by use of only global or static statistics, ignoring context-specific or batch-local variations.

DAPD addresses these issues via dual alignment, i.e., enforcing that the current network’s local prototypes both remain close to (a) long-term global class centroids, and (b) context-specific local prototypes from a reference model (teacher, previous step, or alternate modality). This dual anchoring restricts semantic drift and enhances feature discriminability.

2. Mathematical Framework and Representation

DAPD’s central operations are built on the construction and alignment of prototype sets. In the cross-modal scenario (Jang et al., 17 Jul 2025), DAPD jointly trains a teacher ($\mathcal V$) and student ($\mathcal E$), where each model produces feature embeddings and semantically interpretable prototypes:

• Given paired minibatches $\{(x_s^i, x_t^i, y^i)\}_{i=1}^N$, encode

$$(f_s^i, \mathbf e_s^i) = \mathcal{E}(x_s^i), \quad (f_t^i, \mathbf e_t^i) = \mathcal{V}(x_t^i).$$

• Introduce a class-prototype basis $\Phi = \{\Phi^j\}_{j=1}^C$, with $\Phi^j \in \mathbb{R}^d$.

In incremental segmentation (Zhu et al., 11 Nov 2025), at step $t$ with old (frozen) model $f_{\theta^{t-1}}$ and current model $f_{\theta^t}$:

• For each class $c$, define local batch mean prototype

$$\hat{p}_c = \frac{1}{|F_c|} \sum_{f_i \in F_c} f_i \in \mathbb{R}^K,$$

and maintain global running-average prototype $p_c$.

Local prototypes capture current batch properties. Global prototypes summarize all past observed data for each class. For background (class $0$), DAPD computes an unbiased prototype via fusion across current classes.

3. Dual Alignment Objectives

DAPD’s distinguishing feature is the deployment of two complementary prototype-based losses:

• Local-to-global alignment: Ensures the current model’s local (or unbiased) prototype $\hat{q}_c$ remains close to the global prototype $p_c$,

$$L_{\mathrm{align\_global}} = \frac{1}{|C^{1:t-1}|+1} \sum_{c \in C^{1:t-1}\cup\{0\}} \| \hat{q}_c - p_c \|_2^2.$$

• Local-to-old-local alignment: Aligns the current unbiased prototype to the old model’s local prototype (on the same batch),

$$L_{\mathrm{align\_local}} = \frac{1}{|C^{1:t-1}|+1} \sum_{c \in C^{1:t-1}\cup\{0\}} \| \hat{q}_c - \hat{p}_c^{t-1} \|_2^2.$$

• The DAPD loss is a weighted sum:

$$L_{\mathrm{dapd}} = \lambda_{\mathrm{ll}} L_{\mathrm{align\_local}} + \lambda_{\mathrm{lg}} L_{\mathrm{align\_global}}.$$

In the cross-modal application, a prototype-based similarity module defines a contrastive InfoNCE-style loss between paired embeddings,

$$\mathcal{L}_{\mathrm{sim}} = -\frac1N \sum_{i=1}^N \log \frac{\exp(Q_{ii})}{\sum_{k=1}^N \exp(Q_{ik})},$$

where $Q_{ik}$ is the temperature-scaled cosine similarity. Semantic uncertainty is quantified via Dirichlet evidence parameters computed by similarity to all class prototypes, yielding an aleatoric uncertainty estimate aligned to non-match prototype similarity.

4. Integration with Supervision and Distillation

DAPD is embedded into broader training objectives, always combined with direct task supervision and auxiliary distillation components. The typical full objective comprises:

Component	Symbol	Description
Prototype alignment	$L_{\mathrm{dapd}}$	Dual alignment as above
Cross-modal distillation	$\mathcal{L}_{\mathrm{kd}}$	KL divergence of teacher vs. student predictions
Uncertainty loss	$\mathcal{L}_{\mathrm{unc}}$	Aligns predicted uncertainty with background sim.
Supervised task loss	$L_{\mathrm{ce}}, \mathcal{L}_{\mathrm{task}}$	Cross-entropy (classification), CCC loss (reg.)
Distillation calibration	$L_{\mathrm{orcd}}, L_{\mathrm{crcd}}$	Prototype-based reweighting of distillation

The total loss is a weighted sum, with task-specific hyperparameters adjusting the relative strength of alignment, supervision, and distillation. For cross-modal DAPD,

$$\mathcal{L} = \lambda_1 \mathcal{L}_{\mathrm{sim}} + \lambda_2 \mathcal{L}_{\mathrm{unc}} + \lambda_3 \mathcal{L}_{\mathrm{kd}} + \lambda_4 \mathcal{L}_{\mathrm{task}}.$$

For continual segmentation,

$$L_{\mathrm{total}} = L_{\mathrm{ce}} + \lambda_{\mathrm{orcd}} L_{\mathrm{orcd}} + \lambda_{\mathrm{crcd}} L_{\mathrm{crcd}} + L_{\mathrm{dapd}}.$$

This scheme ensures that prototype alignment is tightly coupled with supervised and uncertainty-aware distillation requirements.

5. Training Workflow and Implementation Characteristics

DAPD’s implementation is characterized by efficiency, modularity, and minimal overhead:

1. Model preparation:
   • In continual learning: freeze prior model, load stored global prototypes.
   • In cross-modal KD: pre-train or fix teacher; student and prototype set are updated.
2. Mini-batch processing:
   • Extract features and embeddings for both teacher/old and student/current models.
   • For segmentation, generate region masks and pseudo-labels via high-confidence predictions of old model.
   • Compute local prototypes for each class or region, merge to form unbiased prototypes.
3. Loss computation:
   • Calculate local–global and local–local prototype alignment losses.
   • Compute all task and distillation losses, including uncertainty and InfoNCE/KL as appropriate.
   • Aggregate to total loss and backpropagate with respect to student/current model and prototype parameters.
4. Prototype updating:
   • Maintain running averages for global prototypes with sample-count weighting (O(K⋅#classes) memory).
   • Local prototypes reuse region-wise means already computed for calibration.
   • For background, update prototype at each step to track distributional drift.

Computational overhead is modest: prototype and alignment loss calculations are low-rank reductions relative to overall model throughput (≈10% training time increase in segmentation). Hyperparameters $\lambda_{\mathrm{ll}}=0.5$, $\lambda_{\mathrm{lg}}=0.1$ are effective; batch-pairing and temperature/uncertainty thresholds are also tuned empirically.

6. Empirical Results and Evaluations

Experimental validation spans both BCI emotion recognition (Jang et al., 17 Jul 2025) and class-incremental medical segmentation (Zhu et al., 11 Nov 2025):

• Brain-Computer Interfaces (MAHNOB-HCI dataset): DAPD achieves 57.1% accuracy and 60.0% F1 on arousal DEC, outperforming best unimodal (MASA-TCN: 46.4/48.7%) and multimodal (CAFNet: 59.4/59.6%) baselines. For regression (valence CER), DAPD yields RMSE 0.043, PCC 0.449, CCC 0.359—consistently better than next-best alternatives by 0.03–0.1. Visualizations confirm tight emotion clusters, supporting improved semantic separation.
• Medical Image Segmentation (BTCV, WORD): Across various incremental protocols, DAPD+PGCD improves old-class DSC by 1.3% (BTCV 4-4) and sustains >90% mean DSC across 5 incremental steps, while comparisons degrade by 5–10%. Ablation reveals each of the dual-alignment terms is crucial: both λ_ll and λ_lg confer 0.6–1.0% individual gain, with joint use yielding a cumulative ≈1.5% improvement.

Ablation studies demonstrate that no single alignment term suffices; omission of either degrades robustness and knowledge retention, confirming the necessity of dual anchoring. A plausible implication is that dual alignment provides complementary regularization against both long-term drift and local, context-dependent prototype distortion.

7. Significance and Broader Implications

DAPD establishes a versatile, low-overhead solution for robust prototype-based regularization in both cross-modal and continual learning environments:

• In cross-modal distillation, it bridges modality gaps while quantifying semantic uncertainty, enabling effective label-projection even under labeling noise and feature-set heterogeneity.
• In lifelong/incremental tasks, dual anchoring reduces catastrophic forgetting without excessive memory or computation, sidestepping the limitations of global-prototype-only replay.
• The generic dual-alignment paradigm is adaptable to settings wherever maintenance of semantic feature consistency is critical under data, label, or model evolution, such as federated learning, unsupervised domain adaptation, and open-world recognition.

A remaining direction is formal characterization of the bias–variance properties of dual-aligned prototype anchoring and investigation of optimal prototype update strategies under resource and drift constraints.