HADUA: Hierarchical Attention and Dynamic Uniform Alignment for Robust Cross-Subject Emotion Recognition

Abstract: Robust cross-subject emotion recognition from multimodal physiological signals remains a challenging problem, primarily due to modality heterogeneity and inter-subject distribution shift. To tackle these challenges, we propose a novel adaptive learning framework named Hierarchical Attention and Dynamic Uniform Alignment (HADUA). Our approach unifies the learning of multimodal representations with domain adaptation. First, we design a hierarchical attention module that explicitly models intra-modal temporal dynamics and inter-modal semantic interactions (e.g., between electroencephalogram(EEG) and eye movement(EM)), yielding discriminative and semantically coherent fused features. Second, to overcome the noise inherent in pseudo-labels during adaptation, we introduce a confidence-aware Gaussian weighting scheme that smooths the supervision from target-domain samples by down-weighting uncertain instances. Third, a uniform alignment loss is employed to regularize the distribution of pseudo-labels across classes, thereby mitigating imbalance and stabilizing conditional distribution matching. Extensive experiments on multiple cross-subject emotion recognition benchmarks show that HADUA consistently surpasses existing state-of-the-art methods in both accuracy and robustness, validating its effectiveness in handling modality gaps, noisy pseudo-labels, and class imbalance. Taken together, these contributions offer a practical and generalizable solution for building robust cross-subject affective computing systems.

• The paper introduces HADUA, integrating hierarchical attention and dynamic uniform alignment to optimize multimodal emotion recognition.
• It employs dual-branch feature extractors with cross-modal attention to robustly fuse EEG and eye-movement data while mitigating noise.
• Experimental results on SEED benchmarks confirm improved accuracy, balanced class performance, and effective domain adaptation.

HADUA: Hierarchical Attention and Dynamic Uniform Alignment for Robust Cross-Subject Emotion Recognition

Introduction

Cross-subject emotion recognition using multimodal physiological signals, specifically electroencephalogram (EEG) and eye-movement (EM), faces persistent challenges arising from modality heterogeneity and pronounced inter-subject distribution shift. Traditional approaches typically address either multimodal fusion or domain adaptation in isolation, resulting in suboptimal generalization performance in real-world, cross-subject scenarios. The paper "HADUA: Hierarchical Attention and Dynamic Uniform Alignment for Robust Cross-Subject Emotion Recognition" (2601.21488) introduces HADUA, a comprehensive and unified adaptive learning architecture that jointly optimizes multimodal representation, pseudo-label reliability, and domain alignment.

Framework Architecture

HADUA is a three-component framework engineered to simultaneously address the nuances of multimodal feature fusion, multi-level distribution alignment, and robust sample-wise pseudo-label optimization.

Figure 1: Overview of the cross-subject multimodal emotion recognition framework, comprising attention-based multimodal fusion, multi-level distribution alignment, and confidence-driven pseudo-label optimization modules.

Hierarchical Attention-Based Fusion

The framework adopts dual-branch modality-specific feature extractors for EEG and EM. Self-attention modules are used to model intra-modality temporal and contextual dependencies for both EEG and EM signals. Critically, a uni-directional cross-modal attention mechanism (EEG $\rightarrow$ EM) exploits the higher discriminative strength of EEG by allowing its features to selectively attend to EM signals, facilitating noise-robust and semantically coherent fusion.

Multi-Level Distribution Alignment

To counteract domain shift, HADUA applies both marginal (MMD-based) and conditional (CMMD-based) distribution alignment between source and target domains. Marginal alignment reduces global feature distribution disparities, while conditional alignment leverages pseudo-label statistics to align class-conditional structure. The efficacy of conditional alignment is directly dependent on the reliability and class-balance of the pseudo-labels assigned to target samples.

Confidence-Driven Pseudo-Label Optimization

A core innovation of HADUA is the introduction of a truncated soft Gaussian weighting function to softly weight target-domain pseudo-labels by prediction confidence. This mechanism ensures that low-reliability pseudo-labels are down-weighted rather than discarded, maintaining supervisory signal while suppressing label noise.

Figure 2: Evolution of the truncated soft Gaussian weighting function. Samples above the adaptive threshold $\mu_t$ receive maximum weight; samples below are down-weighted based on the Gaussian decay.

Additionally, the Uniform Alignment (UA) mechanism regularizes batch-wise pseudo-label distributions toward uniformity, mitigating class imbalance and ensuring reliable estimation of class-conditional statistics during CMMD optimization.

Experimental Results

HADUA is evaluated on SEED and SEED-IV cross-subject emotion recognition benchmarks and demonstrates consistent improvements in accuracy, macro-F1, and AUC compared to a comprehensive suite of existing multimodal and domain adaptation methods.

Class-wise Performance and Robustness

Analysis of the model's confusion matrices highlights highly balanced performance across emotion categories for both three-class (SEED) and four-class (SEED-IV) setups.

Figure 3: Confusion matrices of HADUA on (a) SEED and (b) SEED-IV datasets, revealing balanced category-wise accuracy with errors primarily among semantically adjacent emotions.

HADUA delivers the lowest standard deviation in per-class accuracies among all tested methods, directly attributable to the UA-driven class balancing in pseudo-label generation.

Feature Representation and Alignment

Progressive t-SNE visualizations of fused feature distributions across training epochs confirm the emergence of well-separated, compact, and domain-invariant clusters, indicating successful joint optimization of discriminative and alignment objectives.

Figure 4: t-SNE visualization of feature distributions at epochs 0, 50, and 200 for SEED and SEED-IV; source and target domain samples are closely aligned within emotion clusters by epoch 200.

Neurophysiological Interpretability

Mutual information topographic maps show that the learned representations prioritize physiologically meaningful EEG channels and frequency bands. Frontal electrodes and gamma-band features dominate for the SEED dataset, while combined posterior and frontal relevance emerges for SEED-IV, demonstrating that HADUA preserves underlying neurocognitive correlates of emotion processing.

Figure 5: Feature importance (Mutual Information) on SEED. Frontal electrodes (e.g., F5, Fpz) and gamma band frequency provide the most discriminative information.

Figure 6: Feature importance (Mutual Information) on SEED-IV highlights both posterior (PO7) and frontal areas, with the fear category showing lower informativeness.

Ablation and Sensitivity Analysis

Component-wise ablation studies validate the additive benefits of hierarchical attention, distribution alignment, confidence-weighted pseudo-labeling, and UA regularization. Each addition yields measurable gains in cross-subject recognition accuracy and class balance.

Detailed hyperparameter sensitivity analysis reveals HADUA's robustness to variations in batch size, training epochs, temperature parameter $\tau$, and uniform alignment strength $\alpha$. Optimal accuracy is consistently obtained under a broad and stable range of configurations.

Figure 7: Sensitivity of HADUA to batch size, highlighting robustness and stable improvements with larger batches.

Figure 8: Sensitivity of HADUA to epoch; performance plateaus and remains robust beyond a sufficient number of epochs.

Figure 9: Sensitivity of HADUA with respect to temperature parameter $\tau$ and alignment strength $\alpha$ confirms algorithmic stability over a wide parameter space.

Theoretical and Practical Implications

HADUA's integrative strategy—combining modality-aware feature fusion with nuanced, confidence- and class-regularized domain adaptation—effectively breaks the feedback loop wherein modality heterogeneity propagates prediction noise, corrupts pseudo-labels, and degenerates domain alignment. This mechanism achieves high recognition accuracy, low class imbalance, and interpretable biomarker importance, establishing a new baseline for robust affective computing in variable and heterogenous subject populations.

Theoretically, the approach demonstrates that joint optimization of representation learning and pseudo-label reliability, when augmented with class-aware regularization, is paramount in cross-domain neural signal processing. Practically, HADUA's modularity and parameter-robustness make it suitable for real-world deployment in adaptive human-computer interaction, brain-computer interface settings, and affective monitoring systems.

Future directions may include extension to open-set domain adaptation, incorporation of richer multimodal data, and real-time implementation in interactive scenarios. The confidence-driven and class-regularized pseudo-labeling principles established herein are broadly applicable to other cross-domain and multimodal learning contexts, including health informatics, user state detection, and resilient BCI pipelines.

Conclusion

HADUA presents a robust, end-to-end solution to cross-subject multimodal emotion recognition by synergistically addressing the intertwined challenges of modality fusion and sample/class-level adaptation. Its results suggest that advanced attention modeling and dynamic, class-conscious pseudo-label refinement are indispensable for generalizable affective computing. The methodology and findings outlined in this work are likely to have significant influence on future designs of adaptive and interpretable neural signal analysis systems.