HADUA: Hierarchical Attention & Domain Adaptation

• The paper introduces HADUA, a framework that employs hierarchical attention to fuse EEG and eye movement signals for robust cross-subject emotion recognition.
• It integrates a confidence-aware Gaussian weighting scheme to mitigate pseudo-label noise and uses dynamic uniform alignment to achieve balanced class adaptation.
• Experimental benchmarks on datasets like SEED and SEED-IV demonstrate significant improvements in classification accuracy and class balance under domain shifts.

HADUA (Hierarchical Attention and Dynamic Uniform Alignment) is an adaptive learning framework engineered for robust cross-subject emotion recognition from multimodal physiological signals. Addressing the challenges of modality heterogeneity and inter-subject distribution shift, HADUA unifies multimodal representation learning and domain adaptation by employing a hierarchical attention module, a confidence-aware pseudo-label weighting scheme, and dynamic uniform alignment to achieve discriminative, balanced, and robust feature alignment for affective computing applications (Tang et al., 29 Jan 2026).

1. Problem Formulation and Motivation

Cross-subject emotion recognition confronts two principal sources of complexity: modality heterogeneity and inter-subject distribution shift. The former arises from the disparate temporal scales, statistical distributions, and noise characteristics in signals such as electroencephalogram (EEG) and eye movement (EM), impeding effective multimodal fusion. The latter refers to the mismatch $p_s(x,y) \neq p_t(x,y)$ between labeled source and unlabeled target distributions due to individual physiological differences.

Formally, the framework is constructed around two domains:

• Source domain $𝒟_s = \left\{ (x^{l}_{s,i}, y^{l}_{s,i}) \right\}$, where $x^{l}_{s,i}$ comprises both EEG and eye-movement features and $y^{l}_{s,i}$ is the ground-truth emotion label for the $i$-th sample.
• Target domain $𝒟_t = \left\{ x^{u}_{t,j} \right\}$, consisting of unlabeled samples requiring label estimation.

The objective is to jointly learn a feature extractor $f(\cdot)$ and classifier $g(\cdot)$ such that the composite $g(f(x))$ generalizes for emotion prediction on previously unseen subjects by training on $𝒟_s$ and $𝒟_s = \left\{ (x^{l}_{s,i}, y^{l}_{s,i}) \right\}$0 (with estimated pseudo-labels).

2. Hierarchical Attention Fusion Mechanism

HADUA introduces a hierarchical attention module to explicitly model intra-modal temporal dynamics and inter-modal semantic interactions, generating a discriminative multimodal representation.

• Self-Attention for Intra-Modal Dynamics: For modality-specific feature matrices $𝒟_s = \left\{ (x^{l}_{s,i}, y^{l}_{s,i}) \right\}$1 and $𝒟_s = \left\{ (x^{l}_{s,i}, y^{l}_{s,i}) \right\}$2, multi-head self-attention is used:

$𝒟_s = \left\{ (x^{l}_{s,i}, y^{l}_{s,i}) \right\}$3

For each attention head:

$𝒟_s = \left\{ (x^{l}_{s,i}, y^{l}_{s,i}) \right\}$4

The heads are concatenated and projected to extract $𝒟_s = \left\{ (x^{l}_{s,i}, y^{l}_{s,i}) \right\}$5 and $𝒟_s = \left\{ (x^{l}_{s,i}, y^{l}_{s,i}) \right\}$6.

• Cross-Attention for Inter-Modal Guidance: Utilizing $𝒟_s = \left\{ (x^{l}_{s,i}, y^{l}_{s,i}) \right\}$7 as queries and $𝒟_s = \left\{ (x^{l}_{s,i}, y^{l}_{s,i}) \right\}$8 as keys/values, cross-attention is computed to yield $𝒟_s = \left\{ (x^{l}_{s,i}, y^{l}_{s,i}) \right\}$9, aligning complementary information from both physiological modalities.
• Fusion: The final multimodal embedding $x^{l}_{s,i}$0 is obtained as

$x^{l}_{s,i}$1

which is subsequently used for classification.

This architecture ensures both intra- and inter-modal dependencies are captured in a unified, semantically coherent representation (Tang et al., 29 Jan 2026).

3. Confidence-Aware Gaussian Weighting of Pseudo-Labels

Domain adaptation underpins HADUA by leveraging pseudo-labels assigned to target-domain instances. Pseudo-label noise poses challenges, especially under class imbalance and distribution shift.

To mitigate this, HADUA deploys a confidence-aware Gaussian weighting scheme:

• Let $x^{l}_{s,i}$2 denote the maximum predicted class probability (confidence) for the $x^{l}_{s,i}$3-th target sample.
• The weight is calculated as:

$x^{l}_{s,i}$4

where $x^{l}_{s,i}$5 and $x^{l}_{s,i}$6 are the running mean and standard deviation of confidences, updated via exponential moving averages.

• These weights down-weight the influence of low-confidence pseudo-labels in the conditional alignment process, reducing the adverse impact of label noise without hard thresholding.

This adaptive weighting enables smooth supervision transition in the target domain, critical for stable and robust conditional alignment.

4. Dynamic Uniform Alignment for Class-Balanced Adaptation

HADUA addresses residual class imbalance in pseudo-labels through a dynamic uniform alignment mechanism:

• Batch-predicted class distribution is given by $x^{l}_{s,i}$7.
• The target is the uniform distribution $x^{l}_{s,i}$8 over all $x^{l}_{s,i}$9 emotion classes.
• Each pseudo-probability $y^{l}_{s,i}$0 is transformed:

$y^{l}_{s,i}$1

with temperature $y^{l}_{s,i}$2.

• The final adjusted pseudo-probability

$y^{l}_{s,i}$3

interpolates between the raw and aligned probabilities with $y^{l}_{s,i}$4 increasing over training epochs.

• Uniform alignment regularization imposes the penalty $y^{l}_{s,i}$5, encouraging balanced class contributions.

This dynamic uniform alignment stabilizes conditional matching by mitigating class-imbalance in the target domain, fostering improved generalizability under domain shift.

5. Objective Function and Training Regimen

The comprehensive HADUA optimization objective integrates multiple components:

• Source classification loss:

$y^{l}_{s,i}$6

• Marginal Alignment (MMD loss):

$y^{l}_{s,i}$7

• Conditional Alignment (CMMD loss): Computed over class-specific RKHS means for labeled source and weighted, class-balanced pseudo-labeled target samples using $y^{l}_{s,i}$8 and $y^{l}_{s,i}$9.
• Uniform alignment regularizer: $i$0.

The total objective is: $i$1

The training loop involves: sampling mini-batches, hierarchical attention fusion, prediction and pseudo-label estimation, weight/regularizer computation, CMMD and uniform alignment, and iterative parameter and statistic updates until convergence (Tang et al., 29 Jan 2026).

6. Experimental Evaluation and Benchmarks

HADUA is empirically verified on leading cross-subject multimodal emotion recognition benchmarks:

• Datasets: SEED (3 emotions), SEED-IV (4 emotions), and DEAP (continuous valence/arousal). EEG signals are transformed via short-time Fourier into five frequency bands, with differential entropy features extracted per channel and band. Eye-movement features include time- and frequency-domain statistics. Evaluation uses leave-one-subject-out cross-validation.
• Baselines: Compared to DGCNN, MMDA, CSMM, CFDA-CSF, MACDB, among others.

Quantitative results are summarized as follows:

Dataset	Method	Acc (%)	Macro-F1 (%)	AUC (%)
SEED	HADUA	94.68	94.69	97.68
SEED	CSMM	94.96	95.21	96.20
SEED-IV	HADUA	92.00	92.88	92.02
SEED-IV	CSMM	89.82	90.03	93.01

Ablation studies show successive gains from hierarchical fusion, marginal alignment, Gaussian weighting, and uniform alignment. For example, SEED accuracy improves from 85.18% (base FNN) to 94.68% with the full HADUA pipeline.

Uniform alignment yields notable gains in per-class accuracy standard deviation, with HADUA achieving 1.92% (SEED) and 3.21% (SEED-IV), lower than competing methods.

t-SNE visualizations confirm improved clustering and domain overlap with progressive training.

7. Significance and Implications

HADUA demonstrates the effectiveness of combining hierarchical intra-/inter-modal attention, confidence-adaptive Gaussian weighting, and dynamic uniform alignment for robust cross-subject multimodal domain adaptation. The proposed framework delivers consistently superior robustness, class-balance, and accuracy under substantial domain heterogeneity and label noise.

A plausible implication is that the modular design of HADUA—separating representation fusion, confidence modeling, and distribution regularization—offers an adaptable foundation for general multimodal domain adaptation tasks, extending beyond physiological emotion recognition to other fields characterized by label scarcity, heterogeneity, and distribution shift (Tang et al., 29 Jan 2026).