Masked Generative-Contrastive Representation Learning for Cross-Dataset EEG-Based Emotion Recognition

Abstract: Self-supervised learning (SSL) shows strong potential for cross-dataset transfer by improving feature representation and generalization. However, its application to EEG-based emotion recognition remains largely unexplored. Existing SSL methods struggle to capture the intricate spatiotemporal dependencies of EEG signals under varying channel configurations, extract fine-grained representations resilient to noise, and derive global features that generalize well across subjects. To address these challenges, we propose Masked Generative-Contrastive Representation Learning (MGCRL), a novel SSL framework specifically designed for EEG-based emotion recognition. Built upon a region-aware spatiotemporal encoder, MGCRL integrates generative and contrastive learning to achieve both fine-grained and global discriminative representations for cross-dataset generalization. MGCRL introduces three key designs: 1) a spatiotemporal encoder that incorporates region-based graph convolution to capture localized spatial and functional relationships, enhancing region-specific feature learning and mitigating the impact of varying EEG channel configurations across datasets; 2) a generative learning mechanism based on the joint embedding predictive architecture (JEPA) that utilizes masked features to capture noise robustness fine-grained representations, improving the model's capability to characterize subtle emotional states; and 3) a contrastive learning strategy that leverages masked and original features to learn temporally stable and cross-subject-invariant representations across the same stimuli, boosting emotion discrimination and cross-subject generalization. Under these designs, MGCRL exhibits remarkable ability to learn universal representation. Extensive experiments involving pretraining on the large FACED dataset and fine-tuning on multiple SEED-series datasets demonstrate the effectiveness of MGCRL.

• The paper introduces a masked generative-contrastive learning framework that significantly improves EEG-based emotion recognition across datasets.
• It integrates a region-aware spatiotemporal encoder with JEPA-based generative and dynamic contrastive modules to extract robust, subject-invariant features.
• Experimental evaluations on multiple benchmarks confirm superior cross-subject accuracy and effective transferability even with minimal fine-tuning data.

Masked Generative-Contrastive Representation Learning for Cross-Dataset EEG-Based Emotion Recognition

Motivation and Problem Setting

EEG-based emotion recognition (ER) underpins affective brain-computer interfaces but faces persistent challenges in generalizability due to the heterogeneity of EEG recording setups—device differences, channel configurations, and non-aligned label spaces—across datasets. Existing self-supervised learning (SSL) methods exhibit shortcomings when applied to raw EEG: limited capacity to model intricate spatiotemporal dependencies, poor noise robustness, and inadequate extraction of subject-invariant features. These deficits result in severe generalization gaps for subject- or dataset-transfer tasks.

MGCRL Framework Overview

The paper introduces Masked Generative-Contrastive Representation Learning (MGCRL), an SSL paradigm tailored for EEG-based ER, integrating generative and contrastive learning objectives above a region-aware spatiotemporal encoder. MGCRL’s architecture and learning dynamics are defined by three core components:

1. Region-aware Spatiotemporal Encoder: Utilizes temporal convolutions and intra-brain-region GCNs, dynamically combining anatomical (spatial adjacency) and functional (activity-derived) connectivities. Spatial aggregation follows a 17-region partitioning to accommodate varying channel sets. A subsequent multi-head self-attention mechanism synthesizes long-range temporal dependencies.
2. JEPA-based Generative Learning: Implements a Joint Embedding Predictive Architecture, where the context encoder predicts masked temporal embeddings (not signals), guided by a target encoder updated via EMA. Unlike autoencoder-style reconstruction, this paradigm targets semantically meaningful representations and suppresses sensing noise.
3. Masked Dynamic Contrastive Module: Employs two views (masked and unmasked) of each sample for intra-sample consistency and constructs dynamic, adaptive positive pairs based on temporal stability and cross-subject emotional similarity (with fuzzy weights) under identical stimuli. The contrastive loss function thus enforces both fine-grained and global, subject-invariant feature consistency.

MGCRL merges these objectives, structurally linking the masking mechanism for both tasks, promoting efficient and robust representation learning without extensive labeled supervision.

Experimental Evaluation and Results

MGCRL underwent rigorous evaluation on cross-subject and cross-dataset EEG ER tasks using the FACED, SEED-IV, SEED-V, and SEED-VII benchmarks. Pretraining was performed on large-scale, unlabeled FACED data before fine-tuning and LOSO (leave-one-subject-out) assessment on SEED datasets. Key findings include:

• Superior Cross-Subject Accuracy: MGCRL outperformed all self-supervised and generative/contrastive baselines (e.g., BIOT, TS-TCC, EEG2Rep, LaBraM) by notable margins on SEED-IV (41.47% accuracy, +4.27% over best baseline), SEED-V (+3.01%), and SEED-VII (+3.70%). The robustness of the method was pronounced across most subjects.
• Cross-Session Transferability: MGCRL yielded higher accuracy with only minimal degradation under cross-session settings (FACED→SEED-IV: 40.07%, +4.37%; FACED→SEED-V: 33.56%, +3.00%), indicating resistance to variations in emotional stimuli.
• Ablation Validates Module Contributions: Removing the contrastive loss or region-aware GCN module resulted in significant accuracy drops, underscoring their necessity for discriminative and generalizable representation learning. Importantly, the MDCLL (masked dynamic contrastive loss) performed substantially better than conventional augmentation-based contrastive objectives.
• Pretraining Efficacy: Models initialized with MGCRL pretraining consistently surpassed those trained from scratch, even with far fewer fine-tuning subjects. Notably, using only 3-6 subjects with MGCRL pretraining achieved better accuracy than using twice as many subjects with no pretraining.
• Interpretability: Spatial kernel weights revealed consistency in the importance of frontal, temporal, and parietal regions for emotion recognition across datasets, concordant with neurophysiological literature.

Implications and Future Directions

MGCRL demonstrates that unifying masked generative and contrastive self-supervision with anatomically-informed encoders substantially advances EEG ER generalizability across heterogeneous datasets and subjects. Its robust performance at low sample and fine-tuning regimes emphasizes the value of large-scale unlabeled pretraining in neuro-signal processing.

Key implications:

• Practical Transferability: MGCRL substantially reduces labeled data dependency and enhances ER robustness for real-world BCI deployments with device and protocol heterogeneity.
• Flexibility: The architecture and learning objectives are readily extensible to other EEG-based tasks confronting similar transfer, noise, and annotation challenges.
• Theoretical Advancement: The joint use of adaptive spatiotemporal graph encoding, JEPA-based masked prediction, and dynamic inter/intra-sample contrastive objectives fills a gap in SSL for complex, variable-sized time series.

Directions for further investigation include:

• Investigating domain-adaptive or continual learning extensions for ongoing adaptation to new EEG sensors and protocols
• Applying MGCRL to other cognitive or clinical EEG decoding applications beyond ER
• Enhancing interpretability by incorporating neurobiological priors or explainable AI modules

Conclusion

MGCRL sets a new standard for SSL in EEG-based emotion recognition, achieving both fine-grained and subject-invariant feature extraction via a complementary generative-contrastive strategy anchored in region-aware spatiotemporal encoding. The empirical results on large cross-dataset benchmarks substantiate the approach’s practical utility and generalization capacity, offering a unified foundation for transferable EEG representation learning (2607.04139).