EEGFormer: Transformer-based EEG Representation

• EEGFormer is a transformer-based model for EEG that employs frequency patchification, channel-independent encoding, and vector quantization for robust, interpretable feature learning.
• It leverages self-supervised pretraining on a compound, unlabeled EEG corpus to generate universal representations transferable to tasks like seizure detection and artifact classification.
• Empirical benchmarks show EEGFormer outperforms baselines, achieving approximately 15.8% and 14.1% improvements in AUPRC for neonatal and TUSZ seizure detection tasks.

EEGFormer is a transformer-based foundation model for electroencephalography (EEG) that advances large-scale, transferable, and interpretable EEG representation learning. Designed to utilize abundant unlabeled brain signal data from diverse sources, EEGFormer leverages self-supervised pretraining on compound datasets to generate universal, discrete EEG representations with strong generalization across downstream tasks such as anomaly detection, artifact classification, and seizure identification. A distinguishing feature is its vector quantization approach, which not only regularizes representations but also enables interpretability through discrete codebook analysis, bridging model outcomes and clinical relevance.

1. Model Architecture

EEGFormer uses an encoder–decoder transformer architecture tailored to multi-variate EEG signals $X \in \mathbb{R}^{L \times C}$, where $L$ is the number of time steps and $C$ the number of channels. The design is characterized by:

• Frequency Patchification: Each EEG channel $x_c \in \mathbb{R}^L$ is partitioned into patches of length $P$ (stride $S$) in the frequency domain after FFT. Each patch is mapped to an embedding space with a learnable weight matrix $w_p \in \mathbb{R}^{P \times D}$:

$\hat{x}_c = \left(x_c\right)^\top w_p + w_{pos}$

where $w_{pos} \in \mathbb{R}^{N \times D}$ injects positional information.

• Channel-Independent Transformer Encoder: The encoding is performed separately on each channel, capturing channel-specific temporal dependencies while maintaining modularity.
• Vector Quantization: Each encoder output $h_i$ is quantized against a learnable codebook $\{v_1, \ldots, v_K\}$ by:

$z_i = \arg\min_j \left\| h_i - v_j \right\|_2$

This discretizes feature space, a central mechanism for regularization and later interpretability.

• Transformer Decoder: The quantized embeddings are input to a shallow transformer decoder, which reconstructs the original EEG signal via projection and reshaping.
• Loss Function: The objective for each $X$ is a combination of reconstruction loss (for $X_{rec}$) and codebook commitment/embedding proximity:

$$\| X_{rec} - X \|_2^2 + \sum_{c=1}^C \sum_{j=1}^N \left[ \| \mathrm{sg}(H_{i,j}) - v_{Z_{i,j}} \|_2^2 + \| H_{i,j} - \mathrm{sg}(v_{Z_{i,j}}) \|_2^2 \right]$$

with $\mathrm{sg}[\cdot]$ as a stop-gradient operator.

2. Self-Supervised Pretraining Paradigm

EEGFormer utilizes self-supervised training on large-scale unlabeled data (1.7TB from the TUH Corpus), diverging from typical single-dataset or task-based pretraining strategies. Key mechanisms include:

• Discrete Token Learning: Vector quantization produces a set of discrete tokens (indices) representing recurring EEG patterns, which forms a codebook of signal motifs rather than relying on masked inference or continuous embeddings.
• Reconstruction as Proxy Objective: By enforcing reconstruction from discrete tokens, the model is compelled to learn a representation basis (the codebook) that encodes essential signal structure, capturing both local and global dependencies without human annotations.
• Scalability: The compound EEG pretraining corpus encompasses multiple conditions and tasks, ensuring that representations are universal rather than overfitted to narrow paradigms.

3. Transferability Across Downstream EEG Tasks

EEGFormer demonstrates robust representation transfer in multiple scenarios:

• Downstream Evaluation: Extensively tested on tasks such as abnormality detection, artifact recognition, seizure classification, and neonatal seizure detection in an out-of-domain Neonate dataset, EEGFormer outperforms baselines such as EEGNet, TCN, and EEG-GNN, offering gains of approximately 15.8% (Neonate) and 14.1% (TUSZ) in area under the precision–recall curve (AUPRC).
• Fine-Tuning Flexibility: Both encoder and decoder weights support flexible adaptation. Even with only linear classifier probing on frozen features, performance is competitive, underscoring representation quality.
• Quantitative Superiority: Consistent improvements in metrics such as AUROC, AUPRC, and F1-score relative to established and self-supervised baselines (e.g., BrainBERT).

4. Model Interpretability: Codebook and Token Analysis

Interpretability in EEGFormer pivots on discrete code indices and their clinical meaning:

• n-Gram Feature Extraction: Sequences of quantized tokens (n-grams) can be analyzed with simple statistical tools, such as naive Bayes classifiers, to identify motifs correlated with clinical events (e.g., seizure waveforms).
• Visualization and Localization: High-scoring n-gram tokens localize to known EEG patterns, enabling mapping of model decisions to recognizable EEG phenomena such as epileptiform discharges.
• Clinician-Accessible Explanations: This approach bridges black-box model decisions with the expert understanding required for clinical adoption, providing insight into which parts of a recording and which motifs were instrumental for a given anomaly detection.

5. Performance Benchmarks and Comparative Evaluation

EEGFormer delivers strong empirical results:

Dataset / Task	Baselines	EEGFormer Best	Improvement (AUPRC/F1)
Neonate (Seizure)	EEGNet, TCN, GNN	+15.8%	Yes
TUSZ (Seizure Detection)	EEGNet, TCN, GNN	+14.1%	Yes

Performance is preserved across tasks and with transfer to unseen data distributions. Prolonged pretraining further enhances downstream metrics. Ablation studies validate that longer exposure to diverse data yields more generalizable representations.

6. Applications and Broader Implications

EEGFormer is applicable in multiple scenarios with significant research and clinical implications:

• Clinical EEG Analysis: Automation of complex annotation tasks such as artifact removal, seizure detection, and abnormality identification, reducing the burden on clinical experts.
• Cross-Dataset Adaptation: The universality of learned features allows the model to be applied in new domains or experimental settings without retraining from scratch.
• Enhancing Trust via Interpretability: The codebook-centered interpretability provides transparent mappings between signal patterns and clinical labels, improving the acceptability of automated systems.
• Foundational Model Paradigm in Biomedicine: EEGFormer exemplifies the large-scale, unlabeled pretraining approach in a domain traditionally starved for annotated data, aligning with trends in NLP and computer vision and paving the way for multi-modal or even multi-organ foundation models.

7. Context, Limitations, and Outlook

EEGFormer establishes a standard for scalable, interpretable, and transferable EEG modeling. With its channel-independent patchification, vector quantization, and self-supervised objectives, it moves beyond dataset- and task-specific solutions, elevating reproducibility and generalization. Nonetheless, comparative benchmarks (Xiong et al., 25 Aug 2025) indicate that the architecture could be further improved by explicit spatio-temporal specificity in attention mechanisms or multi-task fine-tuning strategies. Integrations with neurophysiology-informed architectures (e.g., graph neural networks for topographical priors) and more advanced classifier heads also represent promising avenues. The discrete token/patch quantization approach is a promising direction for interpretability and cross-domain generalization, but as benchmarking reveals, there remains a gap between frozen backbone performance and full end-to-end fine-tuning, suggesting current representations, while robust, may be further enhanced for high-level semantic abstraction.