TransformEEG: Transformer Models for EEG Analysis

• TransformEEG is defined as the application of transformer models to EEG data, integrating classical DSP, manifold embeddings, and deep architectures to form compact and interpretable signal representations.
• It employs hierarchical temporal-spatial modeling and hybrid CNN-transformer designs to capture multi-scale dependencies and enhance decoding accuracy.
• Its efficient parameterization, invertibility through reconstruction, and integration of domain-specific tokenizations enable robust clinical and cognitive neuroscience applications.

TransformEEG encompasses a broad spectrum of methodologies leveraging transformer-based architectures to advance electroencephalography (EEG) analysis. Defined as the application of transformer models—or transformer-inspired components—to EEG data, TransformEEG targets key challenges in modern neuroinformatics: generalizability, interpretability, multi-scale and multi-modal representation, efficient parameterization, and signal reconstruction. These approaches unite digital signal processing, Riemannian and graph geometry, domain-specific tokenizations, and hybrid deep architectures to form compact, robust, and explainable EEG representations for both clinical and cognitive neuroscience applications.

1. Foundations: From Classical DSP to Manifold Transformers

A foundational principle in advanced TransformEEG architectures is the coupling of classical digital signal processing (DSP) with manifolds and modern sequence models. MENDR is a canonical example, blending filter bank signal decomposition via discrete wavelet packet transform (DWPT) with symmetric positive definite (SPD) matrix embeddings, mapped onto Riemannian manifolds. The DWPT isolates canonical EEG bands ($\delta$, $\theta$, $\alpha$, $\beta$, $\gamma$), and per-band short-window sample covariance matrices are computed as

$$\mathrm{SCM} = \frac{1}{T_{\text{band}}-1} X_{\text{band}} X_{\text{band}}^T$$

These covariance matrices are trace-normalized and serve as the primary patch-level embeddings. This manifold-aware representation, quite distinct from conventional flat vector encodings, preserves the multivariate dependencies between EEG channels intrinsically and enables geometric reasoning and visual interpretability via ellipsoid rendering in the eigenbasis. The Riemannian Manifold Transformer extends the transformer’s feed-forward and self-attention blocks into the SPD matrix domain using BiMap layers and manifold attention mechanisms with Log-Euclidean metrics, supporting aggregation, normalization, and invertibility (Chen et al., 7 Aug 2025).

2. Architectures: Temporal-Spatial and Multi-Scale Modeling

Modern transformer-based EEG architectures systematically address the joint temporal and spatial correlation structure which is fundamental to EEG signals. Temporal-spatial transformers (e.g., TSERT) operate hierarchically: a shared temporal transformer per channel mines within-channel dynamics, followed by spatial encoders that aggregate at regional and global scales:

• Temporal extractor: segments each channel, applies a shared transformer, and outputs per-channel features.
• Spatial aggregators: regional transformers pool descriptors for predefined brain regions; an upper-level spatial transformer fuses region-level features for holistic decoding.

Self-attention mechanisms at every scale enable the model to focus on dynamically informative temporal segments and spatial regions, adaptively suppressing redundancy and optimizing emotion, task, or pathology decoding (Wang et al., 2022).

Hybrid variants—such as Dual-TSST—extend this paradigm with dual-branch CNN structures: one extracts temporal-spatial features from the raw EEG, while the other operates on wavelet-transformed time-frequency inputs. Their outputs are fused and processed by transformer encoder stacks to model long-range, non-stationary dependencies. This design achieves state-of-the-art performance on multi-class EEG decoding with robust accuracy across datasets (Li et al., 2024).

3. Interpretability, Invertibility, and Model Efficiency

Interpretability and invertibility are core to clinical and scientific deployment. Manifold-transformer models (e.g., MENDR) permit learned SPD embeddings to be visualized as geometric ellipsoids—a mapping that directly links embedding geometry to covariance structure and channel contributions. Critically, the manifold transformer is invertible: learned SPD embeddings undergo logarithmic and linear inversion through dedicated decoders (including ConvTranspose, Squeeze-and-Excitation blocks, and GNN-based channel harmonizers), followed by inverse DWPT, enabling nearly lossless EEG signal reconstruction. This property permits quantitative evaluation of information retention and offers a biological guarantee that the model’s low-dimensional representation does not discard diagnostically relevant components (Chen et al., 7 Aug 2025).

On the efficiency front, these models achieve competitive or near–state-of-the-art results with striking parameter reductions: for instance, MENDR achieves balanced accuracy within 2–4 points of leading baselines on TUAB and TUEV benchmarks while using 1/3–1/20 the parameter count.

Model	Params (M)	TUAB BA	TUAB AUROC	TUEV BA	Notes
MENDR-Tiny	1.8	0.779	0.867	0.545	Geometric/invertible, 5 bands
LaBraM-Base	5.8	0.814	0.902	0.641	SOTA (larger baseline)
BIOT	3.2	0.796	0.882	—	SoTA param-efficient

4. Denoising, Generalizability, and Lightweight Design

Artifact removal and generalizability are critical for robust EEG decoding. Transformer-based denoisers, such as EEGDnet, alternate between self-attention (global, non-local self-similarity) and per-segment feed-forward blocks (local, fine-scale structure). This approach achieves lower temporal and spectral RRMSE and higher correlation coefficients than CNN or RNN baselines while maintaining a compact footprint (0.18M parameters) suitable for wearable and embedded applications (Yi et al., 2021).

Generalizability across datasets and subject variability is systematically addressed in frameworks like TransformEEG for Parkinson’s detection. This architecture leverages a depthwise convolutional tokenizer to generate channel-preserving, channel-specific tokens, followed by a shallow, efficient transformer encoder (L=2, H=1) and global pooling. Evaluated on an aggregated four-dataset cohort (290 subjects, 100-fold nested cross-validation), this model achieves the highest median balanced accuracy (80.10%) and the lowest IQR (5.74%) among eight contemporary EEG models, evidencing substantial improvements in inter-subject consistency (Pup et al., 10 Jul 2025).

5. Downstream Applications and Clinical Integration

TransformEEG models are evaluated on, and optimized for, a diverse suite of large-scale clinical and cognitive benchmarks:

• Sleep staging (ISRUC, HMC): multiclass, long-duration EEG segments; assessed via Cohen’s $\kappa$ and F1.
• Seizure/event detection (TUAB, TUEV, TUSL): binary and multiclass classification with balanced accuracy/AUROC.
• Gait/continuous motor regression (MoBI): Pearson $r$, $\mathrm{RMSE}$.
• Emotion decoding (SEED-V): cross-subject robustness and domain transfer.

Foundation model variants (e.g., MENDR, ALFEE) are pre-trained on massive multi-paradigm corpora (≥4,000–25,000 hours) and fine-tuned with task-specific objectives (classification, regression), masking and forecasting losses, and data harmonization. These approaches routinely match or exceed benchmark performance, with the added benefit of structured, interpretable representations and reduced memory/computation demands (Chen et al., 7 Aug 2025, Xiong et al., 7 May 2025).

6. Future Directions: Self-Supervision, Multi-Modality, and Explainability

A distinguishing direction is the push toward open-label self-supervised pretraining, variable-channel and cross-paradigm adaptation, and further advances in explainability:

• Channel-robust modeling: Hybrid transformers (e.g., ALFEE) employ explicit channel encoders and dataset/task tokens, supporting missing data and multi-montage compatibility (Xiong et al., 7 May 2025).
• Synthetic and generative modeling: Generative EEG transformers (e.g., GET) enable continuous context-aware signal synthesis, facilitating data augmentation and closed-loop BCI simulation (Ali et al., 2024).
• Multi-modal fusion: Manifold Transformers and graph-inspired transformers (e.g., balanced signed graph networks) offer frameworks for principled integration of structural EEG knowledge, graph-topological priors, and external modalities (video, text, fMRI) (Yao et al., 3 Oct 2025).
• Explainability: Embedding geometry (ellipsoidal visualization), self-attention heatmaps, and inverse reconstructions provide direct tools for neuroscientific interpretability, essential for clinical adoption.

The confluence of these techniques within the TransformEEG paradigm establishes a benchmark for interpretable, parameter-efficient, and generalizable EEG representations suitable for both foundational neuroscientific research and translational clinical practice (Chen et al., 7 Aug 2025, Xiong et al., 7 May 2025, Pup et al., 10 Jul 2025).