AdaBrain-Bench: EEG-based BCI Model Evaluation

• AdaBrain-Bench is a standardized benchmark that unifies 13 EEG datasets across seven BCI application areas to evaluate brain foundation models.
• It implements a common adaptation pipeline with strategies like full fine-tuning and linear probing for robust model deployment and performance assessment.
• The benchmark introduces multi-dimensional metrics, including Balanced Accuracy and a novel Transferability Score, to guide improvements in neural decoding.

AdaBrain-Bench is a standardized large-scale benchmark designed to evaluate the performance and generalizability of brain foundation models for non-invasive EEG-based brain-computer interface (BCI) tasks. It unifies diverse EEG decoding paradigms, adapts them to a common evaluation pipeline, and introduces multi-dimensional performance metrics, including a novel measure of transferability. AdaBrain-Bench standardizes preprocessing, adaptation, and evaluation for 13 datasets across seven application areas, providing an extensible platform for reproducible and comparative research in neural decoding and foundation model development for BCIs (Wu et al., 14 Jul 2025).

1. Benchmark Scope and Composition

AdaBrain-Bench systematically samples seven canonical BCI task categories spanning cognitive, augmentation, and clinical applications. The benchmark encompasses 13 public EEG datasets, each standardized in terms of channel configuration, sample rate, and task structure. Unified preprocessing is applied to all datasets: 0.1–75 Hz zero-phase band-pass filtering, FFT-based notch filtering to remove local mains-frequency peaks, resampling to the model’s expected frequency, and normalization (z-score, 95th percentile for outlier-heavy sets, or unit rescale for Things-EEG).

The following table summarizes the key datasets:

Task	Datasets (Channels/Sampling)	Subjects/Samples
Emotion Recognition	SEED (62/1000), SEED-IV (62/1000)	15/144,852, 15/151,845
Workload Classification	EEGMAT (19/500)	36/1,080
Vigilance Estimation	SEED-VIG (17/200)	21/20,355
Motor Imagery	BCI-IV-2A (22/250), SHU (32/250)	9/5,184, 25/11,988
Visual Decoding	Things-EEG (63/1000)	10/821,600
Clinical Anomaly	TUEV (23/256), TUAB (23/var.), Siena	370/112,237; 2,383/409,083; 14/51,307
Sleep Staging	HMC (4/256), SHHS (1/125), Sleep-EDF	151/137,243; 329/324,854; 78/414,961

Channel-wise z-score normalization ($\mu=0$, $\sigma=1$) is standard, with dataset-specific alternatives to optimize for outlier robustness (Wu et al., 14 Jul 2025).

2. Unified Task Adaptation Pipeline

AdaBrain-Bench orchestrates a three-step adaptation process to map foundation models onto BCI tasks:

Feature Preparation: Each preprocessed EEG segment $X \in \mathbb{R}^{C\times T}$ is transformed via a $1 \times 1$ channel-wise convolution to match the foundation model’s required channel count $C_0$—$\hat{X} = \text{Conv}_{1\times1}(X)$.

Adaptation Strategies:

• Full fine-tuning: all parameters $\theta$ are updated using the task-specific loss.
• Linear probing: $\theta$ is frozen while only a lightweight task-specific head $W$ is trained.

Task-Specific Heads and Losses:

• Classification: global pooling, linear layer $W$, cross-entropy ($\sigma=1$0) or binary cross-entropy ($\sigma=1$1).
• Regression: average pooling, two hidden layers, nonlinear activation (ReLU/ELU), and mean squared error ($\sigma=1$2).
• Retrieval: flattened embedding $\sigma=1$3 $\sigma=1$4-normalized projection, InfoNCE contrastive loss.

All models use AdamW (weight decay 0.05), batch size 64, learning rates grid-searched in $\sigma=1$5, and train for up to 50 epochs (Wu et al., 14 Jul 2025).

3. Evaluation Metrics and Transferability

AdaBrain-Bench’s evaluation framework features an array of classical accuracy metrics enriched by a transferability score.

Classification:

• Balanced Accuracy (BA)
• Weighted F1-score ($\sigma=1$6)
• AUROC/AUC-PR for class-imbalanced detection

Regression:

• Pearson correlation coefficient (Corr)
• $\sigma=1$7 coefficient

Retrieval:

• 2-Way accuracy
• Top-5 recall

Transferability Score (TS): This measures the normalized improvement over training from scratch, interpolated between random baseline and oracle upper bound:

$\sigma=1$8

where $\sigma=1$9 is adapted foundation model performance, $X \in \mathbb{R}^{C\times T}$0 is the same architecture trained de novo, and $X \in \mathbb{R}^{C\times T}$1 is the ideal (1.0 for cross-subject, best achievable accuracy for few-shot) (Wu et al., 14 Jul 2025).

4. Software Infrastructure and Extensibility

The AdaBrain-Bench toolkit is implemented in PyTorch and structured for extensibility and reproducibility. The codebase is modularized as follows:

• data/: normalized dataset loaders and stratified splitters
• preprocess/: spectral filters, resamplers, normalization tools
• models/: wrappers for both foundation and traditional baselines, plus dynamic channel adapters
• experiments/: YAML-based configurations, training scripts, metric logging, and plotting

Multi-GPU training, stratified cross-validation, and automated few-shot sampling are included. Each experiment is reproducible via a single configuration file specifying backbone, head, adaptation regime, hyperparameters, data splits, and metrics (Wu et al., 14 Jul 2025).

5. Experimental Protocols and Transfer Scenarios

Three primary transfer settings are formalized:

Cross-Subject Transfer: Data are split into subject-disjoint train/val/test partitions. Performance is averaged over three random splits. The model’s ability to generalize to unseen subjects is assessed.

Multi-Subject Adaptation: For certain datasets, training/validation/test splits are constructed at the session or trial level within each subject, exposing models to intra-subject variability.

Few-Shot Transfer: Within the cross-subject regime, only $X \in \mathbb{R}^{C\times T}$2 fraction of labeled samples per class per subject are available for adaptation. Remainder form the query set. Results are averaged across three random seeds (Wu et al., 14 Jul 2025).

6. Empirical Findings and Recommendations

AdaBrain-Bench has enabled systematic comparison across four self-supervised brain foundation models (BIOT, EEGPT, LaBraM, CBraMod) and four supervised baselines (EEGNet, LDMA, ST-Tran, Conformer). Key outcomes are:

• Foundation models, notably LaBraM and CBraMod, significantly outperform traditional methods on the majority of datasets (LaBraM macro-BA 64.6%, CBraMod 62.7%, Conformer 58.1%).
• The most substantial improvements are observed on limited-data tasks (e.g., +12% BA on EEGMAT).
• Clinical anomaly detection and sleep staging tasks (TUAB, Siena, HMC, Sleep-EDF) demonstrate near-parity or a slight traditional lead, indicating that larger labeled cohorts can close the gap.
• Foundation models demonstrate >6 percent-point BA improvement in multi-subject adaptation, but inter-subject transfer remains more challenging (average BA drops by ~15 points).
• In few-shot regimes, CBraMod achieves the highest average TS (0.28), attributed to dynamic channel embeddings, with strong performance maintained down to $X \in \mathbb{R}^{C\times T}$3.
• Pretraining on large unlabeled EEG yields marked accuracy improvements (+10 points BA for LaBraM in cross-subject), especially when downstream datasets are small.
• Increasing training subjects improves BA but exhibits plateauing gains beyond $X \in \mathbb{R}^{C\times T}$440 subjects.
• Z-score normalization performs robustly across tasks; 95th-percentile scaling is optimal for outlier-prone datasets.
• LaBraM preferred for cross-subject generalization in small cohorts; CBraMod excels with variable channel layouts or pronounced few-shot constraints.
• For very large foundation models (e.g., EEGPT, 25M parameters), linear probing and regularization reduce overfitting risk compared to full fine-tuning (Wu et al., 14 Jul 2025).

7. Significance and Practical Impact

AdaBrain-Bench consolidates benchmarking practice for EEG-based BCI foundation models by standardizing tasks, preprocessing, adaptation, and performance measurement. The platform’s extensible software architecture, comprehensive dataset coverage, and diagnostic metric suite enable reproducible evaluation and systematic progress toward generalizable neural decoding. By providing quantifiable measures of transferability and practical guidance on preprocessing, adaptation strategy, and cohort sizing, AdaBrain-Bench is designed to accelerate the development and deployment of robust BCI algorithms. All code and resources are publicly available at https://github.com/Jamine-W/AdaBrain-Bench, supporting transparent benchmarking and future task/model integration (Wu et al., 14 Jul 2025).