---
title: Brain Foundation Models
url: https://www.emergentmind.com/topics/brain-foundation-model-bfm
type: topic
---

# Brain Foundation Models

Searching arXiv for recent papers on Brain Foundation Models to ground the article in current literature.
Brain foundation model (BFM) denotes a family of large, pretrained neural models for brain signals and brain imaging that learn transferable representations from large-scale unlabeled or weakly labeled neural data. In current usage, the term spans at least three closely related formulations: deep-learning systems pretrained on very large collections of neural recordings such as EEG, fMRI, iEEG, MEG, and fNIRS; neuroimaging-specific foundation models that support zero-shot or light-fine-tuning adaptation across segmentation, classification, synthesis, question-answering, and related tasks; and a broader proposal to train foundation models directly on “brain-generated data” in addition to classical human-generated and device-generated corpora [2503.00580] [2506.13306] [2601.12053]. Across these formulations, the recurring criteria are large-scale pretraining, self-supervised or unsupervised objective functions, downstream adaptability, and some degree of neuroscientific or clinical interpretability.

## 1. Definition and conceptual scope

A precise survey definition describes BFMs as deep-learning systems pretrained on very large collections of neural recordings via self-supervised or weakly supervised objectives. They are intended to learn “universal neural representations” that capture the spatiotemporal structure of brain signals, generalize across different tasks, modalities, and experimental conditions, and support both downstream decoding and brain discovery [2503.00580]. In brain imaging, a stricter definition requires pretraining on massive, diverse medical imaging corpora with at least two brain-relevant modalities or pathologies, use of self-supervised or unsupervised objectives, and zero-shot or light-fine-tuning adaptability across multiple downstream neuroimaging tasks [2506.13306].

The literature also uses the term at a broader systems level. One proposal defines a BFM as a large-scale neural network trained directly on human brain–generated data, alongside classical corpora, with the explicit aim of improving robustness in perception, alignment with human values, executive reasoning, and conceptual integration [2601.12053]. This suggests that “BFM” is not restricted to a single architecture or modality. Rather, it names a pretraining paradigm centered on brain data.

A related point of clarification is terminological. “BFM” usually denotes the model class, whereas “BrainFM,” “BrainFM-MRI,” “BrainHarmonix,” “BrainFIBRE,” and “RABBiT” are names of particular systems within that broader class [2509.00549] [2511.03014] [2509.24693] [2607.00573] [2607.05171].

## 2. Data modalities and architectural patterns

BFMs are designed for a heterogeneous data regime. Surveys and reviews cover EEG, iEEG/ECoG, fMRI, MEG, fNIRS, MRI, CT, PET, SPECT, ultrasound, and paired multimodal streams that include text, images, physiological sensors, or clinical metadata [2506.16009] [2506.13306] [2602.02511]. In brain imaging alone, one review catalogs 161 brain imaging datasets and 86 FM architectures, including 146 3D and 15 2D datasets spanning 541 k total studies [2506.13306].

Two high-level design patterns recur. The first is a unified signal processor: a single Transformer- or Vision-Transformer-style backbone that ingests 1D, 2D, or 3D patches with positional encodings over time, channel, space, or modality [2503.00580]. The second is a multimodal encoder-plus-fusion pattern, in which modality-specific encoders are connected by cross-attention or learned weighting modules [2503.00580]. In brain imaging, dominant backbones include U-Net variants, Vision Transformers or hybrid ResNet+Transformer encoders, and SAM-based segmentation backbones augmented with prompt encoders [2506.13306].

Representative instantiations illustrate how broad this architectural space has become.

| System | Primary modality | Distinctive mechanism |
|---|---|---|
| Modular fMRI BFM [2508.16597] | resting-state fMRI / FC | LMAE + RW-MoE + SSM/Mamba |
| BrainFM-MRI [2511.03014] | multi-sequence brain MRI | single encoder, modality embeddings, CLN |
| BrainHarmonix [2509.24693] | T1 MRI + fMRI | shared brain-hub tokens, TAPE |
| BrainFIBRE [2607.00573] | NODDI-derived maps | SPID + CCC + MoE |
| BrainFM [2509.00549] | CT and MRI | modality-agnostic multi-task 3D U-Net |
| RABBiT [2607.05171] | audio-to-fMRI | LoRA brain-tuning, TBT, SID |

These examples also show that BFMs are not uniformly “LLM-like.” Some are sequence models over ROI time series, some are volumetric vision encoders, some are multimodal fusion models, and some are domain-specific state-space or mixture-of-experts systems [2508.16597] [2502.04892] [2509.24693].

## 3. Self-supervised pretraining and adaptation

The pretraining literature is organized around masked reconstruction, autoregressive prediction, and contrastive learning [2503.00580]. A generic masked objective for multichannel signals is written as
$$
L_{\mathrm{mask}}=\mathbb{E}_{X,m}\left[\|m\odot X-m\odot \hat X\|^2\right],
$$
while a representative contrastive objective is InfoNCE,
$$
L_{\mathrm{InfoNCE}}=- \sum_{i=1}^N \log
\frac{\exp(\mathrm{sim}(h_i,h_i^+)/\tau)}
{\sum_{j=1}^N \exp(\mathrm{sim}(h_i,h_j^-)/\tau)}.
$$
Neuro-SSL surveys further distinguish self-predictive learning, non-contrastive methods such as BYOL, SimSiam, and Barlow-Twins, and multimodal contrastive alignment across EEG, fMRI, MEG, text, images, and wearable sensors [2506.16009].

Adaptation strategies are correspondingly standardized. A pretrained BFM may be used as a frozen encoder with a task head, often described as a linear probe, or it may be fine-tuned partially or end-to-end, sometimes with adapters or lower learning rates to preserve pretrained structure [2506.16009]. Brain imaging reviews identify LoRA and bottleneck adapters as parameter-efficient fine-tuning mechanisms inside frozen backbones [2506.13306].

Some BFMs depart from standard masked-autoencoding pipelines. The stochastic optimal control model also referred to as Brain Dynamics with Optimal control (BDO) formulates representation learning through a continuous–discrete state-space model, a control-cost ELBO, and a locally linear approximation that enables simulation-free inference and an $O(\log k)$ parallel scan algorithm over time steps [2502.04892]. At the opposite end of the spectrum, the proposal to train foundation models directly on brain-generated data introduces reinforcement learning from human brain (RLHB) and chain of thought from human brain (CoTHB) as post-training mechanisms that would use valuation-level and execution-level neural signals as rewards or guidance signals [2601.12053].

This diversity of objective functions is significant. It indicates that BFMs are defined less by any single loss and more by the combination of large-scale pretraining, transferability, and neural-domain specificity.

## 4. Representative systems and empirical exemplars

A concrete realization of a BFM for resting-state fMRI is the modular framework composed of a Local Masked Autoencoder, a Random Walk Mixture of Experts module, and an SSM-based predictor. On Cam-CAN, it reports age-prediction MAE $=5.343\pm0.352$ with PCC $=0.928\pm0.036$, and fluid-intelligence prediction MAE $=2.940\pm0.251$ with PCC $=0.887\pm0.067$; ablations without LMAE or without RW-MoE degrade both tasks, and expert-weight visualization is used to identify functionally coherent ROI subsets [2508.16597].

In brain MRI, “BrainFM-MRI” uses one encoder with learnable modality embeddings, conditional layer normalization, and a masked autoencoding objective that accounts for missing modalities. Pretraining uses FOMO25 with 11 187 subjects, 13 900 sessions, and 60 529 MRI scans from 16 centers. Preliminary transfer on MSLesSeg, using 10 cases for 2 epochs, reports Dice 0.45 and HD95 11.2 for the BrainFM encoder plus segmentation head, compared with Dice 0.39 and HD95 12.8 for nnU-Net with T1+FLAIR [2511.03014].

“BrainHarmonix” extends the paradigm to multimodal structure–function pretraining. It is described as the first multimodal brain foundation model that unifies structural morphology and functional dynamics into compact 1D token representations. Its pretraining data comprise 64,594 T1-weighted structural MRI 3D volumes and 70,933 fMRI time series, with total fMRI samples increased to 252,961 through multi-TR augmentation. The system combines a 3D MAE structure encoder, a JEPA dynamics encoder with Temporal Adaptive Patch Embedding for heterogeneous TRs, and a fusion module with $N_H=128$ brain-hub tokens [2509.24693].

At the microstructural level, “BrainFIBRE” is presented as the first foundation model for brain microstructure, pretrained on NODDI-derived maps from 55,592 UK Biobank participants. Its Self-supervised Partial Information Decomposition objective and Counterfactual Candidate Construction strategy use five experts to separate unique, redundant, and synergistic information across ODI, NDI, and FWF maps. On the UK Biobank held-out test set of $N=4,307$, it reports age MAE $=3.95$ years with Corr $=0.76$ and sex classification F1 $=89.30\%$, while additional results are reported on HCP-Aging and the SINGER Asian cohort [2607.00573].

The modality-agnostic multi-task “BrainFM” addresses a different problem: robustness to contrast, resolution, orientation, and artifacts across CT and MRI. It uses a five-level 3D U-Net trunk with heads for image synthesis, anatomy segmentation, scalp-to-cortical distance, bias-field estimation, and registration, together with “mild-to-severe” intra-subject generation and “real-synth” mix-up. Evaluation spans eleven public datasets. In setup I, aggregate T1w segmentation Dice improves from 0.790 for SCRATCH to 0.822 for Brain-ID and 0.846 for BrainFM, while T1w synthesis L1 improves from 0.025 to 0.017 to 0.015 [2509.00549].

RABBiT targets a different axis of generalization: zero-shot and few-shot prediction of speech-elicited fMRI responses. It combines LoRA-based brain-tuning of wav2vec2.0-base, a Temporal Brain Transformer with learned region-specific attention, and a Shared–Idiosyncratic Decomposition readout. Evaluation on 324 unseen participants across multiple datasets reports accurate zero-shot prediction across auditory and language-selective regions, with further few-shot improvement using as little as 10 minutes of participant-specific data [2607.05171].

## 5. Benchmarking, interpretability, and contested performance

Evaluation practice is now broad enough to support explicit benchmarking. For electrical brain signals, Brain4FMs provides an open platform that integrates 15 representative BFMs and 18 public datasets across 11 tasks, using cross-subject protocols and metrics such as accuracy, AUROC, F1, F2, macro-F1, and Cohen’s $\kappa$ [2602.11558]. Its comparative analysis reports that generative AE-based BFMs consistently outperform contrastive methods on several clinical diagnosis tasks, that CPC outperforms generic augmentation contrast within the contrastive family, and that bidirectional autoencoding outperforms unidirectional autoregressive modeling for classification [2602.11558].

Interpretability is treated as a defining rather than peripheral property in much of this literature. The Cam-CAN modular fMRI model visualizes expert selection frequencies across ROIs to expose expert specialization [2508.16597]. BrainHarmonix analyzes hub-token attention and reports that among 128 hub tokens, 93 attend only to fMRI, 30 only to T1, and 5 to both modalities [2509.24693]. BrainFIBRE uses expert-weight distributions, saliency maps, and CCA alignment to show task- and cohort-specific interaction patterns among uniqueness, redundancy, and synergy experts [2607.00573]. In EEG-based cognitive-load monitoring, Partition SHAP highlights consistent emphasis on prefrontal and dorsolateral prefrontal regions, with longitudinal shifts interpreted as learning progression [2601.21965]. A different line of work studies a neural foundation model through decoding and encoding manifolds, concluding that encoder, recurrent, and readout modules exhibit qualitatively different representational structures [2512.07869].

At the same time, benchmark results do not support an unqualified scaling narrative. A direct challenge comes from the argument that current fMRI BFMs suffer a “variance allocation problem.” Across three state-of-the-art BFMs and multiple readouts, cognition is predicted worse than by linear regression on functional connectivity, with the gap widening at larger scale: under nested 20×10 CV, raw FC reaches approximately $r\approx0.31$–$0.39$ on AOMIC and HCP, whereas BrainLM-111M, BrainLM-650M, and Brain-JEPA do not exceed approximately $r\approx0.18$ on AOMIC or $r\approx0.03$ on HCP. The same work attributes the failure to partial preservation of second-order covariance and destruction of third-order co-skewness, and shows that a Tucker-HOSVD “FC-Tucker” pipeline exceeds raw FC and every pretrained BFM tested [2606.04010].

A common misconception is therefore that a BFM is simply a larger conventional neural model applied to brain data. The literature instead suggests that performance depends on which statistical structure the pretraining objective preserves, how modality heterogeneity is handled, and whether the model supports biologically or clinically meaningful decomposition.

## 6. Open problems, governance, and future directions

Technical reviews converge on several unresolved problems: data scarcity and heterogeneity, inter-subject variability, noise robustness, interpretability, scalability, and the lack of unified evaluation outside a few dominant benchmarks such as BraTS and VQA-RAD [2506.16009] [2506.13306]. Brain imaging reviews additionally emphasize fragmented public access, dataset duplication and leakage risk, under-representation of PET and rare MRI sequences, and limited bias auditing: only six BFMs in one review explicitly address demographic or technical bias, and only seven include human expert validation [2506.13306].

Model-specific papers sharpen these concerns. The modular Cam-CAN framework notes that current pretraining is limited to a single dataset and that HRF variability is mitigated but not fully modeled; proposed directions include multi-site, multi-modal scaling, explicit deconvolution, and voxel-wise or surface-based representations [2508.16597]. BrainFM-MRI frames missing and unseen modalities as a central practical constraint in MRI [2511.03014]. BrainHarmonix treats heterogeneous TRs as a major limitation of existing models and addresses them via Temporal Adaptive Patch Embedding [2509.24693]. Brain4FMs identifies few-shot, zero-shot, instruction-tuned, and in-context electrical-signal decoding as open benchmarking targets [2602.11558].

Governance problems are equally central. Hanley et al. argue that BFMs extend the foundation-model paradigm into a training-data ecosystem composed of public clinical and research archives, proprietary commercial neurotechnology streams, and large-scale cross-context “stitching” between them. They organize the resulting concerns into privacy, consent, bias, benefit sharing, and governance, and recommend membership-inference testing, enhanced consent language, restrictive defaults where consent regimes conflict, standardized dataset and model documentation, tiered access policies, and participatory governance [2602.02511]. The proposal to train foundation models directly on brain-generated data makes the same issues more acute by emphasizing privacy and consent, freedom of thought, and equity in access to expensive neuroimaging resources [2601.12053].

A plausible implication is that the maturation of BFMs will depend on two coupled trajectories. The first is technical: multimodal pretraining, explicit handling of missing data and higher-order statistics, and evaluation regimes that compare BFMs against strong domain-specific baselines rather than only against other large models. The second is institutional: governance frameworks that are appropriate for body-derived data collected under clinical and research norms rather than for text or image corpora. Current literature presents BFMs not as a settled endpoint, but as a rapidly diversifying research program whose scientific and practical value will depend on how these two trajectories are coordinated.

Source: https://www.emergentmind.com/topics/brain-foundation-model-bfm