OmniBrain: Unified Multimodal Modeling

Updated 7 July 2026

OmniBrain is a research initiative that unifies brain imaging, sensory input, and cognitive signals into a single multimodal framework with shared token spaces.
It employs techniques like universal brain encoding, discrete tokenization, and cross-attention fusion to enable robust transfer across subjects and tasks.
Applications include advanced Alzheimer’s diagnosis, functional brain mapping, and closed-loop neural intervention with demonstrated improvements over traditional methods.

OmniBrain is a contemporary research label applied to several partially overlapping efforts to build unified models of brain activity, sensory input, and cognition. In one explicit usage, it denotes a multimodal framework for Alzheimer’s disease diagnosis that integrates brain MRI, radiomics, gene expression, and clinical data within a single cross-attentive model (Sharshar et al., 28 Jul 2025). In a broader systems sense, the label refers to architectures that aim to treat neural signals as one modality among many, using shared token spaces, multimodal fusion, transfer across subjects or datasets, and task unification across encoding, decoding, reasoning, or control (Beliy et al., 2024, Wu et al., 29 Jun 2026, Lu et al., 28 May 2026, Tang et al., 25 Feb 2026, Guo et al., 26 Feb 2026). The term also has an older conceptual backdrop in work that described digital information infrastructure as an “emerging scholarly brain” for a human super-organism (Kurtz, 2010).

1. Terminological scope and conceptual framing

Within neuroscience and machine learning, OmniBrain does not denote a single canonical architecture. Instead, it names a family of ambitions: universal brain encoding across subjects and scanners; unified brain–vision–language modeling; multimodal clinical decision systems; and omnifunctional foundation models spanning fMRI, EEG, and MEG. This suggests that OmniBrain is best understood as a research program centered on shared representational spaces, many-to-many modality mappings, and cross-task reuse rather than as one fixed model specification (Beliy et al., 2024, Wu et al., 29 Jun 2026, Guo et al., 26 Feb 2026).

A central distinction in this literature is between systems that treat brain data as a prediction target and systems that treat brain data as a first-class modality. The former include universal image-to-fMRI encoders that predict voxel responses across many subjects and datasets (Beliy et al., 2024). The latter include tokenized or LLM-centered frameworks in which brain signals, images, audio, and text are all mapped into a shared sequence or embedding space and processed by one generative or discriminative backbone (Wu et al., 29 Jun 2026, Tang et al., 25 Feb 2026, Lu et al., 28 May 2026). A third strand uses OmniBrain in an explicitly clinical sense, emphasizing robustness to missing modalities, cross-dataset generalization, and interpretability in diagnosis (Sharshar et al., 28 Jul 2025).

2. Universal brain encoding across subjects, datasets, and scanners

A direct precursor to OmniBrain on the encoding side is the “Universal Brain-Encoder,” a single image-to-fMRI model trained jointly on many subjects, datasets, and machines rather than per subject and per dataset (Beliy et al., 2024). Its core formulation is a scalar-valued function $\hat{r}_v(x) = f_\theta(x, v)$ that takes an image $x$ and a voxel identifier $v$ and predicts the activation of that voxel. The architecture uses a DINOv2 ViT-G/14 backbone, LoRA-style low-rank adaptation on the self-attention output projection weights $W_o$ , a learned embedding $e_v \in \mathbb{R}^{256}$ for every voxel, and a voxel-centric cross-attention mechanism that performs spatial attention over image patches and functional attention over flattened multi-layer features (Beliy et al., 2024).

The model is trained on three heterogeneous datasets: NSD at 7T, fMRI-on-ImageNet at 3T, and VIM-1 at 4T, without requiring anatomical alignment or shared training images across subjects. Shared parameters include the DINOv2 backbone, layer projections, spatial attention parameters, layer-specific MLPs, and the feature embedding matrix $K$ ; the only subject-specific parameters are the voxel embeddings. Training uses batch size 32 images, 5000 randomly sampled voxels per image, Adam with learning rate $1 \times 10^{-3}$ , and a combined MSE plus cosine proximity loss with $\alpha = 0.1$ (Beliy et al., 2024).

Empirically, multi-subject training improves every NSD subject relative to both a prior subject-specific baseline and a single-subject version of the same universal architecture. On NSD Subject 1, the average rank of the correct image out of 1000 is 1.85, and transfer to unseen subjects or datasets is markedly sample-efficient: for NSD Subject 1, transfer learning achieves more than 77% improvement over from-scratch training for any number of examples, and with only about 100 images it outperforms a from-scratch model trained on the full 9000 images; transfer from NSD yields more than 84% improvement on fMRI-on-ImageNet and about 45% improvement on VIM-1 (Beliy et al., 2024). The learned voxel embeddings also cluster across subjects into semantically selective groups for food, faces, text, and outdoor scenes, suggesting that shared functional organization can emerge without explicit anatomical correspondence (Beliy et al., 2024).

3. Unified brain–vision–language and brain foundation models

Several recent systems move beyond universal encoding toward full multimodal unification. They differ in modality coverage and learning objective, but they converge on a small set of design choices: discrete or tokenized brain representations, shared semantic spaces, and one backbone reused across many tasks.

System	Modalities / scope	Core mechanism
BrainJanus	brain, vision, language	Unified Brain Tokenizer, shared Omni space, all-in-one autoregressive next-token prediction
Mind-Omni	brain, vision, language	Brain Tokenizer plus discrete diffusion over seven encoding and decoding tasks
NOBEL	EEG, MEG, fMRI, stimuli, text	unified EEG/MEG encoder, dual-path fMRI, LLM semantic space
Brain-OF	fMRI, EEG, MEG	ARNESS, DINT attention, Sparse Mixture of Experts, Masked Temporal-Frequency Modeling
UMind	EEG/MEG, image, text	multimodal alignment with CLIP spaces, zero-shot retrieval, classification, reconstruction

BrainJanus introduces a Unified Brain Tokenizer that quantizes continuous neural dynamics into discrete tokens, aligns them with visual and linguistic tokens in a shared Omni space, and trains one autoregressive Transformer by next-token prediction to support image-to-brain and text-to-brain encoding together with brain-to-image and brain-to-text decoding (Wu et al., 29 Jun 2026). Mind-Omni follows a closely related tri-modal strategy but replaces autoregressive modeling with discrete diffusion, unifying seven tasks—Image $\rightarrow$ Brain, Text $\rightarrow$ Brain, Image+Text $x$ 0Brain, Brain $x$ 1Image, Brain $x$ 2Text, Brain $x$ 3Image+Text, and Brain Question Answering—within one conditional discrete diffusion objective (Lu et al., 28 May 2026). Both systems center the problem on a discrete brain tokenizer rather than on a thin adapter into an external vision prior.

NOBEL extends the unification problem to non-invasive multimodal decoding by combining EEG, MEG, fMRI, and external sensory stimuli within the semantic embedding space of Qwen2.5-Omni-7B (Tang et al., 25 Feb 2026). Its architecture uses a unified EEG/MEG encoder derived from BrainOmni, a dual-path fMRI strategy that separates a task-agnostic voxel path from a task-specific beta path, and an LLM backbone that performs brain-to-text generation and stimulus-aware verification. On paired EEG/MEG–fMRI settings, joint modeling improves over the stronger unimodal baseline, reaching 81.8% on LEMON, 92.8% on Nat-View, and 78.5% on SMN4Lang (Tang et al., 25 Feb 2026).

Brain-OF addresses the same multimodal ambition from a foundation-model perspective rather than an LLM-decoding perspective. It is jointly pretrained on fMRI, EEG, and MEG using the Any-Resolution Neural Signal Sampler, DINT attention, a Sparse Mixture of Experts, and Masked Temporal-Frequency Modeling across about 40 datasets (Guo et al., 26 Feb 2026). UMind, by contrast, focuses on zero-shot M/EEG visual decoding, using a shared brain encoder aligned to image and text spaces so that one network supports retrieval, classification, and reconstruction from EEG or MEG (Xu et al., 18 Sep 2025). Taken together, these systems suggest that OmniBrain increasingly denotes not merely multimodal input handling but a shift toward brain signals as native tokens inside foundation-model pipelines.

4. Topography, functional specialization, and closed-loop control

A distinct line of work asks whether an OmniBrain should also be spatially organized like cortex. Topo-Omni answers this by taking Qwen2.5-Omni and forcing visual, auditory, and language/cognitive processing to inhabit a single contiguous in-silico sheet, with a spatial smoothness loss weighted by $x$ 4 during fine-tuning (AlKhamissi et al., 8 Jun 2026). Nearby units are encouraged to have similar response profiles, and the resulting sheet develops smooth retinotopy and tonotopy together with localized clusters for faces, objects, scenes, words, speech, voices, language, multiple demand, and theory of mind (AlKhamissi et al., 8 Jun 2026).

The model’s visual clusters closely match human neuroimaging localizers. Its face cluster shows faces greater than all other categories with $x$ 5, $x$ 6, $x$ 7, and its 10-condition response profile correlates with human FFA at $x$ 8, $x$ 9; its object/LOC-like cluster correlates with human LOC at $v$ 0, $v$ 1, and its speech-selective auditory cluster correlates with human STG at $v$ 2, $v$ 3 (AlKhamissi et al., 8 Jun 2026). The same framework can be causally manipulated: driving a face-selective cluster biases perception toward faces, whereas suppressing it collapses face identification while leaving other categories relatively preserved (AlKhamissi et al., 8 Jun 2026). The model is also used to discover novel animal and natural landscape networks in silico and validate them in human data (AlKhamissi et al., 8 Jun 2026).

Closed-loop intervention appears in a more literal form in OMiSO, which treats the OmniBrain problem as online control of neural population states rather than as passive decoding (Minai et al., 10 Jul 2025). OMiSO uses Factor Analysis to define a low-dimensional latent state space from macaque PFC recordings, models stimulation responses with a CNN-plus-LSTM forward model, learns an inverse policy by behavioral cloning, and then adapts the stimulation policy online with PPO while conditioning on the pre-stimulation state. In 14 test sessions, OMiSO produces significantly smaller errors to the target latent state than both a static state-dependent controller and MiSO, with Wilcoxon signed-rank $v$ 4 and $v$ 5 respectively (Minai et al., 10 Jul 2025). This suggests that an OmniBrain may require not only multimodal representation learning but also state-dependent, adaptive intervention policies.

5. Clinical multimodal diagnosis and benchmarked clinical reasoning

In its explicit clinical usage, OmniBrain is a multimodal deep learning framework for Alzheimer’s disease classification that integrates voxel-level grey matter maps, radiomics from 32 brain regions, gene expression, and clinical metadata through an FT-Transformer for tabular data, a foundation-model MRI encoder, and cross-attention fusion with modality-aware attention masking and modality dropout (Sharshar et al., 28 Jul 2025). The best configuration uses AnatCL for MRI and achieves $v$ 6 accuracy, $v$ 7 recall, and $v$ 8 F1-score on ANMerge, while zero-shot transfer to MRI-only ADNI reaches $v$ 9 accuracy, $W_o$ 0 recall, and $W_o$ 1 F1-score (Sharshar et al., 28 Jul 2025). Explainability is provided by Grad-CAM for MRI and SHAP for tabular features, highlighting hippocampus, parahippocampal gyrus, thalamus, MMSE, APOE, and AD-related genes (Sharshar et al., 28 Jul 2025).

Evaluation of such systems is itself becoming an OmniBrain problem. OmniBrainBench defines a multimodal VQA benchmark for brain imaging analysis with 15 modalities, 15 multi-stage clinical tasks, 9,527 validated VQA pairs, and 31,706 images drawn from 30 verified medical sources and a private hospital cohort (Peng et al., 2 Nov 2025). The benchmark spans Anatomical and Imaging Assessment, Lesion Identification and Localization, Diagnostic Synthesis and Causal Reasoning, Prognostic Judgment and Risk Forecasting, and Therapeutic Cycle Management. Across 24 MLLMs, proprietary models lead overall but still lag physicians: Gemini-2.5-Pro reaches 66.58% overall accuracy, HuatuoGPT-V-34B 63.56%, Qwen3-VL-30B 56.40%, and physician performance 91.35% (Peng et al., 2 Nov 2025). The sharpest deficits appear in Risk Stratification and Preoperative Assessment, which the benchmark characterizes as a visual-to-clinical reasoning gap (Peng et al., 2 Nov 2025).

Outside strict brain-imaging modeling, the OmniBrain idea increasingly overlaps with general omni-modal AI design. OmniBind shows how a shared representation backbone can be built by remapping and binding pretrained specialist spaces for 3D, audio, image, and language into one target space, with lightweight routers and projectors rather than end-to-end retraining (Wang et al., 2024). MiniMind-O demonstrates a small speech-native omni model with text, speech, and image inputs and both text and streaming speech outputs, organized around a Thinker–Talker split and a released multimodal sequence format (Gong, 5 May 2026). Nemotron 3 Nano Omni applies similar omni reasoning to text, images, video, and audio with token-reduction methods for long multimodal contexts (NVIDIA et al., 27 Apr 2026). O-Mem applies the idea to memory, organizing agent memory into persona, episodic, and working memory for long-horizon personalization (Wang et al., 17 Nov 2025). A plausible implication is that OmniBrain is becoming a cross-domain shorthand for systems that unify heterogeneous channels under one representational and inferential substrate.

The most persistent technical motifs across these uses are shared token or embedding spaces, parameter-efficient adaptation, robustness to missing modalities, and explicit mechanisms for transfer or composition. In brain-focused models, those motifs appear as voxel embeddings and cross-attention (Beliy et al., 2024), discrete brain tokenizers and any-to-any generation (Wu et al., 29 Jun 2026, Lu et al., 28 May 2026), LLM-aligned non-invasive signal encoders (Tang et al., 25 Feb 2026), and any-resolution latent samplers with dual-domain self-supervision (Guo et al., 26 Feb 2026). In more speculative or systems-level accounts, the same direction appears as the idea of an “emerging scholarly brain,” in which communication networks and digital memory form a collective cognitive substrate (Kurtz, 2010). Taken together, the literature suggests that OmniBrain is evolving from a metaphor of total integration into a technical agenda: unify subjects, unite modalities, preserve functional organization, support intervention, and expose clinically or scientifically useful abstractions from heterogeneous neural data.