Unified Brain Tokenizer: Cross-Modality Interface

• Unified brain tokenizer is a module that converts heterogeneous neural measurements into a standardized token space, enabling cross-task, cross-subject, and cross-device compatibility.
• It employs various tokenization strategies including discrete time–frequency motifs for EEG, VQ-VAE code indices for fMRI, and continuous hub tokens for structural imaging.
• Integration with foundation models using masked modeling, next-token prediction, and diffusion demonstrates its scalability and effective transfer across multiple modalities.

A unified brain tokenizer is a representation module that maps raw neural measurements into a standardized token space for downstream foundation models. In current usage, the term spans several related constructions: discrete EEG motif vocabularies derived from single-channel time–frequency structure, discrete fMRI codebook indices aligned with image and text tokens, semantically aligned fMRI tokens for discrete diffusion, device-agnostic EEG–MEG token sequences, and, in some multimodal neuroimaging systems, compact continuous 1D hub tokens rather than discrete IDs (Pradeepkumar et al., 22 Feb 2025, Wu et al., 29 Jun 2026, Lu et al., 28 May 2026, Dong et al., 29 Sep 2025, Xiao et al., 18 May 2025, Puyvelde et al., 17 Jun 2026). The unifying idea is not a single architecture but a common interface: raw channels, voxels, or volumes are converted into a tokenized latent representation that can be reused across tasks, subjects, datasets, devices, or modalities.

1. Conceptual scope and definitional variants

Recent papers use “unified brain tokenizer” to denote a module that standardizes heterogeneous brain data before transformer-style modeling. In EEG, it is explicitly described as a module that converts raw neural time series into discrete symbols that can be processed by transformer-style models, in analogy to subword tokenizers for text (Pradeepkumar et al., 22 Feb 2025). In brain–vision–LLMs, it is the component that turns continuous neural recordings into discrete tokens aligned with visual and linguistic representations in a shared space (Wu et al., 29 Jun 2026, Lu et al., 28 May 2026). In multimodal neuroimaging, the same term can refer to continuous-valued 1D tokens that compress structural and functional data into a shared latent sequence (Dong et al., 29 Sep 2025).

This variation matters because “unified” can refer to at least four distinct axes: cross-task reuse, cross-subject transfer, cross-device compatibility, and cross-modality alignment. UMBRAE, for example, uses subject-specific tokenizers plus a shared universal encoder to align fMRI to CLIP’s patchwise visual space rather than a discrete vocabulary (Xia et al., 2024). BrainG3N uses a single frozen 3D MAE encoder and a projected token space for both clinical probing and controllable generation, again without discrete code indices as the primary interface (Puyvelde et al., 17 Jun 2026).

System	Neural input	Token form
TFM-Tokenizer	Single-channel EEG	Discrete time–frequency motif tokens
BrainJanus / Mind-Omni	fMRI	Discrete VQ-VAE codebook indices
BrainOmni	EEG and MEG	Discrete RVQ tokens from latent source variables
BrainHarmonix	T1 MRI and fMRI	Continuous 1D brain hub tokens
UMBRAE	fMRI	Continuous CLIP-aligned brain token sequence
BrainG3N	3D brain MRI	Projected MAE tokens for probing and diffusion

A common misconception is that a unified brain tokenizer must always be a single discrete vocabulary shared by all neural modalities. The literature does not support that restriction. Some systems are fully discrete and codebook-based (Pradeepkumar et al., 22 Feb 2025, Wu et al., 29 Jun 2026, Lu et al., 28 May 2026, Xiao et al., 18 May 2025), whereas others unify modalities through continuous latent tokens or CLIP-aligned token sequences (Dong et al., 29 Sep 2025, Xia et al., 2024, Puyvelde et al., 17 Jun 2026).

2. EEG and MEG tokenization as motif or source discretization

In EEG, one line of work formulates tokenization as discrete time–frequency motif discovery. TFM-Tokenizer defines single-channel EEG tokenization as learning a map from $x \in \mathbb{R}^{T}$ to a sequence of codebook vectors, one token per short time window, where the tokens represent time–frequency motifs such as a spike, spindle, or rhythmic delta burst (Pradeepkumar et al., 22 Feb 2025). The implementation uses STFT magnitude with sampling rate 200 Hz, $n_{\text{fft}} = L = 200$, hop length $H = 100$, Hann window, one-sided output, and magnitude only. Temporal patches are aligned one-to-one with STFT columns, and the tokenizer combines a localized spectral window encoder, a temporal CNN, a temporal transformer, and vector quantization with a typical vocabulary size $k = 8192$ (Pradeepkumar et al., 22 Feb 2025). A notable design choice is the removal of positional encoding inside the tokenizer so that the same motif is not assigned different tokens at different absolute times (Pradeepkumar et al., 22 Feb 2025).

BrainOmni generalizes tokenization across EEG and MEG by making the sensor array itself part of the tokenizer. Its BrainTokenizer accepts raw signals $\mathbf{X} \in \mathbb{R}^{C \times T}$, sensor positions and orientations $\mathbf{L} \in \mathbb{R}^{C \times 6}$, and sensor types $$s_i \in \{0\;(\text{EEG}), 1\;(\text{GRAD}), 2\;(\text{MAG})\}$$ (Xiao et al., 18 May 2025). A SEANet temporal encoder extracts per-sensor temporal features, a Sensor Encoder maps geometry and type into learned embeddings, and a cross-attention layer compresses arbitrary sensor arrays into $C' = 16$ latent source variables before residual vector quantization with four quantizer layers and codebook size 512 (Xiao et al., 18 May 2025). The tokenizer is trained with a composite loss consisting of time-domain reconstruction, frequency-domain amplitude and phase losses, a PCC-based trend consistency term, and RVQ commitment loss (Xiao et al., 18 May 2025). This design explicitly targets cross-device and cross-modality compatibility.

NeuroRVQ extends EEG tokenization toward high-fidelity generative modeling. Its tokenizer combines four temporal branches with kernel pairs $(21,9)$, $(15,7)$, $n_{\text{fft}} = L = 200$0, and $n_{\text{fft}} = L = 200$1, followed by shared transformer layers and hierarchical RVQ codebooks (Barmpas et al., 15 Oct 2025). The final configuration uses $n_{\text{fft}} = L = 200$2 scales, eight codebooks per scale, $n_{\text{fft}} = L = 200$3 entries per codebook, and quantization dimension $n_{\text{fft}} = L = 200$4 (Barmpas et al., 15 Oct 2025). Training couples log-amplitude reconstruction, unit-circle-aware phase loss, time-domain reconstruction, and VQ loss, explicitly to preserve high-frequency dynamics that earlier EEG tokenizers underrepresented (Barmpas et al., 15 Oct 2025). This suggests that multi-scale temporal decomposition and residual codebooks are emerging as core design patterns for brainwave tokenizers.

3. fMRI and structural neuroimaging tokenizers

For fMRI, BrainJanus introduces a Unified Brain Tokenizer as a VQ-VAE that converts a trial’s beta-weight vector in visual cortex ROIs into a sequence of discrete codebook indices (Wu et al., 29 Jun 2026). The tokenizer uses a codebook $n_{\text{fft}} = L = 200$5 with $n_{\text{fft}} = L = 200$6 and embedding dimension $n_{\text{fft}} = L = 200$7, and the main experiments use a compression ratio of $n_{\text{fft}} = L = 200$8 (Wu et al., 29 Jun 2026). Quantization follows nearest-neighbor lookup,

$n_{\text{fft}} = L = 200$9

with a VQ-VAE loss combining reconstruction, codebook loss, commitment loss with $H = 100$0, and entropy regularization ratio 0.1 (Wu et al., 29 Jun 2026). The resulting brain token sequence lives in the same Omni space as image and text tokens, enabling a single autoregressive model over mixed-modality sequences (Wu et al., 29 Jun 2026).

Mind-Omni also tokenizes fMRI with a VQ-VAE, but adds explicit semantic alignment. Its Brain Tokenizer processes single-trial fMRI responses registered to MNI152 space, using an encoder $H = 100$1 built from a 1D CNN and MLP, a codebook $H = 100$2, and a symmetric MLP decoder (Lu et al., 28 May 2026). The final tokenizer uses codebook size $H = 100$3, code dimension $H = 100$4, number of tokens per sample $H = 100$5, EMA decay 0.99, and commitment weight $H = 100$6 (Lu et al., 28 May 2026). What distinguishes it is the addition of coarse-grained InfoNCE alignment between pooled brain tokens and CLIP image/text features, fine-grained token-level alignment via cross-attention to masked text tokens, and a perceptual loss that requires reconstructed fMRI to remain CLIP-decodable (Lu et al., 28 May 2026). In this formulation, unification is not just compression but semantic co-location with visual and linguistic feature spaces.

Not all neuroimaging tokenizers in this area are discrete. BrainHarmonix maps structural MRI and fMRI into shared continuous 1D hub tokens $H = 100$7, with $H = 100$8 and $H = 100$9 in the ViT-B configuration (Dong et al., 29 Sep 2025). Structural MRI is patchified into $k = 8192$0 tokens, while fMRI is tokenized with Temporal Adaptive Patch Embedding so that each temporal token corresponds to a constant physical duration $k = 8192$1 s despite heterogeneous repetition times (Dong et al., 29 Sep 2025). BrainG3N similarly adopts a non-discrete formulation: a frozen 3D MAE encoder produces $k = 8192$2 tokens from a $k = 8192$3 brain MRI volume, those tokens are projected to $k = 8192$4, and a separate CNN decoder reconstructs voxels from the projected tokens (Puyvelde et al., 17 Jun 2026). These systems indicate that the tokenizer concept in neuroimaging includes both discrete codebooks and compact continuous token interfaces.

4. Integration with foundation models and training objectives

Once tokenized, unified brain representations are typically paired with masked modeling, next-token prediction, or discrete diffusion. In TFM-Tokenizer, the tokenizer is pretrained and then frozen; the downstream TFM-Encoder performs embedding lookup from the learned codebook, flattens tokens across channels and time, adds channel and position embeddings, prepends a $k = 8192$5 token, and applies a four-layer transformer with linear attention (Pradeepkumar et al., 22 Feb 2025). Pretraining uses masked token prediction over the discrete vocabulary, conceptually identical to BERT-style masked language modeling, followed by downstream fine-tuning with task-specific losses (Pradeepkumar et al., 22 Feb 2025).

BrainJanus treats the tokenized brain as a first-class modality in a decoder-only Janus-7B-based autoregressive transformer. Its All-in-One architecture models

$k = 8192$6

where $k = 8192$7 may be a brain, image, or text token (Wu et al., 29 Jun 2026). Tokens from different modalities can be arbitrarily interleaved within the sequence, which allows image-to-brain and text-to-brain encoding as well as brain-to-image and brain-to-text decoding in the same next-token framework (Wu et al., 29 Jun 2026). Mind-Omni instead uses an MM-DiT discrete diffusion backbone, with one unified objective and seven tasks: I→B, T→B, I+T→B, B→I, B→T, B→I+T, and BQA (Lu et al., 28 May 2026). In that formulation, the brain tokenizer makes neural activity directly manipulable by the same absorbing-state discrete diffusion process used for image and text tokens (Lu et al., 28 May 2026).

BrainOmni follows a two-stage recipe closer to HuBERT-style self-supervision. Stage 1 trains BrainTokenizer as an autoencoder over 2 s segments; Stage 2 freezes it and trains a Criss-Cross Transformer on 30 s token grids with 50% masking, predicting codebook indices with cross-entropy over the RVQ vocabularies (Xiao et al., 18 May 2025). The transformer explicitly separates spatial attention across latent sources from temporal attention across token time steps, matching the $k = 8192$8 token grid produced by the tokenizer (Xiao et al., 18 May 2025).

Continuous-token approaches integrate differently. UMBRAE maps fMRI vectors into a fixed-length sequence of brain tokens, aligns them to CLIP’s $k = 8192$9 patch grid by MSE in feature space, and then reuses frozen Shikra or LLaVA adapters and Vicuna LLMs (Xia et al., 2024). BrainG3N uses the same frozen token space for linear probes, a conditional diffusion transformer trained on projected tokens with flow matching, and a longitudinal DiT that maps baseline tokens to future tokens through a Brownian-bridge-style interpolant (Puyvelde et al., 17 Jun 2026). This suggests that unification can occur either at the level of discrete symbolic prediction or at the level of a stable latent token geometry reused by multiple downstream optimizers.

5. Empirical properties: accuracy, interpretability, and generalization

The strongest quantitative evidence for unified brain tokenizers comes from cross-task and cross-domain reuse. TFM-Tokenizer reports experiments across four EEG datasets. On TUEV, the full TFM model reaches balanced accuracy $\mathbf{X} \in \mathbb{R}^{C \times T}$0, Cohen’s Kappa $\mathbf{X} \in \mathbb{R}^{C \times T}$1, and weighted F1 $\mathbf{X} \in \mathbb{R}^{C \times T}$2, improving balanced accuracy and Kappa by about 5% over the best baselines (Pradeepkumar et al., 22 Feb 2025). On TUAB it reaches balanced accuracy $\mathbf{X} \in \mathbb{R}^{C \times T}$3, AUC-PR $\mathbf{X} \in \mathbb{R}^{C \times T}$4, and AUROC $\mathbf{X} \in \mathbb{R}^{C \times T}$5, while using 1.9M parameters compared with 5.8M for the LaBraM classifier (Pradeepkumar et al., 22 Feb 2025). Its learned tokens are also more compact and class-specific than LaBraM’s neural tokenizer: on TUEV, token utilization is 9.78% and class-token uniqueness is 2.14%, versus 21.13% and 0.034% for the neural tokenizer (Pradeepkumar et al., 22 Feb 2025).

BrainOmni aggregates 1,997 hours of EEG and 656 hours of MEG for pretraining and reports that BrainOmni outperforms both existing foundation models and state-of-the-art task-specific models on a range of downstream tasks (Xiao et al., 18 May 2025). It also demonstrates strong generalization to unseen EEG and MEG devices, and joint EEG-MEG training yields consistent improvements across both modalities (Xiao et al., 18 May 2025). The tokenizer ablations show that removing the Sensor Encoder degrades TUAB, ASD74, and SomatoMotor performance, indicating that sensor geometry and type are not auxiliary metadata but part of the tokenization itself (Xiao et al., 18 May 2025).

For brain–vision–language settings, BrainJanus reports superior performance across diverse benchmarks, zero-shot generalization, and preservation of interpretable biological topography (Wu et al., 29 Jun 2026). Mind-Omni provides direct evidence that semantic alignment changes token quality: without $\mathbf{X} \in \mathbb{R}^{C \times T}$6, codebook usage falls to about 30–40% and retrieval is approximately chance, whereas adding semantic alignment raises self-reconstruction rPCC from 0.43 to 0.64, retrieval Top-50 to about 0.6, and codebook usage to about 80–100% (Lu et al., 28 May 2026). This is a substantive result because it shows that a brain tokenizer can collapse if it is optimized only for reconstruction, even when it is discrete.

Continuous-token systems also show strong transfer. BrainHarmonix reports that linear probing over fused hub tokens already surpasses previous approaches on ABIDE-II and ADHD-200, and the multimodal model reaches Spearman $\mathbf{X} \in \mathbb{R}^{C \times T}$7 on HCP-A cognition prediction (Dong et al., 29 Sep 2025). BrainG3N shows that a single 3D brain-MRI embedding space can support 23-task linear probing and conditional generation: the frozen encoder outperforms or matches BrainIAC, BrainSegFounder, and MedicalNet on 21 of 23 tasks, reaches IDH1 AUC $\mathbf{X} \in \mathbb{R}^{C \times T}$8, and achieves brain-age MAE $\mathbf{X} \in \mathbb{R}^{C \times T}$9 years (Puyvelde et al., 17 Jun 2026). UMBRAE shows the same pattern in cross-subject fMRI decoding, with forward retrieval 94.2%, backward retrieval 91.3%, and exemplar retrieval 93.8% while using a single universal encoder across subjects (Xia et al., 2024).

6. Limitations, misconceptions, and future directions

The central limitation is that current “unified” tokenizers are usually unified only along selected axes. BrainJanus and Mind-Omni are limited to visual-cortex fMRI in NSD-like settings; BrainOmni covers EEG and MEG but not invasive recordings or hemodynamic modalities; TFM-Tokenizer is single-channel EEG and dataset-specific; BrainHarmonix unifies T1 and fMRI but not electrophysiology; UMBRAE is confined to NSD fMRI aligned with COCO imagery; BrainG3N is 3D brain MRI only (Wu et al., 29 Jun 2026, Lu et al., 28 May 2026, Xiao et al., 18 May 2025, Pradeepkumar et al., 22 Feb 2025, Dong et al., 29 Sep 2025, Xia et al., 2024, Puyvelde et al., 17 Jun 2026). This suggests that a truly modality-general tokenizer remains an open problem.

A second misconception is that tokenization and interpretability are automatically aligned. TFM-Tokenizer provides clinically plausible motif visualizations, class-token uniqueness, and retrieval analysis, but it also notes a fixed-window failure mode: if a motif spans two windows, the tokenizer may split it into different tokens (Pradeepkumar et al., 22 Feb 2025). BrainJanus notes that discretization implies information loss and that generative priors may prioritize perceptual plausibility over strict biological fidelity (Wu et al., 29 Jun 2026). BrainG3N shows that its frozen embeddings strongly encode acquisition variables—site AUC 0.998, field strength AUC 0.986, vendor AUC 0.993—which means a unified token space can preserve confounds as effectively as clinical content (Puyvelde et al., 17 Jun 2026).

Method-specific limitations are also recurrent. TFM-Tokenizer uses STFT magnitude only and ignores phase; the paper explicitly notes that some modalities or tasks might benefit from phase and that training is dataset-specific rather than cross-dataset (Pradeepkumar et al., 22 Feb 2025). Mind-Omni acknowledges that experiments are limited to fMRI and NSD, that voxel-level fidelity is lower than some CLIP-based baselines, and that its encoders are designed for generation rather than maximum brain alignment (Lu et al., 28 May 2026). BrainOmni states that, although relatively large for EEG/MEG, its corpus is still small compared with text or speech, and MEG downstream benchmarks remain scarce (Xiao et al., 18 May 2025). BrainHarmonix identifies frozen unimodal encoders during fusion, parcellation dependence, and partial opacity of hub-token meaning as current constraints (Dong et al., 29 Sep 2025).

The forward direction is correspondingly clear. TFM-Tokenizer explicitly proposes training a single tokenizer across many datasets and modalities, using modality-specific channel embeddings with a shared vocabulary, multi-resolution time–frequency transforms, hierarchical or product quantization, next-token prediction, cross-modal predictive tasks, dynamic segmentation, and phase-aware extensions (Pradeepkumar et al., 22 Feb 2025). BrainOmni suggests that the same sensor-aware tokenization principles can be extended to fNIRS, ECoG, and iEEG (Xiao et al., 18 May 2025). BrainHarmonix points toward additional modalities such as DTI, EEG, and MEG in the same hub-token space (Dong et al., 29 Sep 2025). Taken together, these proposals indicate that the most plausible future unified brain tokenizer is likely to be multi-resolution, sensor-aware, partially modality-agnostic, and designed to preserve both reconstructability and semantic compatibility rather than optimizing only one of those objectives.