--- title: 'Unified Brain Tokenizer: Cross-Modality Interface' url: https://www.emergentmind.com/topics/unified-brain-tokenizer type: topic --- # Unified Brain Tokenizer: Cross-Modality Interface A unified brain tokenizer is a representation module that maps raw neural measurements into a standardized token space for downstream foundation models. In current usage, the term spans several related constructions: discrete EEG motif vocabularies derived from single-channel time–frequency structure, discrete fMRI codebook indices aligned with image and text tokens, semantically aligned fMRI tokens for discrete diffusion, device-agnostic EEG–MEG token sequences, and, in some multimodal neuroimaging systems, compact continuous 1D hub tokens rather than discrete IDs [2502.16060][2606.30319][2605.29591][2509.24693][2505.18185][2606.19651]. The unifying idea is not a single architecture but a common interface: raw channels, voxels, or volumes are converted into a tokenized latent representation that can be reused across tasks, subjects, datasets, devices, or modalities. ## 1. Conceptual scope and definitional variants Recent papers use “unified brain tokenizer” to denote a module that standardizes heterogeneous brain data before transformer-style modeling. In EEG, it is explicitly described as a module that converts raw neural time series into discrete symbols that can be processed by transformer-style models, in analogy to subword tokenizers for text [2502.16060]. In brain–vision–language models, it is the component that turns continuous neural recordings into discrete tokens aligned with visual and linguistic representations in a shared space [2606.30319][2605.29591]. In multimodal neuroimaging, the same term can refer to continuous-valued 1D tokens that compress structural and functional data into a shared latent sequence [2509.24693]. This variation matters because “unified” can refer to at least four distinct axes: cross-task reuse, cross-subject transfer, cross-device compatibility, and cross-modality alignment. UMBRAE, for example, uses subject-specific tokenizers plus a shared universal encoder to align fMRI to CLIP’s patchwise visual space rather than a discrete vocabulary [2404.07202]. BrainG3N uses a single frozen 3D MAE encoder and a projected token space for both clinical probing and controllable generation, again without discrete code indices as the primary interface [2606.19651]. | System | Neural input | Token form | |---|---|---| | TFM-Tokenizer | Single-channel EEG | Discrete time–frequency motif tokens | | BrainJanus / Mind-Omni | fMRI | Discrete VQ-VAE codebook indices | | BrainOmni | EEG and MEG | Discrete RVQ tokens from latent source variables | | BrainHarmonix | T1 MRI and fMRI | Continuous 1D brain hub tokens | | UMBRAE | fMRI | Continuous CLIP-aligned brain token sequence | | BrainG3N | 3D brain MRI | Projected MAE tokens for probing and diffusion | A common misconception is that a unified brain tokenizer must always be a single discrete vocabulary shared by all neural modalities. The literature does not support that restriction. Some systems are fully discrete and codebook-based [2502.16060][2606.30319][2605.29591][2505.18185], whereas others unify modalities through continuous latent tokens or CLIP-aligned token sequences [2509.24693][2404.07202][2606.19651]. ## 2. EEG and MEG tokenization as motif or source discretization In EEG, one line of work formulates tokenization as discrete time–frequency motif discovery. TFM-Tokenizer defines single-channel EEG tokenization as learning a map from \(x \in \mathbb{R}^{T}\) to a sequence of codebook vectors, one token per short time window, where the tokens represent time–frequency motifs such as a spike, spindle, or rhythmic delta burst [2502.16060]. The implementation uses STFT magnitude with sampling rate 200 Hz, \(n_{\text{fft}} = L = 200\), hop length \(H = 100\), Hann window, one-sided output, and magnitude only. Temporal patches are aligned one-to-one with STFT columns, and the tokenizer combines a localized spectral window encoder, a temporal CNN, a temporal transformer, and vector quantization with a typical vocabulary size \(k = 8192\) [2502.16060]. A notable design choice is the removal of positional encoding inside the tokenizer so that the same motif is not assigned different tokens at different absolute times [2502.16060]. BrainOmni generalizes tokenization across EEG and MEG by making the sensor array itself part of the tokenizer. Its BrainTokenizer accepts raw signals \( \mathbf{X} \in \mathbb{R}^{C \times T} \), sensor positions and orientations \( \mathbf{L} \in \mathbb{R}^{C \times 6} \), and sensor types \(s_i \in \{0\;(\text{EEG}), 1\;(\text{GRAD}), 2\;(\text{MAG})\}\) [2505.18185]. A SEANet temporal encoder extracts per-sensor temporal features, a Sensor Encoder maps geometry and type into learned embeddings, and a cross-attention layer compresses arbitrary sensor arrays into \(C' = 16\) latent source variables before residual vector quantization with four quantizer layers and codebook size 512 [2505.18185]. The tokenizer is trained with a composite loss consisting of time-domain reconstruction, frequency-domain amplitude and phase losses, a PCC-based trend consistency term, and RVQ commitment loss [2505.18185]. This design explicitly targets cross-device and cross-modality compatibility. NeuroRVQ extends EEG tokenization toward high-fidelity generative modeling. Its tokenizer combines four temporal branches with kernel pairs \((21,9)\), \((15,7)\), \((9,5)\), and \((5,3)\), followed by shared transformer layers and hierarchical RVQ codebooks [2510.13068]. The final configuration uses \(S = 4\) scales, eight codebooks per scale, \(K = 8192\) entries per codebook, and quantization dimension \(D = 128\) [2510.13068]. Training couples log-amplitude reconstruction, unit-circle-aware phase loss, time-domain reconstruction, and VQ loss, explicitly to preserve high-frequency dynamics that earlier EEG tokenizers underrepresented [2510.13068]. This suggests that multi-scale temporal decomposition and residual codebooks are emerging as core design patterns for brainwave tokenizers. ## 3. fMRI and structural neuroimaging tokenizers For fMRI, BrainJanus introduces a Unified Brain Tokenizer as a VQ-VAE that converts a trial’s beta-weight vector in visual cortex ROIs into a sequence of discrete codebook indices [2606.30319]. The tokenizer uses a codebook \(\mathcal{C} = \{e_k\}_{k=1}^{K}\) with \(K = 128\) and embedding dimension \(d_O = 32\), and the main experiments use a compression ratio of \(1/128\) [2606.30319]. Quantization follows nearest-neighbor lookup, \[ z_q(x) = \arg\min_{c \in \mathcal{C}} \| z_e(x) - c \|_2^2, \] with a VQ-VAE loss combining reconstruction, codebook loss, commitment loss with \(\beta = 0.25\), and entropy regularization ratio 0.1 [2606.30319]. The resulting brain token sequence lives in the same Omni space as image and text tokens, enabling a single autoregressive model over mixed-modality sequences [2606.30319]. Mind-Omni also tokenizes fMRI with a VQ-VAE, but adds explicit semantic alignment. Its Brain Tokenizer processes single-trial fMRI responses registered to MNI152 space, using an encoder \(E_{\text{brain}}\) built from a 1D CNN and MLP, a codebook \(\mathcal{Z} \in \mathbb{R}^{K \times D}\), and a symmetric MLP decoder [2605.29591]. The final tokenizer uses codebook size \(K = 128\), code dimension \(D = 16\), number of tokens per sample \(L = 64\), EMA decay 0.99, and commitment weight \(\beta = 0.8\) [2605.29591]. What distinguishes it is the addition of coarse-grained InfoNCE alignment between pooled brain tokens and CLIP image/text features, fine-grained token-level alignment via cross-attention to masked text tokens, and a perceptual loss that requires reconstructed fMRI to remain CLIP-decodable [2605.29591]. In this formulation, unification is not just compression but semantic co-location with visual and linguistic feature spaces. Not all neuroimaging tokenizers in this area are discrete. BrainHarmonix maps structural MRI and fMRI into shared continuous 1D hub tokens \(\widetilde{\mathbf{H}} \in \mathbb{R}^{N_H \times d}\), with \(N_H = 128\) and \(d = 768\) in the ViT-B configuration [2509.24693]. Structural MRI is patchified into \(N_S = 1200\) tokens, while fMRI is tokenized with Temporal Adaptive Patch Embedding so that each temporal token corresponds to a constant physical duration \(\tau = 35.28\) s despite heterogeneous repetition times [2509.24693]. BrainG3N similarly adopts a non-discrete formulation: a frozen 3D MAE encoder produces \(1200 \times 1152\) tokens from a \(160 \times 192 \times 160\) brain MRI volume, those tokens are projected to \(1200 \times 32\), and a separate CNN decoder reconstructs voxels from the projected tokens [2606.19651]. These systems indicate that the tokenizer concept in neuroimaging includes both discrete codebooks and compact continuous token interfaces. ## 4. Integration with foundation models and training objectives Once tokenized, unified brain representations are typically paired with masked modeling, next-token prediction, or discrete diffusion. In TFM-Tokenizer, the tokenizer is pretrained and then frozen; the downstream TFM-Encoder performs embedding lookup from the learned codebook, flattens tokens across channels and time, adds channel and position embeddings, prepends a \([CLS]\) token, and applies a four-layer transformer with linear attention [2502.16060]. Pretraining uses masked token prediction over the discrete vocabulary, conceptually identical to BERT-style masked language modeling, followed by downstream fine-tuning with task-specific losses [2502.16060]. BrainJanus treats the tokenized brain as a first-class modality in a decoder-only Janus-7B-based autoregressive transformer. Its All-in-One architecture models \[ p(z_1,\dots,z_T)=\prod_{t=1}^{T} p_\theta(z_t \mid z_{