BrainFLORA: Unified Multimodal Neural Decoding

• BrainFLORA is a unified multimodal neural decoding framework that maps EEG, MEG, and fMRI visual responses into a common latent space aligned with CLIP image embeddings.
• It employs modality-specific encoders and a Mixture of Experts universal projector to overcome spatiotemporal misalignment and support tasks like retrieval, reconstruction, and captioning.
• State-of-the-art retrieval performance on THINGS datasets showcases its effectiveness in cross-modal visual concept representation analysis.

Searching arXiv for BrainFLORA and closely related work to ground the article. BrainFLORA is a unified multimodal neural decoding framework for learning a shared representation of visual information from EEG, MEG, and fMRI. It was introduced to address the spatiotemporal misalignment that arises when neural modalities with complementary properties—high temporal resolution for EEG and MEG, high spatial resolution for fMRI—are analyzed in isolation rather than within a common latent space. The framework maps modality-specific neural responses elicited by visual stimuli into a shared semantic representation aligned to CLIP image embeddings, and supports retrieval, reconstruction, and captioning from the same learned neural embedding space. In the reported experiments, BrainFLORA achieves state-of-the-art performance on a joint-subject visual retrieval benchmark built from THINGS-family datasets, while also being used as an analytic tool for studying visual concept representation across neural modalities (Li et al., 13 Jul 2025).

1. Concept and scientific motivation

BrainFLORA is defined around two linked objectives: improved multimodal neural decoding and analysis of cross-modal concept representation. The paper frames the central methodological problem as the lack of a truly unified multimodal latent space across EEG, MEG, and fMRI. Prior systems are described as either modality-specific or otherwise unable to cope effectively with severe spatiotemporal heterogeneity, and therefore limited in their ability to provide a holistic view of how visual concepts are represented across neural measurements (Li et al., 13 Jul 2025).

The framework is explicitly organized around a shared neural embedding space. Neural recordings induced by visual stimuli are passed through modality-specific encoders and then aligned by a universal projection module into a common semantic space, where they can be compared directly to one another and to CLIP image embeddings. The paper also characterizes this shared representation as a multimodal neural embedding: a modality-agnostic latent code intended to preserve visual-semantic content while suppressing modality-specific nuisance variation (Li et al., 13 Jul 2025).

The scientific claim is narrower than a full mechanistic account of visual cortex. BrainFLORA is presented as evidence that visual concept representations from different neural modalities can be organized into a shared geometry that correlates with real-world object concept spaces. The paper does not provide brain-region hierarchical localization, voxelwise interpretability, or explicit temporal-dynamics decomposition in the main text. This suggests that its principal contribution lies in cross-modal representational alignment rather than anatomically localized explanatory neuroscience (Li et al., 13 Jul 2025).

2. Architecture and shared embedding design

The architecture has three major components: modality-specific neural feature extraction, a universal projection module for cross-modal alignment, and downstream task-specific heads. Although BrainFLORA is described as a unified framework, it still uses separate modality encoders before alignment. These encoders are designed to handle subject heterogeneity and temporal structure through a multi-granularity patching and attention mechanism inspired by MedFormer (Li et al., 13 Jul 2025).

For neural input $X$, the encoder produces an embedding

$Z = \mathcal{E}(X) \in \mathbb{R}^{N \times F},$

while the paired image $I$ is encoded by CLIP into $\hat Z \in \mathbb{R}^{N \times F}$. For an input neural signal $X \in \mathbb{R}^{T \times C}$, the encoder applies multi-granularity time patching with exponentially increasing patch lengths $\{2^1,2^2,\dots,2^n\}$. For each granularity $i$, the patch sequence is

$$x_p^{(i)} \in \mathbb{R}^{N_i \times (2^i \cdot C)}, \qquad N_i = \lceil T/2^i \rceil,$$

and the tokenization step is

$$x^{(i)} = x_p^{(i)}W^{(i)} + W_{pos}[1 : N_i] + W_{gr}^{(i)}.$$

Each granularity also has a router token

$u^{(i)} = W_{pos}[N_i + 1] + W_{gr}^{(i)}.$

The encoder then applies a two-stage attention process. Intra-granularity attention updates patch tokens and routers within each scale after concatenation,

$Z = \mathcal{E}(X) \in \mathbb{R}^{N \times F},$0

followed by inter-granularity interaction via the router tokens. The attended scale-specific features are concatenated into

$Z = \mathcal{E}(X) \in \mathbb{R}^{N \times F},$1

after which a temporal-spatial convolution module and an MLP produce the final modality embedding $Z = \mathcal{E}(X) \in \mathbb{R}^{N \times F},$2 (Li et al., 13 Jul 2025).

The universal projection module is the primary alignment mechanism. It is implemented as a Mixture of Experts projection with $Z = \mathcal{E}(X) \in \mathbb{R}^{N \times F},$3 experts $Z = \mathcal{E}(X) \in \mathbb{R}^{N \times F},$4, each a two-layer MLP, and a soft router $Z = \mathcal{E}(X) \in \mathbb{R}^{N \times F},$5. Given an input token $Z = \mathcal{E}(X) \in \mathbb{R}^{N \times F},$6, routing is defined as

$Z = \mathcal{E}(X) \in \mathbb{R}^{N \times F},$7

For embeddings $Z = \mathcal{E}(X) \in \mathbb{R}^{N \times F},$8, the routing weights are

$Z = \mathcal{E}(X) \in \mathbb{R}^{N \times F},$9

and the aligned representation is

$I$0

This module is intended to reduce heterogeneity across EEG, MEG, and fMRI while still allowing modality-dependent transformations (Li et al., 13 Jul 2025).

3. Training objectives and downstream tasks

BrainFLORA is trained progressively and multitask. The first stage performs visual-neural contrastive training of modality encoders and the universal projector. The second stage performs cross-modal neural alignment through the universal projector. The third stage performs task-specific fine-tuning for retrieval, reconstruction, and captioning (Li et al., 13 Jul 2025).

The main training objective is a CLIP-style soft contrastive loss:

$I$1

with

$I$2

Here, $I$3 is the aligned neural embedding, $I$4 is the CLIP embedding of the paired image, and $I$5 is a temperature hyperparameter. The overall objective is written as

$I$6

The paper references BiMixCo but does not separately define it mathematically in the provided text (Li et al., 13 Jul 2025).

For reconstruction, the aligned embedding is passed through an additional CNN upsampler to predict an approximation $I$7 of an image feature map $I$8, and an image is reconstructed using the SDXL VAE decoder. The reported low-level reconstruction loss is

$I$9

For captioning, the shared neural embedding is decoded using OpenFlamingo under a one-shot prompt setting. The framework is therefore not limited to retrieval, even though retrieval is the best-specified benchmark task in the paper (Li et al., 13 Jul 2025).

This task structure places BrainFLORA between classical neural decoding systems and broader multimodal representation learners. A plausible implication is that the framework is intended as a general-purpose neural representation model with multiple readout heads rather than as a single-task decoder.

4. Data, implementation, and evaluation protocol

The data setup is centered on THINGS-family datasets. BrainFLORA is jointly trained on THINGS-EEG2, THINGS-MEG, and THINGS-fMRI, all tied to the THINGS object image database. THINGS-EEG2 contains 10 subjects and 1,654 concepts; THINGS-MEG contains 4 subjects and 1,854 concepts; THINGS-fMRI contains 3 subjects and 720 concepts. The introduction also mentions 16,540 paired EEG-image samples from THINGS-EEG1, 19,848 paired MEG-image samples, and 8,640 paired fMRI-image samples when describing multimodal scale, but the main experiments are reported on THINGS-EEG2, THINGS-MEG, and THINGS-fMRI (Li et al., 13 Jul 2025).

The retrieval benchmark is a joint-subject visual decoding benchmark. The model is trained across subjects and evaluated on average across all subjects rather than by fitting a separate model for each individual. The paper contrasts this setup with prior work that often trains and tests within-subject, although the main text does not fully specify whether every experiment follows a leave-one-subject-out protocol or a pooled subject-average setting (Li et al., 13 Jul 2025).

All experiments used two NVIDIA H20-96GB GPUs. CLIP ViT-L/14 was used for image embeddings. Each modality encoder was trained on the original THINGS training set for 50 epochs with learning rate $\hat Z \in \mathbb{R}^{N \times F}$0, batch size 360, and AdamW. For retrieval, the paper separately states that the model was trained for 150 epochs in the joint-subject setting across EEG, MEG, and fMRI while optimizing only the contrastive loss. The main text does not specify scheduler, weight decay, warmup, augmentations, gradient clipping, or the number of MoE experts $\hat Z \in \mathbb{R}^{N \times F}$1 (Li et al., 13 Jul 2025).

The benchmark uses 2-way, 4-way, 10-way, and full-set retrieval: 200-way for EEG and MEG and 100-way for fMRI. Metrics include Top-1 and Top-5 accuracy. Reconstruction is evaluated with PixCorr, SSIM, AlexNet(2), AlexNet(5), Inception, CLIP, EffNet-B, and SwAV. Captioning is evaluated with BLEU-1/2/3/4, METEOR, ROUGE, CIDEr, SPICE, CLIP-S, and RefCLIP-S (Li et al., 13 Jul 2025).

5. Empirical performance and concept-space analysis

On THINGS-EEG2, BrainFLORA-uni reaches 95.55% in 2-way retrieval, 86.90% in 4-way, 73.45% in 10-way, 25.35% in 200-way Top-1, and 57.30% in 200-way Top-5. BrainFLORA-multi reaches 94.05%, 87.30%, 73.15%, 25.05%, and 56.35%, respectively. On THINGS-MEG, BrainFLORA-uni obtains 81.75%, 64.50%, 46.62%, 8.00%, and 24.38%, while BrainFLORA-multi obtains 80.50%, 61.88%, 39.75%, 6.88%, and 23.38%. On THINGS-fMRI, BrainFLORA-multi is strongest, with 92.33%, 84.67%, 70.67%, 28.33%, and 63.33% on 2-way, 4-way, 10-way, 100-way Top-1, and 100-way Top-5, respectively. These results are reported as state of the art under the authors’ joint-subject benchmark and evaluation protocol (Li et al., 13 Jul 2025).

The paper is careful to note that MEG results are lower than some standard benchmarks because the evaluation uses single-trial neural responses for test images, thereby removing the signal-averaging advantage present in many standard setups. The reported $\hat Z \in \mathbb{R}^{N \times F}$2Acc of -48.78% relative to original splits is interpreted by the authors as reflecting the increased difficulty of the protocol rather than a model weakness alone (Li et al., 13 Jul 2025).

For reconstruction, the paper reports average multimodal performance rather than a direct baseline table. On THINGS-EEG2, reported scores are PixCorr 0.101, SSIM 0.360, AlexNet(2) 0.653, AlexNet(5) 0.708, Inception 0.607, CLIP 0.635, EffNet-B 0.819, and SwAV 0.625. Captioning is reported separately by modality because of GPU memory constraints. THINGS-EEG2 yields BLEU4 19.67, METEOR 20.01, ROUGE 45.86, CIDEr 2.86, SPICE 15.62, CLIP-S 42.74, and RefCLIP-S 51.80; THINGS-fMRI is slightly better on several metrics, including BLEU4 20.11 and RefCLIP-S 52.96 (Li et al., 13 Jul 2025).

BrainFLORA is also used as a representational analysis tool. The learned neural embeddings are projected into a 42-dimensional concept space using the projection layer from CoCoG, and representational similarity matrices over objects excluded from BrainFLORA training are compared with measured similarity. The reported correlations are 0.49 for EEG, 0.55 for MEG, and 0.40 for fMRI. The paper interprets this as evidence that the learned embeddings preserve concept similarity structure and align with a visual concept space (Li et al., 13 Jul 2025).

To analyze category structure, the authors visualize embeddings using MDS-initialized t-SNE on Subject-01 across modalities and report that objects from the same semantic category cluster together consistently across EEG, MEG, and fMRI. This suggests that the shared latent space supports category-structured representational geometry. Because the paper repeatedly characterizes this as an implicit mapping from brain concept representations to physical-world stimuli, a plausible implication is that BrainFLORA is meant to function simultaneously as a decoder and as a compact hypothesis space for concept-level neuroscience (Li et al., 13 Jul 2025).

6. Position in the research landscape, limitations, and prospective significance

BrainFLORA belongs to a broader movement toward large-scale, multimodal, and transferable neuro-AI models, but its design choices are specific. Unlike cloud research infrastructure such as brainlife.io, which is organized around Projects, Datatypes, Apps, provenance graphs, and decentralized cloud/HPC orchestration, BrainFLORA is a decoding and representation-learning model rather than an end-to-end FAIR workflow platform (Hayashi et al., 2023). Unlike BRAINCELL-AID, which uses a three-agent LLM-plus-RAG workflow for literature-grounded annotation of 5,322 mouse brain cell clusters and 21,275 marker gene sets, BrainFLORA addresses neural signal alignment for visual concept representation rather than atlas-scale molecular annotation (Li et al., 20 Oct 2025).

A closer methodological comparison arises with BrainHarmonix, which is presented as the first multimodal brain foundation model that unifies structural morphology and functional dynamics into compact 1D token representations from T1 MRI and resting-state fMRI. BrainHarmonix uses staged unimodal pretraining followed by multimodal fusion through shared brain hub tokens, whereas BrainFLORA aligns EEG, MEG, and fMRI to CLIP image embeddings through modality-specific encoders and a MoE-based universal projector. This suggests that BrainFLORA is more closely tied to stimulus-driven decoding and semantic alignment, while BrainHarmonix is framed as a foundation model for morphology-function integration and downstream clinical or cognitive transfer (Dong et al., 29 Sep 2025).

The paper acknowledges several limitations. Sample distribution is uneven across modalities and classes. Parameter sharing may hurt specialized tasks such as reconstruction and captioning relative to methods designed for one modality and one task. The THINGS datasets are limited in size and have category overlap issues between train and test, especially in fMRI. Subject variability remains a challenge. Computational cost is substantial because the framework uses CLIP, SDXL, IP-Adapter, and OpenFlamingo, and training was conducted on high-memory GPUs. The main text also omits several details needed for exact reproduction, including exact preprocessing for EEG, MEG, and fMRI, embedding dimension $\hat Z \in \mathbb{R}^{N \times F}$3, exact architectural depths and widths, the number of MoE experts $\hat Z \in \mathbb{R}^{N \times F}$4, the precise zero-shot split protocol, and the exact definitions of BrainFLORA-uni versus BrainFLORA-multi (Li et al., 13 Jul 2025).

The broader significance of BrainFLORA lies in the claim that severe cross-modal neural heterogeneity can nevertheless be mapped into a common semantic latent space that is useful both for decoding and for concept analysis. Within the current literature, that places BrainFLORA at the intersection of multimodal representation learning, neural decoding, and computational cognitive neuroscience. The evidence reported in the paper supports the narrower conclusion that shared embeddings across EEG, MEG, and fMRI can be operationalized effectively for joint-subject visual retrieval and can recover concept-similarity structure. Stronger claims about mechanistic neural coding, anatomical grounding, or general multimodal brain foundation modeling would require additional evidence beyond what is presently reported (Li et al., 13 Jul 2025).