SynBrain: A Probabilistic Visual-to-fMRI Framework

• SynBrain is a probabilistic visual-to-fMRI synthesis framework that models neural variability and semantic consistency as a one-to-many conditional generation problem.
• It employs a frozen CLIP visual encoder, BrainVAE for latent fMRI representation learning, and an 8-layer Transformer S2N mapper to project image semantics into neural space.
• The framework supports subject-specific synthesis, few-shot adaptation, and synthetic data augmentation, improving image retrieval and downstream fMRI-to-image decoding.

SynBrain most directly denotes a probabilistic visual-to-fMRI synthesis framework that models visual-to-neural mapping as a one-to-many conditional generation problem rather than as deterministic regression. It is designed to represent two properties simultaneously: biological variability, whereby repeated presentations of the same image need not yield identical fMRI responses, and functional consistency, whereby those responses still preserve stable semantic structure. In this formulation, SynBrain learns a semantic-conditioned distribution over plausible fMRI responses using a variational neural manifold, a semantic-to-neural projection module, and a frozen CLIP visual encoder (Mai et al., 14 Aug 2025). The term should be distinguished from SymBrain, whose official name refers instead to a neonatal MRI symmetry dataset with brain midline annotations (Gucciardi et al., 2024).

1. Definition and problem setting

SynBrain addresses visual-to-fMRI synthesis: given an image $x$, or more precisely its CLIP-derived semantic representation, generate an fMRI response $y_{\mathrm{fMRI}}$ that is biologically plausible and semantically consistent with the stimulus. The paper frames the target as the conditional distribution $p(y_{\mathrm{fMRI}} \mid x)$, or more specifically $p(y_{\mathrm{fMRI}} \mid z_{\mathrm{CLIP}})$, rather than a single point estimate (Mai et al., 14 Aug 2025).

This formulation is motivated by a specific critique of deterministic brain encoding. The paper argues that identical images can evoke different fMRI responses across repeated trials, attentional states, contexts, and subjects, while still preserving shared semantic function. Deterministic mappings tend to collapse this one-to-many structure into an averaged response, which may suppress meaningful neural variability. In that sense, SynBrain belongs to a broader shift from direct encoding to conditional generation. A related system, MindSimulator, also treats visual-to-fMRI as a one-to-many problem, but SynBrain explicitly argues that prior generative approaches can still remain limited if the core neural autoencoder is deterministic (Bao et al., 4 Mar 2025).

The framework’s stated contributions are correspondingly narrow and technical. It proposes BrainVAE for probabilistic fMRI representation learning with semantic constraints, a Semantic-to-Neural (S2N) Mapper for direct projection from CLIP semantics into the learned neural latent space, improved subject-specific visual-to-fMRI synthesis, few-shot adaptation to novel subjects, and the use of synthesized fMRI as augmentation for downstream fMRI-to-image decoding (Mai et al., 14 Aug 2025).

2. System architecture and probabilistic formulation

SynBrain has a two-stage design. A frozen CLIP visual encoder first maps an image to a semantic embedding,

$z_{\mathrm{CLIP}} = \mathcal V(x).$

BrainVAE then learns a probabilistic latent manifold of fMRI responses, while S2N learns to project visual semantics into that manifold. At inference, the pipeline is

$$x \xrightarrow{\mathcal V} z_{\mathrm{CLIP}} \xrightarrow{f_{\mathrm{S2N}}} z_{\mathrm{Align}} \xrightarrow{\mathcal D} \hat y_{\mathrm{fMRI}}.$$

The separation is deliberate: BrainVAE learns the structure of neural variability from measured fMRI, and S2N learns how to enter that structure from vision alone (Mai et al., 14 Aug 2025).

Component	Function	Key detail
Frozen visual encoder	Extract visual semantics	OpenCLIP ViT-bigG/14
BrainVAE	Learn probabilistic fMRI manifold	Gaussian posterior with CLIP alignment
S2N Mapper	Map image semantics into neural latent space	8-layer Transformer
Decoder	Reconstruct voxel-space fMRI	Decodes latent to subject-specific voxel vector

The framework is described as biologically interpretable because its latent space is meant to reflect stochastic neural responses constrained by visual semantics. That is not an explicit factorized model of separate “semantic” and “variability” latents; rather, semantic consistency is enforced through CLIP alignment, while stochasticity is introduced through variational learning and optional latent perturbation. This suggests a structured manifold interpretation rather than a strict disentanglement claim.

3. BrainVAE: probabilistic neural representation learning

BrainVAE is the core generative component. Given an fMRI input $y_{\mathrm{fMRI}} \in \mathbb{R}^{1 \times n}$, its encoder predicts the parameters of a Gaussian posterior,

$$q(z \mid y_{\mathrm{fMRI}}) = \mathcal N(\mu,\sigma^2),$$

samples a latent $z$, and decodes it back to $\hat y_{\mathrm{fMRI}}$. The prior is $y_{\mathrm{fMRI}}$0 (Mai et al., 14 Aug 2025).

Architecturally, the encoder begins with a 1D convolution from $y_{\mathrm{fMRI}}$1 channels with kernel size $y_{\mathrm{fMRI}}$2 and padding $y_{\mathrm{fMRI}}$3, followed by adaptive max pooling to length $y_{\mathrm{fMRI}}$4. It then uses a hierarchical ResNet-style 1D backbone with $y_{\mathrm{fMRI}}$5, $y_{\mathrm{fMRI}}$6, and $y_{\mathrm{fMRI}}$7, with downsampling only after the first level. A middle block inserts self-attention through

$y_{\mathrm{fMRI}}$8

and produces $y_{\mathrm{fMRI}}$9. Two identical MLP-based pre-projectors then map encoder features to $p(y_{\mathrm{fMRI}} \mid x)$0 and $p(y_{\mathrm{fMRI}} \mid x)$1, each using LayerNorm, GELU, Linear$p(y_{\mathrm{fMRI}} \mid x)$2, another LayerNorm–GELU–Linear$p(y_{\mathrm{fMRI}} \mid x)$3, and a final Linear$p(y_{\mathrm{fMRI}} \mid x)$4, yielding

$p(y_{\mathrm{fMRI}} \mid x)$5

The training objective is a weighted sum of reconstruction, KL regularization, and semantic alignment: $p(y_{\mathrm{fMRI}} \mid x)$6 with

$p(y_{\mathrm{fMRI}} \mid x)$7

and

$p(y_{\mathrm{fMRI}} \mid x)$8

The role of the SoftCLIP term is central. It anchors the neural latent space to visual semantics, so the manifold is not merely reconstructive but also semantically organized. The paper reports that a simpler MLP-VAE produced unstable training, collapsed reconstructions, and divergent MSE, motivating the convolution-plus-attention design.

4. Semantic-to-Neural Mapper, training protocol, and inference

The S2N Mapper transforms visual semantics into the BrainVAE latent space. Its input is the CLIP embedding $p(y_{\mathrm{fMRI}} \mid x)$9; in the appendix the concrete shape is

$p(y_{\mathrm{fMRI}} \mid z_{\mathrm{CLIP}})$0

S2N is an 8-layer Transformer with 13 attention heads per layer, fixed sinusoidal positional encoding, pre-layer normalization, multi-head self-attention, feedforward MLP with GELU, and residual connections, implementing

$p(y_{\mathrm{fMRI}} \mid z_{\mathrm{CLIP}})$1

It is trained by latent-space regression against the BrainVAE encoder output: $p(y_{\mathrm{fMRI}} \mid z_{\mathrm{CLIP}})$2 Here the target $p(y_{\mathrm{fMRI}} \mid z_{\mathrm{CLIP}})$3 is obtained from real fMRI via the pretrained BrainVAE encoder (Mai et al., 14 Aug 2025).

The paper’s main justification for S2N is to avoid the train–test mismatch of diffusion-from-noise priors. In the appendix, the average Euclidean distance from noisy fMRI to the original fMRI manifold is reported as $p(y_{\mathrm{fMRI}} \mid z_{\mathrm{CLIP}})$4, whereas the distance from pure Gaussian noise to the manifold is $p(y_{\mathrm{fMRI}} \mid z_{\mathrm{CLIP}})$5. S2N avoids starting from unstructured noise. The paper attributes the performance difference between BrainVAE-DiT and SynBrain to this elimination of diffusion train–test mismatch.

Inference is optionally stochastic. After predicting the aligned latent, the framework can sample

$p(y_{\mathrm{fMRI}} \mid z_{\mathrm{CLIP}})$6

where $p(y_{\mathrm{fMRI}} \mid z_{\mathrm{CLIP}})$7 is a noise factor controlling synthesis variability. When $p(y_{\mathrm{fMRI}} \mid z_{\mathrm{CLIP}})$8, generation is deterministic from the predicted latent center; larger values introduce semantically constrained variation.

Training is staged rather than end-to-end. BrainVAE is trained first with AdamW, $p(y_{\mathrm{fMRI}} \mid z_{\mathrm{CLIP}})$9, learning rate $z_{\mathrm{CLIP}} = \mathcal V(x).$0, weight decay $z_{\mathrm{CLIP}} = \mathcal V(x).$1, batch size $z_{\mathrm{CLIP}} = \mathcal V(x).$2, and early stopping. S2N is then trained with BrainVAE frozen, using the same optimizer settings for 50,000 steps. Few-shot adaptation to a new subject with 1 hour of data finetunes the entire BrainVAE and updates only the MLP submodules inside the S2N Transformer. Training was performed on 4 NVIDIA A100 GPUs and completed within 2 hours (Mai et al., 14 Aug 2025).

5. Dataset, empirical results, and downstream uses

Experiments use the Natural Scenes Dataset (NSD) with four subjects: Sub-1, Sub-2, Sub-5, Sub-7. Per subject, the paper uses 9,000 unique training images, 1,000 shared test images, and 3 trials per test image. Appendix details report 27,000 fMRI training trials, 3,000 test trials, 1.8 mm resolution, ROI data from early and higher ventral visual regions, and subject-specific voxel counts

$z_{\mathrm{CLIP}} = \mathcal V(x).$3

Evaluation covers voxel-level metrics (MSE, Pearson correlation, Cosine similarity), semantic-level metrics obtained by decoding synthesized fMRI to images with MindEye2, and retrieval based on cosine similarity between fMRI embeddings and CLIP image embeddings over 300 candidate images (Mai et al., 14 Aug 2025).

On subject-specific synthesis averaged across four subjects, SynBrain outperforms MindSimulator. The reported averages are:

• MindSimulator (Trials=1): $z_{\mathrm{CLIP}} = \mathcal V(x).$4
• MindSimulator (Trials=5): $z_{\mathrm{CLIP}} = \mathcal V(x).$5
• SynBrain (Trials=1): $z_{\mathrm{CLIP}} = \mathcal V(x).$6 (Mai et al., 14 Aug 2025).

One of the paper’s strongest observations is that synthesized fMRI supports better image retrieval than raw fMRI. The authors interpret this as evidence that SynBrain distills task-relevant semantic structure from sparse and noisy measured signals. On Subject 1, the appendix reports

$z_{\mathrm{CLIP}} = \mathcal V(x).$7

Few-shot cross-subject adaptation is also a major result. With only 1 hour of target-subject data, transfer from Sub1 yields:

• Sub1 $z_{\mathrm{CLIP}} = \mathcal V(x).$8 Sub2: $z_{\mathrm{CLIP}} = \mathcal V(x).$9
• Sub1 $$x \xrightarrow{\mathcal V} z_{\mathrm{CLIP}} \xrightarrow{f_{\mathrm{S2N}}} z_{\mathrm{Align}} \xrightarrow{\mathcal D} \hat y_{\mathrm{fMRI}}.$$0 Sub5: $$x \xrightarrow{\mathcal V} z_{\mathrm{CLIP}} \xrightarrow{f_{\mathrm{S2N}}} z_{\mathrm{Align}} \xrightarrow{\mathcal D} \hat y_{\mathrm{fMRI}}.$$1
• Sub1 $$x \xrightarrow{\mathcal V} z_{\mathrm{CLIP}} \xrightarrow{f_{\mathrm{S2N}}} z_{\mathrm{Align}} \xrightarrow{\mathcal D} \hat y_{\mathrm{fMRI}}.$$2 Sub7: $$x \xrightarrow{\mathcal V} z_{\mathrm{CLIP}} \xrightarrow{f_{\mathrm{S2N}}} z_{\mathrm{Align}} \xrightarrow{\mathcal D} \hat y_{\mathrm{fMRI}}.$$3.

The framework also serves as synthetic-data augmentation for data-limited fMRI-to-image decoding. For Subject 1 with 1 hour of real data, MindEye2 (1h) reports $$x \xrightarrow{\mathcal V} z_{\mathrm{CLIP}} \xrightarrow{f_{\mathrm{S2N}}} z_{\mathrm{Align}} \xrightarrow{\mathcal D} \hat y_{\mathrm{fMRI}}.$$4, while MindEye2 (1h) + DA(1h) improves to $$x \xrightarrow{\mathcal V} z_{\mathrm{CLIP}} \xrightarrow{f_{\mathrm{S2N}}} z_{\mathrm{Align}} \xrightarrow{\mathcal D} \hat y_{\mathrm{fMRI}}.$$5. The paper notes that moderate augmentation helps most, whereas larger synthetic additions plateau or mildly decline.

Ablations reinforce the design choices. On Subject 1, removing variation sampling reduces performance from $$x \xrightarrow{\mathcal V} z_{\mathrm{CLIP}} \xrightarrow{f_{\mathrm{S2N}}} z_{\mathrm{Align}} \xrightarrow{\mathcal D} \hat y_{\mathrm{fMRI}}.$$6 and $$x \xrightarrow{\mathcal V} z_{\mathrm{CLIP}} \xrightarrow{f_{\mathrm{S2N}}} z_{\mathrm{Align}} \xrightarrow{\mathcal D} \hat y_{\mathrm{fMRI}}.$$7 to $$x \xrightarrow{\mathcal V} z_{\mathrm{CLIP}} \xrightarrow{f_{\mathrm{S2N}}} z_{\mathrm{Align}} \xrightarrow{\mathcal D} \hat y_{\mathrm{fMRI}}.$$8 and $$x \xrightarrow{\mathcal V} z_{\mathrm{CLIP}} \xrightarrow{f_{\mathrm{S2N}}} z_{\mathrm{Align}} \xrightarrow{\mathcal D} \hat y_{\mathrm{fMRI}}.$$9. Removing contrastive learning causes near-total retrieval collapse, and removing S2N produces severe semantic degradation. Compared with BrainVAE-DiT, SynBrain improves from $y_{\mathrm{fMRI}} \in \mathbb{R}^{1 \times n}$0 to $y_{\mathrm{fMRI}} \in \mathbb{R}^{1 \times n}$1 (Mai et al., 14 Aug 2025).

6. Relation to adjacent research, naming ambiguities, and limitations

SynBrain sits within a broader family of synthetic brain-representation systems, but its scope is specific. Relative to MindSimulator, it shares the one-to-many visual-to-fMRI motivation yet replaces diffusion-from-noise with a variational latent manifold and a one-step Transformer mapping from CLIP semantics to neural latents (Bao et al., 4 Mar 2025). Relative to ViBE, which extends visual-to-neural synthesis to EEG and MEG through a spatio-temporal VAE and distribution-aligned projection, SynBrain is specialized to static-image, static-fMRI synthesis rather than high-temporal-resolution electrophysiology (Xu et al., 29 Apr 2026). Relative to Text2Brain, which synthesizes whole-brain activation maps from free-form text queries, SynBrain is a visual-to-fMRI encoder rather than a language-conditioned meta-analytic generator (Ngo et al., 2021).

A recurrent source of confusion is naming. SymBrain—with an $y_{\mathrm{fMRI}} \in \mathbb{R}^{1 \times n}$2—is the official name of the neonatal MRI symmetry dataset introduced in “SymBrain: A Large-Scale Dataset of MRI Images for Neonatal Brain Symmetry Analysis,” and searches for “SynBrain” sometimes retrieve it. That resource is a 2D coronal slice dataset with straight-line brain midline annotations, not a visual-to-fMRI synthesis framework (Gucciardi et al., 2024).

The limitations of SynBrain are explicit. The method relies on pretrained vision models such as CLIP, whose representational biases may not match neural processing perfectly. It models variability, but cannot account for all sources of neural variance, including attentional and neuromodulatory effects. Despite its biological interpretation, it does not explicitly isolate each source of variability in a factorized latent model. More broadly, it is a static-image, static-fMRI framework and does not address temporal neural dynamics (Mai et al., 14 Aug 2025).

A related misconception is to treat synthesized fMRI as direct measurement. The paper does not make that claim. It instead argues that the generated signals are useful because they preserve semantically and cortically interpretable structure, improve retrieval, and augment downstream decoding. A plausible implication is that SynBrain is best understood as a structured probabilistic prior over visual cortical responses rather than a substitute for experimental fMRI.

In summary, SynBrain is a semantic-conditioned probabilistic framework for generating fMRI responses from visual stimuli. Its distinctive contribution lies in combining a variational neural manifold, CLIP-grounded semantic alignment, and one-step semantic-to-neural projection so that neural variability is represented without discarding semantic consistency. Within current synthetic-brain research, it occupies a precise niche: a biologically motivated visual-to-fMRI generator that is empirically useful for subject-specific encoding, few-shot subject transfer, and synthetic-data augmentation, while remaining explicitly limited by pretrained vision priors, unmodeled sources of variance, and the static nature of its imaging setting (Mai et al., 14 Aug 2025).