Rest2Visual: Conditional ve-fMRI Mapping

• Rest2Visual is a conditional generative model that synthesizes individualized, stimulus-aligned activation maps by integrating resting-state connectivity with visual stimulus embeddings.
• It employs a volumetric encoder–decoder architecture with adaptive normalization to seamlessly fuse multiscale brain features with semantic image representations.
• Evaluations demonstrate superior voxel-wise and representational similarity, preserving subject-specific details crucial for precision neuroscience.

Rest2Visual is a conditional generative model for predicting visually evoked functional magnetic resonance imaging (ve-fMRI) from resting-state fMRI (rs-fMRI) and 2D visual stimuli. It is presented as a method for linking spontaneous brain activity and stimulus-driven neural responses at the voxel level, with the stated goal of generating subject-specific, stimulus-aligned ve-fMRI maps from task-free data and images. The method uses a volumetric encoder–decoder in which multiscale 3D features extracted from rs-fMRI are modulated by image embeddings via adaptive normalization, and it is trained on a large-scale triplet dataset derived from the Natural Scenes Dataset (NSD). Reported evaluations indicate agreement between predicted and measured activations across voxel-wise, representational, signal-quality, and downstream decoding criteria, while preserving subject-specific structure (Zhou et al., 17 Sep 2025).

1. Problem formulation and scope

Rest2Visual addresses a central problem in cognitive neuroscience: how spontaneous activity observed during rest relates to neural responses evoked by visual stimulation. In the formulation given for the model, task-based fMRI is described as capturing localized stimulus-evoked activation but being costly, time-consuming, and difficult to scale, whereas rs-fMRI is abundant and task-free but lacks direct interpretability with respect to stimulus processing (Zhou et al., 17 Sep 2025).

The specific problem setting is supervised prediction of a 3D ve-fMRI activation map from two inputs: a resting-state intrinsic function map, denoted $\mathbf{z}^{(s,t)}$, and a 2D visual stimulus, denoted $\mathbf{v}_i^{(s,t)}$. The target is the corresponding stimulus-evoked beta map, $\tilde{\mathbf{y}_i^{(s,t)}}$, for subject $s$, session $t$, and image $i$. The model is explicitly framed as subject-specific and stimulus-aligned rather than as a population-average predictor.

A notable claim in the source description is that Rest2Visual is the first model to predict subject-specific, stimulus-aligned ve-fMRI maps from resting-state data and image stimuli (Zhou et al., 17 Sep 2025). The broader significance assigned to this formulation is “task-free” brain modeling: rs-fMRI is treated not merely as an index of intrinsic connectivity but as a substrate from which individualized, functionally interpretable, stimulus-conditioned responses can be synthesized.

2. Generative framework and architectural design

Rest2Visual is implemented as a conditional generative framework with a volumetric encoder–decoder design. Its forward mapping is given as

$$\hat{\mathbf{y}_i^{(s,t)}} = f_{\theta}(\mathbf{z}^{(s,t)}, \mathbf{v}_i^{(s,t)}).$$

The system contains three principal components: an image encoder $\mathcal{E}_{\text{img}}$, a volumetric encoder $\mathcal{E}_{\text{vol}}$, and a conditional decoder $\mathcal{D}_{\text{cond}}$ (Zhou et al., 17 Sep 2025).

Component	Input	Output or role
$\mathbf{v}_i^{(s,t)}$0	2D visual stimulus	Semantic embeddings in $\mathbf{v}_i^{(s,t)}$1
$\mathbf{v}_i^{(s,t)}$2	Resting-state intrinsic function map	Multiscale 3D feature maps
$\mathbf{v}_i^{(s,t)}$3	Volumetric features plus image embeddings	Predicted 3D ve-fMRI activation map

The image encoder processes the RGB stimulus and produces semantic embeddings. The volumetric encoder uses 3D convolutions to extract multiscale 3D features from the rs-fMRI-derived connectivity representation. The conditional decoder reconstructs the predicted ve-fMRI map while integrating the image embedding at multiple scales via adaptive group normalization. According to the description, this modulation mechanism ensures that global and local visual semantics influence spatially distributed brain features, thereby enabling spatially accurate and stimulus-specific synthesis (Zhou et al., 17 Sep 2025).

This architecture is important because the model does not treat the resting-state input as a low-dimensional nuisance covariate. Instead, rs-fMRI enters as a spatial volumetric prior, and the visual input provides stimulus semantics that modulate that prior. A plausible implication is that the model operationalizes evoked response prediction as a fusion problem between individualized intrinsic functional structure and externally specified visual content.

3. Triplet dataset construction and resting-state representation

Training is enabled by a triplet dataset constructed from the Natural Scenes Dataset. The source dataset comprises 8 adults scanned at 7T, with thousands of natural images from MS COCO and extensive per-subject, per-session rs-fMRI and ve-fMRI measurements (Zhou et al., 17 Sep 2025).

Each sample is a triplet

$\mathbf{v}_i^{(s,t)}$4

where $\mathbf{v}_i^{(s,t)}$5 is the resting-state connectivity map for subject $\mathbf{v}_i^{(s,t)}$6 and session $\mathbf{v}_i^{(s,t)}$7, $\mathbf{v}_i^{(s,t)}$8 is the stimulus image, and $\mathbf{v}_i^{(s,t)}$9 is the corresponding ve-fMRI beta map after cleaning and masking for validity (Zhou et al., 17 Sep 2025).

All brains are registered to MNI152 space at 2 mm isotropic resolution using continuous interpolation. For resting-state feature extraction, the Posterior Cingulate Cortex (PCC) is used as a seed region. The mean PCC BOLD trace is bandpass filtered and HRF convolved, after which voxelwise regression coefficients are computed and z-scored to form the intrinsic function map $\tilde{\mathbf{y}_i^{(s,t)}}$0. On the stimulus-evoked side, trial-level GLMs are used to generate beta maps per stimulus, and outliers are masked (Zhou et al., 17 Sep 2025).

The use of a PCC-seeded intrinsic function map is a defining methodological choice. It means that Rest2Visual does not ingest raw rs-fMRI time series directly; instead, it uses a session-specific spatial map derived from functional connectivity estimation. This suggests that the model relies on a distilled representation of intrinsic organization rather than on full temporal dynamics.

4. Optimization objective and evaluation protocol

Rest2Visual is trained by minimizing voxel-wise mean squared error between predicted and true ve-fMRI. The loss is specified as

$\tilde{\mathbf{y}_i^{(s,t)}}$1

Optimization uses batch-averaged MSE over triplets and the Adam optimizer (Zhou et al., 17 Sep 2025).

The train/validation/test split is stratified by subject and image, with a test set of 1,000 held-out subject-stimulus pairs. A cross-subject model is trained on all 8 participants in order to leverage inter-individual pattern variability (Zhou et al., 17 Sep 2025).

Evaluation is performed along several axes. Voxel-wise similarity is measured with Pearson correlation, mean absolute error (MAE), and structural similarity index (SSIM). Representational similarity analysis (RSA) quantifies correspondence between the representational geometry of predicted and ground-truth multistimulus responses by correlating the upper-triangular entries of their representational dissimilarity matrices. Signal quality is evaluated with signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR), both for anatomically defined visual cortex regions V1–V5 and for the whole brain. Additional assessment uses downstream decoding: synthesized ve-fMRI is passed through an fMRI-to-image decoder, and resulting reconstructions are compared to the original images using low-level metrics such as SSIM and AlexNet similarity and high-level metrics such as Inception, CLIP, and SwAV similarity. Subject-specificity is tested with inter-subject correlation matrices (Zhou et al., 17 Sep 2025).

This evaluation design is broader than a simple regression benchmark. It treats success as requiring voxel-level agreement, preservation of multistimulus representational geometry, acceptable signal characteristics, utility for downstream reconstruction, and retention of subject identity.

5. Empirical findings, robustness, and failure modes

The reported quantitative results state that Rest2Visual yields substantial improvements over both random and resting-state-only baselines across all metrics, with Pearson correlation, RSA, and SSIM highest for Rest2Visual, especially in visual cortex regions of interest (Zhou et al., 17 Sep 2025).

Qualitatively, predicted maps are described as recovering retinotopic and anatomically coherent activation patterns. An especially notable observation is that SNR and CNR in the predictions exceed those of the ground-truth maps, which is interpreted in the summary as indicating sharper, less noisy activation and potentially reflecting denoising or reconstruction effects (Zhou et al., 17 Sep 2025). This point is important because such results should not be read as showing that the predictions are more biologically “true” than the measurements; rather, they indicate that the synthesis process may regularize or smooth measured noise.

Subject-specificity is reported to be preserved. Inter-subject correlation matrices show low off-diagonal elements and patterns closely mirroring ground truth, which is presented as evidence that the model avoids collapsing to a population mean and transfers individual functional signatures from rs-fMRI to predicted ve-fMRI (Zhou et al., 17 Sep 2025).

In downstream visual decoding, image reconstructions from predicted ve-fMRI retain object identity, color, and scene layout, with feature-similarity metrics only slightly lower than those obtained from true ve-fMRI. The reported explanation for this gap is a minor domain shift (Zhou et al., 17 Sep 2025).

The robustness analyses clarify the functional roles of the two inputs. Noisy resting-state input produces spatially nonsensical, semantically meaningless activations, which is presented as evidence for the necessity of functional priors. Noisy stimulus input yields spatially plausible activations, including retinotopy, but semantically random content. Generalization to unseen stimuli remains strong, whereas generalization to an unseen subject degrades, indicating reliance on individualized functional priors. Region-specific analysis places the best correlation and representational alignment in mid-level visual areas V3 and V4, associated in the summary with object and scene perception (Zhou et al., 17 Sep 2025).

6. Scientific significance, limitations, and relation to adjacent work

The principal significance attributed to Rest2Visual is that it demonstrates the feasibility of synthesizing individualized, functionally interpretable, stimulus-aligned activation patterns from rs-fMRI. In the summary, this is positioned as a practical alternative to task fMRI for neural representation mapping, especially in populations unable to perform tasks, including clinical and pediatric settings (Zhou et al., 17 Sep 2025).

The work also has a clear scalability argument. Because rs-fMRI is more abundant and task-free, the pipeline is described as enabling rapid “virtual” functional mapping at scale. The preservation of individual functional signatures is emphasized as relevant to precision neuroscience (Zhou et al., 17 Sep 2025).

At the same time, the reported performance drop for unseen subjects is an explicit limitation. The model’s dependence on individualized functional priors implies that subject generalization remains constrained. Another limitation, stated indirectly through the dataset construction, is that the method relies on aligned triplets of resting-state maps, stimulus images, and stimulus-evoked beta maps for training, which ties its current instantiation to richly annotated datasets such as NSD.

A useful comparison can be made with direct visual decoding methods. DREAM, for example, reconstructs viewed images from fMRI by reversing pathways specialized for semantics, color, and depth, and is also evaluated on NSD (Xia et al., 2023). Rest2Visual differs in target space: it predicts ve-fMRI maps rather than images. This suggests that Rest2Visual is more naturally interpreted as a method for generating individualized functional surrogates, whereas direct reconstruction systems operate at the level of image synthesis. The downstream decoding results reported for Rest2Visual indicate that these two research directions can be connected: a model may first synthesize stimulus-aligned brain activation and then decode images from that synthesized activity (Zhou et al., 17 Sep 2025).

In that sense, Rest2Visual occupies a specific position within neuro-AI modeling. It is neither a conventional resting-state connectivity analysis nor a direct fMRI-to-image reconstruction pipeline. It is a bridge model from intrinsic functional architecture to stimulus-conditioned activation, with implications for rest-to-task mapping, scalable functional brain modeling, and subject-specific computational neuroscience (Zhou et al., 17 Sep 2025).