NeuroSwift: Cross-Subject fMRI Decoding

• NeuroSwift is a cross-subject framework that decodes complex natural scenes from fMRI data by integrating low-level and high-level feature adapters.
• It employs a diffusion-based decoder with a frozen generative backbone to seamlessly merge structural and semantic information while requiring minimal fine-tuning.
• Benchmarked on the Natural Scenes Dataset, NeuroSwift outperforms previous methods in both accuracy and computational efficiency, setting a new standard in visual neural decoding.

NeuroSwift is a cross-subject framework for reconstructing complex natural scenes from fMRI brain activity, designed to operate effectively under inter-subject variability and computational constraints. It achieves high spatial and semantic accuracy by integrating low-level and high-level feature adapters with a frozen diffusion-based decoder, and requires minimal fine-tuning for new subjects. In benchmarking, NeuroSwift outperforms prior methods both in accuracy and computational efficiency on the Natural Scenes Dataset (NSD), establishing new standards for cross-subject visual neural decoding (Zhang et al., 2 Oct 2025).

1. Problem Formulation: Cross-Subject Visual Decoding

Decoding visual experiences from fMRI signals fundamentally faces anatomical and functional heterogeneity across subjects, which persists after standard spatial normalization. Voxel–feature mappings become blurred due to variations in sulcal/gyral morphology and individualized ROI boundaries. Additionally, natural scenes introduce complexity via clutter, occlusion, and dense object arrangements. Previous decoders, such as MindEye2 and MindTuner, often trade off between low-level structure and high-level semantics while requiring extensive computational resources—eight A100 GPUs or more—yet these typically underperform compared to single-subject approaches (Zhang et al., 2 Oct 2025).

2. NeuroSwift Architecture

NeuroSwift comprises two adapters that process fMRI signals into latent spaces exploited by state-of-the-art generative models, alongside fixed (pre-trained, frozen) generative backbones.

2.1 AutoKL Adapter: Low-Level Feature Extraction

• Input: Preprocessed voxel activations, masked by individualized occipital ROIs.
• Network: Two linear projections with LayerNorm, SiLU activation, and Dropout; followed by residual MLP stacks, upsampling, and GroupNorm to produce $z_{\mathrm{pred}}$, a latent spatial code aligned to AutoKL (the VAE backbone within Stable Diffusion).
• Loss: Mean squared error (MSE) in latent space:

$$\mathcal{L}_{\mathrm{AutoKL}} = \frac{1}{n}\sum_{i=1}^n \left\| z^{(i)}_{\mathrm{pred}} - z^{(i)}_{\mathrm{gt}} \right\|_2^2,$$

where $z_{\mathrm{gt}}$ is generated by the (frozen) AutoKL encoder from the image stimulus.

2.2 CLIP Adapter: High-Level Semantic Encoding

• Motivation: Emulate higher visual cortex semantic abstraction using joint image/text embeddings.
• Training Data: COCO captions as semantic text and Stable Diffusion–generated “semantic images” that highlight core object structure.
• Architecture: Same dual-projection and residual MLP backbone as AutoKL Adapter; outputs two vectors: $$e_{\mathrm{img\_pred}} \in \mathbb{R}^{d_{\mathrm{img}}}$$, $$e_{\mathrm{txt\_pred}} \in \mathbb{R}^{d_{\mathrm{txt}}}$$ matching CLIP embedding size.
• Losses:
  • Batchwise SoftCLIP contrastive loss:

  $$\mathcal{L}_{\mathrm{SoftCLIP}}(P, T) = -\sum_{i=1}^N\sum_{j=1}^N \frac{\exp(T_i \cdot T_j / \tau)}{\sum_m \exp(T_i \cdot T_m / \tau)} \log \frac{\exp(P_i \cdot T_j / \tau)}{\sum_m \exp(P_i \cdot T_m / \tau)}$$ - MSE regularization in embedding space; total objectives:

  $$\begin{aligned} \mathcal{L}_{\mathrm{CLIP\_img}} &= \mathcal{L}_{\mathrm{SoftCLIP}}\left(e_{\mathrm{img\_pred}}, e_{\mathrm{img\_clip}}\right) + \mathcal{L}_{\mathrm{MSE}}\left(e_{\mathrm{img\_pred}}, e_{\mathrm{img\_clip}}\right) \ \mathcal{L}_{\mathrm{CLIP\_txt}} &= \mathcal{L}_{\mathrm{SoftCLIP}}\left(e_{\mathrm{txt\_pred}}, e_{\mathrm{txt\_clip}}\right) + \mathcal{L}_{\mathrm{MSE}}\left(e_{\mathrm{txt\_pred}}, e_{\mathrm{txt\_clip}}\right) \end{aligned}$$

2.3 Diffusion Decoder: Structural and Semantic Fusion

The latent $z_{\mathrm{pred}}$ is combined with Gaussian noise at diffusion step $\tau = N-\lfloor sN \rfloor$:

$$z_\tau = \sqrt{\alpha_\tau} z_{\mathrm{pred}} + \sqrt{1-\alpha_\tau} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$

At each timestep $$\mathcal{L}_{\mathrm{AutoKL}} = \frac{1}{n}\sum_{i=1}^n \left\| z^{(i)}_{\mathrm{pred}} - z^{(i)}_{\mathrm{gt}} \right\|_2^2,$$0, the UNet denoiser (with frozen weights except for cross-attention) predicts $$\mathcal{L}_{\mathrm{AutoKL}} = \frac{1}{n}\sum_{i=1}^n \left\| z^{(i)}_{\mathrm{pred}} - z^{(i)}_{\mathrm{gt}} \right\|_2^2,$$1. The update is:

$$\mathcal{L}_{\mathrm{AutoKL}} = \frac{1}{n}\sum_{i=1}^n \left\| z^{(i)}_{\mathrm{pred}} - z^{(i)}_{\mathrm{gt}} \right\|_2^2,$$2

After denoising, $$\mathcal{L}_{\mathrm{AutoKL}} = \frac{1}{n}\sum_{i=1}^n \left\| z^{(i)}_{\mathrm{pred}} - z^{(i)}_{\mathrm{gt}} \right\|_2^2,$$3 is decoded by the frozen AutoKL decoder to produce a reconstructed image.

3. Training and Fine-Tuning Regimen

NeuroSwift implements an efficient cross-subject transfer protocol:

• Pretraining on a representative subject (Subject 01, full 40 h NSD: $$\mathcal{L}_{\mathrm{AutoKL}} = \frac{1}{n}\sum_{i=1}^n \left\| z^{(i)}_{\mathrm{pred}} - z^{(i)}_{\mathrm{gt}} \right\|_2^2,$$48859 images for training, 982 held out) for both adapters and diffusion interfaces.
• Transfer: For each new subject, only the fully connected layers in the adapters (about 17% of total model parameters) are fine-tuned, while all base components and the diffusion decoder are frozen.
• Computational Resources: Each subject-specific fine-tuning run executes in approximately one hour on three NVIDIA RTX 4090 GPUs. By comparison, MindEye2 and MindTuner require at least eight A100 GPUs or substantially higher compute (Zhang et al., 2 Oct 2025).

4. Dataset, Evaluation Metrics, and Empirical Performance

Dataset

• Source: Natural Scenes Dataset (NSD, 7T fMRI).
• Subjects: 01, 02, 05, 07.
• Voxels: Ventral visual cortex, with manually delineated, subject-specific ROIs (V1–V3, V4, V5/MT, ITC, FFA, PPC).

Evaluation Metrics

Metric Type	Metric Name	Purpose
Low-level	SSIM, PixCorr	Spatial similarity
Mid-level texture	AlexNet L2/L5	Texture reconstruction
High-level	CLIP, Inception, EffNet-B, SwAV	Semantic similarity

Results

Setting	PixCorr	SSIM	CLIP (%)	AlexNet(2) (%)
Cross-subject (1 h/3×4090)	0.253	0.431	85.9	90.7
MindTuner Cross-subject	0.224	—	—	—
MindEye2 Cross-subject	0.195	—	—	—
Single-subject (40 h/4090)	0.335	0.437	97.1	—

These results indicate that NeuroSwift achieves superior cross-subject reconstruction across both pixel-level and semantic-based metrics.

5. Pipeline Overview

The full fMRI-to-image pipeline is formalized as follows:

$$\mathcal{L}_{\mathrm{AutoKL}} = \frac{1}{n}\sum_{i=1}^n \left\| z^{(i)}_{\mathrm{pred}} - z^{(i)}_{\mathrm{gt}} \right\|_2^2,$$5

6. Limitations and Prospective Advances

NeuroSwift's empirical validation is limited to NSD; generalization to other datasets such as BOLD5000 remains untested. The reliance on manual ROI delineation facilitates performance but may inhibit scalability; future development of automated parcellation is indicated. Currently, only fMRI modality is supported, constraining temporal dynamics. Upgrades to generative backbones (e.g., latent transformers or fully multimodal diffusion) could enhance perceptual fidelity. Plans for cross-dataset transfer and domain adaptation (e.g., train on NSD, test on BOLD5000) are cited as essential for establishing cross-domain robustness.

In summary, NeuroSwift’s dual-adapter, diffusion-based paradigm combines fine-grained structural preservation and semantic enforcement with computationally efficient transfer, yielding state-of-the-art cross-subject fMRI visual decoding in substantially less compute time than prior art (Zhang et al., 2 Oct 2025).