Natural Scenes Dataset

• Natural Scenes Dataset is a large-scale fMRI collection capturing neural responses to 73,000 real-world images, providing a detailed mapping of visual cortical activity.
• It employs standardized data splits, region-of-interest mapping, and robust preprocessing, supporting rigorous evaluation of image-to-brain encoding models.
• The dataset accelerates research in neuroscience and AI by enabling the development and benchmarking of models that bridge image features with cortical representations.

The Natural Scenes Dataset (NSD) is a high-resolution, large-scale dataset of functional magnetic resonance imaging (fMRI) responses to thousands of real-world images, designed to advance computational modeling of the human visual brain. With approximately 73,000 unique naturalistic colored scenes drawn from the COCO database, and densely sampled neural recordings from eight subjects, NSD represents an unparalleled resource for both neuroscience and artificial intelligence research. Its scale, diversity of stimuli, and the quality of neural recordings make NSD an ideal foundation for developing and validating models that link image features to distributed cortical representations during natural vision (Gifford et al., 2023).

1. Composition and Data Structure

NSD contains high-field 7T fMRI data acquired while eight subjects viewed complex, natural images with central fixation. The core features of the dataset include:

• Stimulus Set: 73,000 separate images, each a naturalistic, colored scene from COCO.
• Subjects and Coverage: Each of eight subjects viewed unique subsets of the images. For each subject, tens of thousands of stimuli are associated with voxel-wise fMRI responses.
• Data Format: Preprocessed BOLD amplitudes are mapped to a standard cortical surface (FreeSurfer’s fsaverage), separated by hemisphere, and further divided by region-of-interest (ROI) indices associated with visual cortex subregions.
• Splits: The dataset is partitioned into non-overlapping training and test splits per subject, typically with ~8,000–9,800 images for training and 159–395 images for testing (the latter’s responses withheld for evaluation).

NSD’s structure is tailored to support both full-brain encoding analyses and more focused investigations restricted to high-level or early visual subregions.

2. Methodological Framework

NSD is explicitly organized for computational modeling—primarily, the training and evaluation of encoding models that map from image features to neural activity. The typical research pipeline includes:

• Model Inputs: RGB pixel data or higher-order features derived by deep neural networks, semantic segmentations, or other stimulus encodings.
• Model Outputs: Predicted BOLD response amplitudes per cortical vertex or in aggregate across ROIs.
• Evaluation Metric: The core challenge uses noise-normalized squared correlation,

$$\text{metric} = \mathrm{Mean}_v \left[ \frac{R_v^2}{NC_v} \right]$$

where $R_v$ is the Pearson correlation for vertex $v$ between the predicted and ground-truth responses on the test set, and $NC_v$ is the noise ceiling for that vertex. This metric quantifies the fraction of predictable neural variance explained by the computational model.

Such a framework enables rigorous benchmarking and comparison of models ranging from shallow linear encoders to advanced deep neural networks trained end-to-end on NSD distributions.

3. Experimental Protocol and Modeling Approaches

Subjects performed repeated scanning sessions under central fixation, ensuring high SNR and dense sampling per stimulus. Key elements of the experimental protocol and its implications include:

• Visual Stimulus Presentation: Each image was presented for a brief duration, with interleaved resting periods and randomization to minimize adaptation and anticipation.
• Data Preprocessing: BOLD responses were mapped onto a standardized surface, motion-corrected, temporally denoised, and partitioned into train-test splits.
• ROI Analysis: ROI indices enable restricting analyses to classic visual areas (e.g., V1, V2, V4, IT), facilitating targeted modeling of region-specific representations.
• Baseline Models: The Colab tutorial and default baseline utilize a pre-trained AlexNet to extract image features, followed by a linear mapping to fMRI responses, providing a benchmark noise-normalized metric of 40.48%.

Advanced models can incorporate architectures from both neuroscience (e.g., biologically inspired recurrent layers) and computer vision (e.g., vision transformers, contrastive pretraining), and may use the full cortical surface or ROI-masked data for parameter estimation.

4. Impact on Neuroscience and Artificial Intelligence

The NSD enables major progress in understanding the neural coding of complex visual information:

• For Neuroscience:
  • It facilitates identification of neural population codes underlying category, object, and semantic scene representations.
  • By covering a breadth of real-world stimuli, NSD allows for robust generalization analyses and hypothesis-driven modeling (e.g., testing hierarchical, feedforward, or recurrent neural architectures).
  • The unprecedented resolution and size enable probing intra- and inter-subject cortical mapping fidelity under natural viewing conditions.
• For AI Research:
  • NSD supplies a biologically grounded testbed for the development of encoding models and biologically inspired architectures.
  • Integrating neural data constrains and motivates AI inductive biases, offering guidance for architectures capable of generalization, robustness, and transfer learning.
  • The dataset supports the empirical evaluation of artificial neural networks’ similarity to brain computations at multiple representational levels.

Collaborative challenges such as the Algonauts Project directly leverage NSD for joint benchmarking, code and model sharing, and iterative improvement (Gifford et al., 2023).

5. Benchmarks, Leaderboards, and Community Practices

NSD is at the heart of open model development exemplified by the Algonauts Project:

• Public Leaderboard: Submissions are evaluated using the noise-normalized squared correlation metric, with scores automatically posted for transparency.
• Code and Data Release: Top models require public code release for reproducibility and peer evaluation.
• Conference Sessions: High-performing models are presented at conferences such as Cognitive Computational Neuroscience (CCN), fostering cross-disciplinary dialogue.
• Educational Resources: Tutorials and Colab notebooks lower the barrier for the broader scientific community.

Such mechanisms ensure transparency, enable comparative research, and promote rapid progress in computational neuroscience and biologically inspired AI.

6. Research Implications and Future Directions

The scale and richness of NSD are anticipated to yield multiple advances:

• Model Generalization and Robustness: The extensive test set and real-world image diversity allow probing the limits of generalization in both brain and machine models.
• Precision Mapping of Representations: By connecting image properties to distributed cortical activity, NSD enables new insights into the representational geometry of the visual system.
• Bridging AI and Brains: The dataset underpins new generations of AI models co-developed with neuroscientific constraints, potentially yielding more explainable and capable artificial visual systems.
• Extension to Other Modalities: NSD’s infrastructure and data model could be extended to multimodal stimuli or jointly analyzed with non-visual neuroimaging studies, enhancing its impact.

Long-term, results using NSD have the potential to deepen understanding of perception, drive the development of brain-constrained AI, and anchor interdisciplinary collaborations that unify theory, empirical data, and application.