NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagery (2506.06898v1)

Abstract: We release NSD-Imagery, a benchmark dataset of human fMRI activity paired with mental images, to complement the existing Natural Scenes Dataset (NSD), a large-scale dataset of fMRI activity paired with seen images that enabled unprecedented improvements in fMRI-to-image reconstruction efforts. Recent models trained on NSD have been evaluated only on seen image reconstruction. Using NSD-Imagery, it is possible to assess how well these models perform on mental image reconstruction. This is a challenging generalization requirement because mental images are encoded in human brain activity with relatively lower signal-to-noise and spatial resolution; however, generalization from seen to mental imagery is critical for real-world applications in medical domains and brain-computer interfaces, where the desired information is always internally generated. We provide benchmarks for a suite of recent NSD-trained open-source visual decoding models (MindEye1, MindEye2, Brain Diffuser, iCNN, Takagi et al.) on NSD-Imagery, and show that the performance of decoding methods on mental images is largely decoupled from performance on vision reconstruction. We further demonstrate that architectural choices significantly impact cross-decoding performance: models employing simple linear decoding architectures and multimodal feature decoding generalize better to mental imagery, while complex architectures tend to overfit visual training data. Our findings indicate that mental imagery datasets are critical for the development of practical applications, and establish NSD-Imagery as a useful resource for better aligning visual decoding methods with this goal.

• The paper introduces NSD-Imagery, a benchmark dataset that extends fMRI vision decoding methods to the domain of mental imagery.
• The paper evaluates various fMRI-to-image models, revealing that simpler linear and multimodal methods generalize better to mental imagery than complex architectures.
• The paper demonstrates that enhancements in visual decoding models can translate into accurate mental image reconstruction, with promising clinical and communication applications.

Analysis of NSD-Imagery Dataset for Extending fMRI Vision Decoding Methods

The paper presents NSD-Imagery, a novel benchmark dataset designed to facilitate the paper of mental imagery using fMRI data. This dataset is an expansion of the Natural Scenes Dataset (NSD), which previously catered to decoding seen images from fMRI signals. NSD-Imagery specifically focuses on enhancing the evaluation of models that decode mental imagery, a challenging task due to lower signal-to-noise ratios and spatial resolution compared to vision-based fMRI data.

Key Contributions

The authors highlight three fundamental contributions that NSD-Imagery offers to the field:

1. Introduction of NSD-Imagery Dataset: This dataset complements NSD by including fMRI data collected during mental imagery tasks performed by the same subjects. It allows researchers to assess the models' ability to generalize from seen to imagined images.
2. Evaluation of Vision-Decoding Models: Using NSD-Imagery, the authors evaluated state-of-the-art fMRI-to-image models on mental imagery tasks. They report that the performance on mental imagery is independent of vision decoding performance. Simple linear architectures and multimodal feature decoding methods exhibited better generalization capabilities than complex models, which tended to overfit.
3. Analysis of Model Behavior and Human Raters: Experimentation with human raters showed that some models could indeed generalize to mental imagery, which reflects potential for practical applications where mental image reconstruction is required.

Results and Implications

The paper details how the researchers conducted an extensive analysis, applying five fMRI-to-image models to both visions seen in NSD-Imagery trials and imagined stimuli. Among the models evaluated, MindEye1 and Brain Diffuser showed strong accuracy rates when decoding mental images, demonstrating their robustness in generalizing beyond training conditions aligned with vision tasks.

Challenges such as aligning the training distribution were identified, emphasizing the impact of a model’s prior on decoding performance. Importantly, this paper posits that improvements in vision decoding may translate into mental imagery decoding, although the architectural design is crucial for ensuring these improvements.

Future Directions and Ethical Considerations

The paper recognizes the limitations posed by the small size of mental imagery datasets, underscoring the need for larger datasets to fully exploit model training for mental imagery. In terms of broader impacts, the authors envisage applications in clinical settings, where decoding mental imagery can assist unresponsive patients or those with communication difficulties.

Additionally, the paper calls for the establishment of ethical guidelines to ensure responsible deployment of brain-decoding technologies, as these technologies advance and potentially transform diagnostic and therapeutic practices.

In summary, NSD-Imagery is pivotal for aligning the development of visual decoding models with real-world applications involving mental images. It provides a significant opportunity for future research into decoding internal representations from brain activity, with profound implications for human health and ethical deployment in practice.