Brain-Grasp: Graph-based Saliency Priors for Improved fMRI-based Visual Brain Decoding

Abstract: Recent progress in brain-guided image generation has improved the quality of fMRI-based reconstructions; however, fundamental challenges remain in preserving object-level structure and semantic fidelity. Many existing approaches overlook the spatial arrangement of salient objects, leading to conceptually inconsistent outputs. We propose a saliency-driven decoding framework that employs graph-informed saliency priors to translate structural cues from brain signals into spatial masks. These masks, together with semantic information extracted from embeddings, condition a diffusion model to guide image regeneration, helping preserve object conformity while maintaining natural scene composition. In contrast to pipelines that invoke multiple diffusion stages, our approach relies on a single frozen model, offering a more lightweight yet effective design. Experiments show that this strategy improves both conceptual alignment and structural similarity to the original stimuli, while also introducing a new direction for efficient, interpretable, and structurally grounded brain decoding.

• The paper introduces a novel GNN-based method to predict saliency priors directly from CLIP–fMRI embeddings, boosting structural and semantic fidelity in reconstructions.
• It integrates orthogonal 'what' (semantic) and 'where' (spatial) cues via a streamlined diffusion model, outperforming previous pipelines on metrics like PixCorr, SSIM, and CLIP similarity.
• Ablation studies showcase that both saliency and textual priors are essential, setting a new benchmark for efficient, subject-specific fMRI-based visual brain decoding.

Brain-GraSP: Graph-based Saliency Priors for Improved fMRI-based Visual Brain Decoding

Introduction

Brain-GraSP introduces a saliency-driven approach to fMRI-based visual brain decoding (VBD) that addresses persistent limitations in preserving object-level structure and semantic fidelity in reconstructed images. Building upon the computational backbone of precomputed CLIP–fMRI embeddings (from MindEye), the framework incorporates graph neural network (GNN)-based saliency priors and semantic cues extracted from embeddings to condition a single-stage diffusion-based generative model. This design enables efficient, subject-specific decoding that outperforms prior pipelines in both structural and semantic metrics.

Figure 1: The Brain-GraSP architecture incorporating saliency priors and textual cues for fMRI-based visual reconstruction.

Technical Contributions

Brain-GraSP’s primary technical advance is the use of GNNs to predict saliency maps directly from fMRI–CLIP embeddings, bypassing reliance on downstream (often artifact-prone) image generations for spatial priors. These embeddings ($E \in \mathbb{R}^{257 \times 768}$) are modeled as nodes in a graph, with learned adjacency reflecting spatial or semantic proximity. Among the tested architectures—GCN, GAT, and GraphSAGE—the inductive representation power of GraphSAGE yielded the highest fidelity saliency masks, as established across multiple metrics on the NSD dataset.

The predicted saliency map $S(x, y)$ reflects the spatial distribution of object importance as informed by brain activity, providing pre-synthesis spatial conditioning. Semantic cues are separately extracted by mapping the embeddings to CLIP’s text-aligned space, yielding descriptors used for prompt conditioning in the diffusion model. This orthogonal extraction of "what" (semantics) and "where" (saliency) cues from a shared embedding ensures conceptual and structural priors are mutually compatible.

Conditioning is operationalized with a frozen Stable Diffusion backbone extended by an IP-Adapter, which injects both the fMRI-derived embedding and auxiliary priors. This modular approach avoids retraining large-scale generative models and maximizes efficiency, supporting scalable analysis across subjects.

Experimental Analysis

Brain-GraSP was evaluated against competing pipelines (MindEye, MindBridge, and BOI) using a controlled subset of the NSD dataset (681 images, 4 subjects). The performance was assessed using complementary metrics: PixCorr (correlation for low-level fidelity), SSIM (perceptual structure), high-level visual features (AlexNet, EfficientNet, SwAV), Inception Score, and CLIP similarity for semantic alignment. Quantitative results show that Brain-GraSP achieves the best performance on PixCorr (0.326), SSIM (0.299), Inception (97.85%), CLIP (98.80%), and SwAV (0.315), establishing advantages in both structure and semantics.

Ablation studies on Subject 1 confirm that both saliency and textual priors materially enhance performance. Removing saliency priors results in substantial drops in PixCorr and SSIM, highlighting the critical role of spatial cues injected upstream of image generation. Notably, leveraging MindEye’s embeddings within the more structured pipeline of Brain-GraSP yields higher scores than the original MindEye model, underscoring the architectural improvements rather than gains from richer embeddings alone.

Qualitative evaluation demonstrates that Brain-GraSP reconstructions reliably preserve object localization and semantic interpretation, often correcting distortions seen in prior models.

Figure 2: Qualitative comparison of reconstructions from Brain-GraSP, MindEye, MindBridge, and BOI, showing enhanced object-level fidelity and semantic alignment for Brain-GraSP.

Implications and Future Directions

Brain-GraSP’s integration of GNN-based saliency priors sets a new benchmark for leveraging structural signals in VBD, emphasizing the necessity of modular pipelines that allow spatial and semantic cues to be manipulated orthogonally. This paradigm supports efficiency (via reuse of upstream embeddings and frozen generative models), interpretability (through explicit spatial priors), and extensibility to new decoding hypotheses.

Practically, subject-specific optimization of the saliency module unlocks high-resolution personalization—potentially enabling studies of individual neural representations of spatial attention and object saliency. The separation of "what" and "where" priors facilitates targeted analysis of failures in object/scene reconstructions, a current challenge in applied neuro-AI.

Theoretically, this work motivates future decoding frameworks to exploit mid-level representations for priors, minimize generative artifacts, and utilize modular, replaceable components for sustainable AI. The GNN-based saliency modeling also opens research into more robust, interpretable graph representations fitted directly to neural data.

Anticipated next steps include training with human-annotated saliency masks (surpassing the limitations of EDN-generated proxies), exploring more expressive GNNs, and extending the multi-prior paradigm to other sensory domains or generative tasks.

Conclusion

Brain-GraSP demonstrates that graph-derived saliency priors, paired with semantic cues, substantially improve the structural and conceptual fidelity of fMRI-based visual reconstructions. By decoupling spatial and semantic priors and integrating them using a computationally lightweight, modular pipeline, the framework achieves state-of-the-art results and establishes a robust foundation for future research into interpretable, efficient brain–image decoding.