Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MindBridge: A Cross-Subject Brain Decoding Framework (2404.07850v1)

Published 11 Apr 2024 in cs.CV and cs.AI

Abstract: Brain decoding, a pivotal field in neuroscience, aims to reconstruct stimuli from acquired brain signals, primarily utilizing functional magnetic resonance imaging (fMRI). Currently, brain decoding is confined to a per-subject-per-model paradigm, limiting its applicability to the same individual for whom the decoding model is trained. This constraint stems from three key challenges: 1) the inherent variability in input dimensions across subjects due to differences in brain size; 2) the unique intrinsic neural patterns, influencing how different individuals perceive and process sensory information; 3) limited data availability for new subjects in real-world scenarios hampers the performance of decoding models. In this paper, we present a novel approach, MindBridge, that achieves cross-subject brain decoding by employing only one model. Our proposed framework establishes a generic paradigm capable of addressing these challenges by introducing biological-inspired aggregation function and novel cyclic fMRI reconstruction mechanism for subject-invariant representation learning. Notably, by cycle reconstruction of fMRI, MindBridge can enable novel fMRI synthesis, which also can serve as pseudo data augmentation. Within the framework, we also devise a novel reset-tuning method for adapting a pretrained model to a new subject. Experimental results demonstrate MindBridge's ability to reconstruct images for multiple subjects, which is competitive with dedicated subject-specific models. Furthermore, with limited data for a new subject, we achieve a high level of decoding accuracy, surpassing that of subject-specific models. This advancement in cross-subject brain decoding suggests promising directions for wider applications in neuroscience and indicates potential for more efficient utilization of limited fMRI data in real-world scenarios. Project page: https://littlepure2333.github.io/MindBridge

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1):116–126, 2022.
  2. Autoencoders. Machine learning for data science handbook: data mining and knowledge discovery handbook, pages 353–374, 2023.
  3. From voxels to pixels and back: Self-supervision in natural-image reconstruction from fmri. Advances in Neural Information Processing Systems, 32, 2019.
  4. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912–9924, 2020.
  5. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22710–22720, 2023.
  6. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021.
  7. fmri brain decoding and its applications in brain–computer interface: A survey. Brain Sciences, 12(2):228, 2022.
  8. Stable diffusion is unstable. Advances in Neural Information Processing Systems, 36, 2023.
  9. Structural pruning for diffusion models. In Advances in Neural Information Processing Systems, 2023.
  10. Deep learning. MIT press, 2016.
  11. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  12. Decoding natural image stimuli from fmri data with a surface-based convolutional network. arXiv preprint arXiv:2212.02409, 2022.
  13. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  14. A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of physiology, 117(4):500, 1952.
  15. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1):15037, 2017.
  16. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  17. Principles of neural science. McGraw-hill New York, 2000.
  18. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
  19. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  20. Minddiffuser: Controlled image reconstruction from human brain activity with semantic and structural diffusion. In Proceedings of the 31st ACM International Conference on Multimedia, pages 5899–5908, 2023.
  21. Deepcache: Accelerating diffusion models for free. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
  22. Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity. arXiv preprint arXiv:2308.07428, 2023.
  23. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–609, 1996.
  24. Brain-diffuser: Natural scene reconstruction from fmri signals using generative latent diffusion. arXiv preprint arXiv:2303.05334, 2023.
  25. Reconstruction of perceived images from fmri patterns and semantic brain exploration using instance-conditioned gans. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2022.
  26. Improving the accuracy of single-trial fmri response estimates using glmsingle. Elife, 11:e77599, 2022.
  27. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  28. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  29. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79–87, 1999.
  30. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  31. Learning representations by back-propagating errors. nature, 323(6088):533–536, 1986.
  32. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  33. Linear reconstruction of perceived images from human brain activity. NeuroImage, 83:951–961, 2013.
  34. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  35. Reconstructing the mind’s eye: fmri-to-image with contrastive learning and diffusion priors. arXiv preprint arXiv:2305.18274, 2023.
  36. Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage, 181:775–785, 2018.
  37. Deep image reconstruction from human brain activity. PLoS computational biology, 15(1):e1006633, 2019.
  38. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
  39. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  40. High-resolution image reconstruction with latent diffusion models from human brain activity. biorxiv. 2022.
  41. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  42. Reconstructing faces from fmri patterns using deep generative neural networks. Communications biology, 2(1):193, 2019.
  43. Sparse coding and decorrelation in primary visual cortex during natural vision. Science, 287(5456):1273–1276, 2000.
  44. Pangu-π𝜋\piitalic_π: Enhancing language model architectures via nonlinearity compensation. In arXiv:2312.17276, 2023.
  45. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004.
  46. Patch diffusion: Faster and more data-efficient training of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
  47. Dream: Visual decoding from reversing human visual system. arXiv preprint arXiv:2310.02265, 2023.
  48. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7754–7765, 2023.
  49. Diffusion model as representation learner. In IEEE/CVF International Conference on Computer Vision, 2023.
  50. Diffusion probabilistic model made slim. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  51. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
  52. Unipc: A unified predictor-corrector framework for fast sampling of diffusion models. arXiv preprint arXiv:2302.04867, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Shizun Wang (10 papers)
  2. Songhua Liu (33 papers)
  3. Zhenxiong Tan (14 papers)
  4. Xinchao Wang (203 papers)
Citations (10)
View on Semantic Scholar

Summary

An Overview of MindBridge: A Cross-Subject Brain Decoding Framework

The paper "MindBridge: A Cross-Subject Brain Decoding Framework" introduces a sophisticated model aimed at advancing the domain of brain decoding using functional magnetic resonance imaging (fMRI). Brain decoding, which endeavors to reconstruct stimuli from brain signal data, predominantly operates in a per-subject-per-model framework, which inherently limits the scalability and applicability of these approaches to broader populations. MindBridge addresses this limitation by proposing a single model capable of handling cross-subject data, thereby potentially broadening the application domain in practical settings like neuroscience and brain-computer interfaces.

Technical Insights and Methodology

The authors identify three significant challenges in the current paradigm of brain decoding: inter-subject variability in brain size, unique intrinsic neural patterns, and limited data availability for new subjects. To counter these, MindBridge leverages a biologically-inspired aggregation function and introduces a novel cyclic fMRI reconstruction mechanism to facilitate subject-invariant representation learning.

Key Methodologies Include:

  1. Adaptive Signal Aggregation: By incorporating neural-scientific principles about sparse neural activation, MindBridge employs adaptive max pooling to extract salient information and normalize the input dimension across different subjects. This technique plays a crucial role in dealing with the variability across individual brain structures.
  2. Subject-Invariant Representation: MindBridge proposes a unique method of subject-invariant representation learning using cyclic fMRI reconstruction, enabling the synthesis of novel fMRI data. This approach allows the model to learn generalized embeddings that persist across individual differences, aligning them within a consistent representational space.
  3. Reset-Tuning Finetuning Strategy: For scenarios involving new subjects with limited data, the paper introduces a reset-tuning approach. This method resets shallow layers while preserving deeper layers that contain transferable knowledge, facilitating more effective adaptation to new subjects.
  4. Versatile Diffusion Model Integration: MindBridge utilizes a multi-modal versatile diffusion (VD) model, driven by vector representations from both visual and textual data, allowing it to generate images that better capture semantic fidelity when reconstructing stimuli.

Empirical Validation

The paper demonstrates the efficacy of MindBridge using the Natural Scenes Dataset (NSD), which is composed of high-resolution fMRI data collected from multiple subjects. The experimental results reveal that MindBridge can achieve comparable performance to state-of-the-art methods that require subject-specific models, even when using a single model for all subjects. Moreover, it shows promising results in scenarios with novel subject adaptation, significantly reducing the amount of fMRI data needed.

Quantitative metrics, including PixCorr, SSIM, and CLIP similarity, along with qualitative assessments highlight MindBridge's capability to reconstruct images with high semantic accuracy and visual fidelity across multiple subjects. Additionally, the novelty of using cyclic fMRI reconstruction for novel fMRI synthesis underscores MindBridge's capacity to augment available data, surmounting data scarcity challenges.

Implications and Future Directions

The success of MindBridge in cross-subject brain decoding could redefine the utility of brain-computer interface applications, making them more versatile and accessible. By enabling accurate decoding with minimal data, the framework promises to decrease the overhead for training models on new subjects, which is critical for real-world application, especially in medical and assistive technology domains.

Future research could expand MindBridge’s applicability by integrating larger and more diverse datasets to assess its generalizability further. Additionally, ethical considerations and safeguards should accompany the advancement of such technology, given the privacy concerns associated with brain data.

In conclusion, MindBridge presents a significant step towards flexible, efficient, and scalable brain decoding, showcasing both methodological innovation and practical potential. Its success lays a foundation upon which further advancements can build, fostering a more inclusive and effective application of AI in neuroscience research.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets