Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 62 tok/s
Gemini 2.5 Pro 47 tok/s Pro
GPT-5 Medium 12 tok/s Pro
GPT-5 High 10 tok/s Pro
GPT-4o 91 tok/s Pro
Kimi K2 139 tok/s Pro
GPT OSS 120B 433 tok/s Pro
Claude Sonnet 4 31 tok/s Pro
2000 character limit reached

Wills Aligner: Multi-Subject Collaborative Brain Visual Decoding (2404.13282v2)

Published 20 Apr 2024 in cs.CV and cs.MM

Abstract: Decoding visual information from human brain activity has seen remarkable advancements in recent research. However, the diversity in cortical parcellation and fMRI patterns across individuals has prompted the development of deep learning models tailored to each subject. The personalization limits the broader applicability of brain visual decoding in real-world scenarios. To address this issue, we introduce Wills Aligner, a novel approach designed to achieve multi-subject collaborative brain visual decoding. Wills Aligner begins by aligning the fMRI data from different subjects at the anatomical level. It then employs delicate mixture-of-brain-expert adapters and a meta-learning strategy to account for individual fMRI pattern differences. Additionally, Wills Aligner leverages the semantic relation of visual stimuli to guide the learning of inter-subject commonality, enabling visual decoding for each subject to draw insights from other subjects' data. We rigorously evaluate our Wills Aligner across various visual decoding tasks, including classification, cross-modal retrieval, and image reconstruction. The experimental results demonstrate that Wills Aligner achieves promising performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience 25, 1 (2022), 116–126.
  2. BEiT: BERT Pre-Training of Image Transformers. In International Conference on Learning Representations.
  3. Deep Recurrent Encoder: an end-to-end network to model magnetoencephalography at scale. Neurons, Behavior, Data analysis, and Theory (2022).
  4. Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 22710–22720.
  5. Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity. In Advances in Neural Information Processing Systems.
  6. Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature. Neuroimage 53, 1 (2010), 1–15.
  7. Decoding visual neural representations by multimodal learning of brain-visual-linguistic features. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
  8. GLaM: Efficient Scaling of Language Models with Mixture-of-Experts. In International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162). 5547–5569.
  9. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research 23 (2022), 120:1–120:39.
  10. Through their eyes: multi-subject Brain Decoding with simple alignment techniques. arXiv preprint arXiv:2309.00627 (2023).
  11. High-resolution intersubject averaging and a coordinate system for the cortical surface. Human brain mapping 8, 4 (1999), 272–284.
  12. Lite-Mind: Towards Efficient and Versatile Brain Representation Network. CoRR abs/2312.03781 (2023).
  13. Hyperalignment: Modeling shared information encoded in idiosyncratic cortical topographies. elife 9 (2020), e56601.
  14. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
  15. Tomoyasu Horikawa and Yukiyasu Kamitani. 2017. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications 8, 1 (2017), 15037.
  16. Lora: Low-rank adaptation of large language models. International Conference on Learning Representation.
  17. Bridging low-level features and high-level semantics via fMRI brain imaging for video classification. In Proceedings of the 18th ACM International Conference on Multimedia. 451–460.
  18. Yukiyasu Kamitani and Frank Tong. 2005. Decoding the visual and subjective contents of the human brain. Nature neuroscience 8, 5 (2005), 679–685.
  19. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. In International Conference on Learning Representations. OpenReview.net.
  20. Mind reader: Reconstructing complex images from brain activities. Advances in Neural Information Processing Systems 35 (2022), 29624–29636.
  21. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision, Vol. 8693. 740–755.
  22. David Linden. 2021. Section 3 - Introduction. In fMRI Neurofeedback, Michelle Hampson (Ed.). Academic Press, 161–169. https://doi.org/10.1016/B978-0-12-822421-2.00008-9
  23. BrainCLIP: Bridging Brain and Visual-Linguistic Representation via CLIP for Generic Natural Visual Stimulus Decoding from fMRI. arXiv preprint arXiv:2302.12971 (2023).
  24. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. International Conference on Learning Representation.
  25. MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion. In Proceedings of the 31st ACM International Conference on Multimedia. 5899–5908.
  26. Encoding and decoding in fMRI. Neuroimage 56, 2 (2011), 400–410.
  27. BMI-Net: A Brain-inspired Multimodal Interaction Network for Image Aesthetic Assessment. In Proceedings of the 31st ACM International Conference on Multimedia. 5514–5522.
  28. Furkan Ozcelik and Rufin VanRullen. 2023. Brain-diffuser: Natural scene reconstruction from fmri signals using generative latent diffusion. arXiv preprint arXiv:2303.05334 (2023).
  29. Decoding brain representations by multimodal learning of neural activity and visual features. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 11 (2020), 3833–3849.
  30. Neural Networks for Efficient Bayesian Decoding of Natural Images from Retinal Neurons. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems. 6434–6445.
  31. Toward a universal decoder of linguistic meaning from brain activation. Nature communications 9, 1 (2018), 963.
  32. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748–8763.
  33. Global Filter Networks for Image Classification. In Advances in Neural Information Processing Systems (NeurIPS).
  34. GFNet: Global Filter Networks for Visual Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 45, 9 (2023), 10960–10973.
  35. Scaling Vision with Sparse Mixture of Experts. In Advances in Neural Information Processing Systems. 8583–8595.
  36. Mind reading and writing: the future of neurotechnology. Trends in cognitive sciences 22, 7 (2018), 598–610.
  37. Reconstructing the Mind’s Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors. arXiv preprint arXiv:2305.18274 (2023).
  38. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In 5th International Conference on Learning Representations. OpenReview.net.
  39. Deep image reconstruction from human brain activity. PLoS computational biology 15, 1 (2019), e1006633.
  40. Yu Takagi and Shinji Nishimoto. 2023. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14453–14463.
  41. Aligning individual brains with fused unbalanced Gromov Wasserstein. Advances in neural information processing systems 35 (2022), 21792–21804.
  42. The WU-Minn human connectome project: an overview. Neuroimage 80 (2013), 62–79.
  43. Attention is all you need. Advances in neural information processing systems 30 (2017).
  44. Brain state decoding for rapid image retrieval. In Proceedings of the 17th ACM International Conference on Multimedia. 945–954.
  45. Mixture-of-Experts Learner for Single Long-Tailed Domain Generalization. In Proceedings of the 31st ACM International Conference on Multimedia. 290–299.
  46. Image as a foreign language: BEiT pretraining for vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  47. Decoding pixel-level image features from two-photon calcium signals of macaque visual cortex. Neural Computation 34, 6 (2022), 1369–1397.
  48. CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding. (2024).
  49. Adaptive Mixture of Experts Learning for Generalizable Face Anti-Spoofing. In Proceedings of the 30th ACM International Conference on Multimedia.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb On Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 2 posts and received 0 likes.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up to View