Brain-optimized inference improves reconstructions of fMRI brain activity (2312.07705v1)
Abstract: The release of large datasets and developments in AI have led to dramatic improvements in decoding methods that reconstruct seen images from human brain activity. We evaluate the prospect of further improving recent decoding methods by optimizing for consistency between reconstructions and brain activity during inference. We sample seed reconstructions from a base decoding method, then iteratively refine these reconstructions using a brain-optimized encoding model that maps images to brain activity. At each iteration, we sample a small library of images from an image distribution (a diffusion model) conditioned on a seed reconstruction from the previous iteration. We select those that best approximate the measured brain activity when passed through our encoding model, and use these images for structural guidance during the generation of the small library in the next iteration. We reduce the stochasticity of the image distribution at each iteration, and stop when a criterion on the "width" of the image distribution is met. We show that when this process is applied to recent decoding methods, it outperforms the base decoding method as measured by human raters, a variety of image feature metrics, and alignment to brain activity. These results demonstrate that reconstruction quality can be significantly improved by explicitly aligning decoding distributions to brain activity distributions, even when the seed reconstruction is output from a state-of-the-art decoding algorithm. Interestingly, the rate of refinement varies systematically across visual cortex, with earlier visual areas generally converging more slowly and preferring narrower image distributions, relative to higher-level brain areas. Brain-optimized inference thus offers a succinct and novel method for improving reconstructions and exploring the diversity of representations across visual brain areas.
- A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. In Nat Neurosci.
- Self-Supervised Natural Image Reconstruction and Large-Scale Semantic Classification from Brain Activity. NeuroImage, 254: 119121.
- Decoding natural image stimuli from fMRI data with a surface-based convolutional network. In Medical Imaging with Deep Learning.
- Compressive spatial summation in human visual cortex. Journal of Neurophysiology, 110(2): 481–494. PMID: 23615546.
- Reconstructing seen images from human brain activity via guided stochastic search. Conference on Cognitive Computational Neuroscience.
- Mind Reader: Reconstructing complex images from brain activities. In Oh, A. H.; Agarwal, A.; Belgrave, D.; and Cho, K., eds., Advances in Neural Information Processing Systems.
- Microsoft COCO: Common Objects in Context. In Fleet, D.; Pajdla, T.; Schiele, B.; and Tuytelaars, T., eds., Computer Vision – ECCV 2014, 740–755. Cham: Springer International Publishing. ISBN 978-3-319-10602-1.
- MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion. arXiv:2303.14139.
- Encoding and decoding in fMRI. NeuroImage, 56(2).
- Brain-Diffuser: Natural scene reconstruction from fMRI signals using generative latent diffusion. arXiv:2303.05334.
- Learning Transferable Visual Models From Natural Language Supervision. In Meila, M.; and Zhang, T., eds., Proceedings of the 38th International Conference on Machine Learning, volume 139, 8748–8763. PMLR.
- High-Resolution Image Synthesis with Latent Diffusion Models. CoRR, abs/2112.10752.
- Reconstructing the Mind’s Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors. In 37th Conference on Neural Information Processing Systems.
- Brain-optimized neural networks learn non-hierarchical models of representation in human visual cortex. bioRxiv.
- Generative Adversarial Networks Conditioned on Brain Activity Reconstruct Seen Images. In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 1054–1061. Institute of Electrical and Electronics Engineers Inc.
- High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14453–14463.
- Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputs. arXiv:2306.11536.
- Sensory uncertainty decoded from visual cortex predicts behavior. Nature neuroscience, 18.
- Versatile Diffusion: Text, Images and Variations All in One Diffusion Model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 7754–7765.