Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Decoding Natural Images from EEG for Object Recognition (2308.13234v3)

Published 25 Aug 2023 in cs.HC, cs.AI, eess.SP, and q-bio.NC

Abstract: Electroencephalography (EEG) signals, known for convenient non-invasive acquisition but low signal-to-noise ratio, have recently gained substantial attention due to the potential to decode natural images. This paper presents a self-supervised framework to demonstrate the feasibility of learning image representations from EEG signals, particularly for object recognition. The framework utilizes image and EEG encoders to extract features from paired image stimuli and EEG responses. Contrastive learning aligns these two modalities by constraining their similarity. With the framework, we attain significantly above-chance results on a comprehensive EEG-image dataset, achieving a top-1 accuracy of 15.6% and a top-5 accuracy of 42.8% in challenging 200-way zero-shot tasks. Moreover, we perform extensive experiments to explore the biological plausibility by resolving the temporal, spatial, spectral, and semantic aspects of EEG signals. Besides, we introduce attention modules to capture spatial correlations, providing implicit evidence of the brain activity perceived from EEG data. These findings yield valuable insights for neural decoding and brain-computer interfaces in real-world scenarios. The code will be released on https://github.com/eeyhsong/NICE-EEG.

View on arXiv
Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. Object classification from randomized EEG trials. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  3844–3853, Nashville, TN, USA, June 2021. IEEE. ISBN 978-1-66544-509-2. doi: 10.1109/CVPR46437.2021.00384.
  2. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1):116–126, January 2022. ISSN 1546-1726. doi: 10.1038/s41593-021-00962-x.
  3. A map of object space in primate inferotemporal cortex. Nature, 583(7814):103–108, July 2020. ISSN 1476-4687. doi: 10.1038/s41586-020-2350-5.
  4. Visual Areas Exert Feedforward and Feedback Influences through Distinct Frequency Channels. Neuron, 85(2):390–401, January 2015. ISSN 0896-6273. doi: 10.1016/j.neuron.2014.12.018.
  5. Brain decoding: Toward real-time reconstruction of visual perception, October 2023.
  6. How Attentive are Graph Attention Networks?, January 2022.
  7. VigilanceNet: Decouple Intra- and Inter-Modality Learning for Multimodal Vigilance Estimation in RSVP-Based BCI. In Proceedings of the 30th ACM International Conference on Multimedia, pp.  209–217, Lisboa Portugal, October 2022. ACM. ISBN 978-1-4503-9203-7. doi: 10.1145/3503161.3548367.
  8. A M/EEG-fMRI Fusion Primer: Resolving Human Brain Responses in Space and Time. Neuron, 107(5):772–781, September 2020. ISSN 0896-6273. doi: 10.1016/j.neuron.2020.07.001.
  9. Resolving human object recognition in space and time. Nature Neuroscience, 17(3):455–462, March 2014. ISSN 1546-1726. doi: 10.1038/nn.3635.
  10. Fast and accurate deep network learning by exponential linear units (elus). In International Conference on Learning Representations, 2016.
  11. Aligning Model and Macaque Inferior Temporal Cortex Representations Improves Model-to-Human Behavioral Alignment and Adversarial Robustness. In The Eleventh International Conference on Learning Representations, February 2023.
  12. Untangling invariant object recognition. Trends in Cognitive Sciences, 11(8):333–341, August 2007. ISSN 1364-6613. doi: 10.1016/j.tics.2007.06.010.
  13. LGGNet: Learning From Local-Global-Graph Representations for Brain–Computer Interface. IEEE Transactions on Neural Networks and Learning Systems, pp.  1–14, 2023. ISSN 2162-2388. doi: 10.1109/TNNLS.2023.3236635.
  14. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, April 2021.
  15. Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp.  1–17, 2023. ISSN 1939-3539. doi: 10.1109/TPAMI.2023.3263181.
  16. Finding Gamma. Neuron, 58(3):303–305, May 2008. ISSN 0896-6273. doi: 10.1016/j.neuron.2008.04.020.
  17. Multi-View Multi-Label Fine-Grained Emotion Decoding From Human Brain Activity. IEEE Transactions on Neural Networks and Learning Systems, pp.  1–15, 2022. ISSN 2162-2388. doi: 10.1109/TNNLS.2022.3217767.
  18. Interface, interaction, and intelligence in generalized brain–computer interfaces. Trends in Cognitive Sciences, 25(8):671–684, August 2021. ISSN 1364-6613. doi: 10.1016/j.tics.2021.04.003.
  19. A large and rich EEG dataset for modeling human visual object recognition. NeuroImage, 264:119754, December 2022. ISSN 1053-8119. doi: 10.1016/j.neuroimage.2022.119754.
  20. Multivariate pattern analysis for MEG: A comparison of dissimilarity measures. NeuroImage, 173:434–447, June 2018. ISSN 1053-8119. doi: 10.1016/j.neuroimage.2018.02.044.
  21. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.  770–778, Las Vegas, NV, USA, June 2016. IEEE. ISBN 978-1-4673-8851-1. doi: 10.1109/CVPR.2016.90.
  22. THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. eLife, 12:e82580, February 2023. ISSN 2050-084X. doi: 10.7554/eLife.82580.
  23. C. S. Herrmann and T. Demiralp. Human EEG gamma oscillations in neuropsychiatric disorders. Clinical Neurophysiology, 116(12):2719–2733, December 2005. ISSN 1388-2457. doi: 10.1016/j.clinph.2005.07.007.
  24. Inter-individual deep image reconstruction via hierarchical neural code conversion. NeuroImage, 271:120007, May 2023. ISSN 1053-8119. doi: 10.1016/j.neuroimage.2023.120007.
  25. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, 8(1):15037, May 2017. ISSN 2041-1723. doi: 10.1038/ncomms15037.
  26. Unsupervised changes in core object recognition behavior are predicted by neural plasticity in inferior temporal cortex. eLife, 10:e60830, June 2021. ISSN 2050-084X. doi: 10.7554/eLife.60830.
  27. Decoding the visual and subjective contents of the human brain. Nature Neuroscience, 8(5):679–685, May 2005. ISSN 1546-1726. doi: 10.1038/nn1444.
  28. Identifying natural images from human brain activity. Nature, 452(7185):352–355, March 2008. ISSN 1476-4687. doi: 10.1038/nature06713.
  29. Visual masking and RSVP reveal neural competition. Trends in Cognitive Sciences, 6(3):120–125, March 2002. ISSN 1364-6613. doi: 10.1016/S1364-6613(00)01852-0.
  30. SPD domain-specific batch normalization to crack interpretable unsupervised domain adaptation in EEG. Advances in Neural Information Processing Systems, 35:6219–6235, December 2022.
  31. EEGNet: A compact convolutional neural network for EEG-based brain–computer interfaces. Journal of Neural Engineering, 15(5):056013, July 2018. ISSN 1741-2552. doi: 10.1088/1741-2552/aace8c.
  32. The Perils and Pitfalls of Block Design for EEG Classification Experiments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1):316–333, January 2021. ISSN 1939-3539. doi: 10.1109/TPAMI.2020.2973153.
  33. Mind Reader: Reconstructing complex images from brain activities. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp.  29624–29636. Curran Associates, Inc., 2022.
  34. Improving the Performance of Individually Calibrated SSVEP-BCI by Task- Discriminant Component Analysis. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 29:1998–2007, 2021. ISSN 1558-0210. doi: 10.1109/TNSRE.2021.3114340.
  35. Brain-Machine Coupled Learning Method for Facial Emotion Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp.  1–15, 2023. ISSN 1939-3539. doi: 10.1109/TPAMI.2023.3257846.
  36. Alpha-Beta and Gamma Rhythms Subserve Feedback and Feedforward Influences among Human Visual Cortical Areas. Neuron, 89(2):384–397, January 2016. ISSN 0896-6273. doi: 10.1016/j.neuron.2015.12.018.
  37. Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders. Neuron, 60(5):915–929, December 2008. ISSN 0896-6273. doi: 10.1016/j.neuron.2008.11.004.
  38. Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11):3833–3849, November 2021. ISSN 0162-8828. doi: 10.1109/TPAMI.2020.2995909.
  39. MAtt: A Manifold Attention Network for EEG Decoding. Advances in Neural Information Processing Systems, 35:31116–31129, December 2022.
  40. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748–8763. PMLR, 2021.
  41. Photorealistic text-to-image diffusion models with deep language understanding. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp.  36479–36494. Curran Associates, Inc., 2022.
  42. Identification of competing neural mechanisms underlying positive and negative perceptual hysteresis in the human visual system. NeuroImage, 221:117153, November 2020. ISSN 1053-8119. doi: 10.1016/j.neuroimage.2020.117153.
  43. Deep learning with convolutional neural networks for EEG decoding and visualization. Human Brain Mapping, 38(11):5391–5420, 2017. ISSN 1097-0193. doi: 10.1002/hbm.23730.
  44. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):360–368, May 2023. ISSN 1476-4687. doi: 10.1038/s41586-023-06031-6.
  45. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In 2017 IEEE International Conference on Computer Vision (ICCV), pp.  618–626, October 2017. doi: 10.1109/ICCV.2017.74.
  46. Deep image reconstruction from human brain activity. PLOS Computational Biology, 15(1):e1006633, 2019. ISSN 1553-7358. doi: 10.1371/journal.pcbi.1006633.
  47. Representative-Based Cold Start for Adaptive SSVEP-BCI. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:1521–1531, 2023. ISSN 1558-0210. doi: 10.1109/TNSRE.2023.3245654.
  48. EEG Conformer: Convolutional Transformer for EEG Decoding and Visualization. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:710–719, 2023. ISSN 1558-0210. doi: 10.1109/TNSRE.2022.3230250.
  49. Deep Learning Human Mind for Automated Visual Classification. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.  4503–4511, July 2017. doi: 10.1109/CVPR.2017.479.
  50. Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. 2023 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  51. Representation Learning with Contrastive Predictive Coding, January 2019.
  52. Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research, 9(86):2579–2605, 2008. ISSN 1533-7928.
  53. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  54. Graph Attention Networks. In International Conference on Learning Representations, April 2018.
  55. OmniVL: One foundation model for image-language and video-language tasks. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp.  5696–5710. Curran Associates, Inc., 2022.
  56. High-performance brain-to-text communication via handwriting. Nature, 593(7858):249–254, May 2021. ISSN 1476-4687. doi: 10.1038/s41586-021-03506-2.
  57. A temporal hierarchy of object processing in human visual cortex, September 2023.
  58. Transient Induced Gamma-Band Response in EEG as a Manifestation of Miniature Saccades. Neuron, 58(3):429–441, May 2008. ISSN 0896-6273. doi: 10.1016/j.neuron.2008.03.027.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Yonghao Song (13 papers)
  2. Bingchuan Liu (4 papers)
  3. Xiang Li (1003 papers)
  4. Nanlin Shi (4 papers)
  5. Yijun Wang (60 papers)
  6. Xiaorong Gao (9 papers)
Citations (12)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now