Deep Learning for Visual Neuroprosthesis (2401.03639v1)
Abstract: The visual pathway involves complex networks of cells and regions which contribute to the encoding and processing of visual information. While some aspects of visual perception are understood, there are still many unanswered questions regarding the exact mechanisms of visual encoding and the organization of visual information along the pathway. This chapter discusses the importance of visual perception and the challenges associated with understanding how visual information is encoded and represented in the brain. Furthermore, this chapter introduces the concept of neuroprostheses: devices designed to enhance or replace bodily functions, and highlights the importance of constructing computational models of the visual pathway in the implementation of such devices. A number of such models, employing the use of deep learning models, are outlined, and their value to understanding visual coding and natural vision is discussed.
- Toward the Next Generation of Retinal Neuroprosthesis: Visual Computation with Spikes. Engineering. 2020;6(4):449–461. Available from: \urlhttps://www.sciencedirect.com/science/article/pii/S2095809920300357.
- Nirenberg S, Pandarinath C. Retinal prosthetic strategy with the capacity to restore normal vision. Proceedings of the National Academy of Sciences of the United States of America. 2012 Sep;109:15012–15017.
- A high-performance neural prosthesis enabled by control algorithm design. Nature neuroscience. 2012 Dec;15:1752–1757.
- Seeber BU, Bruce IC. The history and future of neural modeling for cochlear implants. Network (Bristol, England). 2016;27:53–66.
- Inference of neuronal functional circuitry with spike-triggered non-negative matrix factorization. Nature Communications. 2017 Jul;8(1):149. Publisher: Springer Nature.
- Simple model for encoding natural images by retinal ganglion cells with nonlinear spatial integration. PLoS computational biology. 2022 Mar;18:e1009925.
- Gollisch T, Meister M. Eye smarter than scientists believed: neural computations in circuits of the retina. Neuron. 2010 Jan;65:150–164.
- Retinal bipolar cells: elementary building blocks of vision. Nature reviews Neuroscience. 2014 Aug;15:507–519.
- Liu JK, Gollisch T. Spike-Triggered Covariance Analysis Reveals Phenomenological Diversity of Contrast Adaptation in the Retina. PLoS Computational Biology. 2015 Jul;11(7):e1004425. Publisher: Public Library of Science (PLoS).
- Grill-Spector K, Malach R. The human visual cortex. Annual review of neuroscience. 2004;27:649–677.
- DeYoe EA, Van Essen DC. Concurrent processing streams in monkey visual cortex. Trends in neurosciences. 1988 May;11:219–226.
- Object vision and spatial vision: two cortical pathways. Trends in Neurosciences. 1983;6:414–417. Available from: \urlhttps://www.sciencedirect.com/science/article/pii/016622368390190X.
- A neurological dissociation between perceiving objects and grasping them. Nature. 1991 Jan;349:154–156.
- Retinal Ganglion Cells-Diversity of Cell Types and Clinical Relevance. Frontiers in neurology. 2021;12:661938.
- Neuroscience. 2nd edition. Sunderland (MA): Sinauer Associates; 2001. Available from: \urlhttps://www.ncbi.nlm.nih.gov/books/NBK10885/.
- Kaneko A. Physiological and morphological identification of horizontal, bipolar and amacrine cells in goldfish retina. The Journal of physiology. 1970 May;207:623–633.
- Versatile functional roles of horizontal cells in the retinal circuit. Scientific reports. 2017 Jul;7:5540.
- Kaneko A. Retinal bipolar cells: their function and morphology. Trends in Neurosciences. 1983;6:219–223. Available from: \urlhttps://www.sciencedirect.com/science/article/pii/016622368390098X.
- Tsukamoto Y, Omi N. Classification of Mouse Retinal Bipolar Cells: Type-Specific Connectivity with Special Reference to Rod-Driven AII Amacrine Pathways. Frontiers in neuroanatomy. 2017;11:92.
- Spikes in retinal bipolar cells phase-lock to visual stimuli with millisecond precision. Current biology : CB. 2011 Nov;21:1859–1869.
- Kuffler SW. Discharge patterns and functional organization of mammalian retina. Journal of neurophysiology. 1953 Jan;16:37–68.
- StatPearls. StatPearls Publishing; 2022.
- StatPearls. 2022 Jan;.
- Hubel DH, Wiesel TN. Receptive fields of single neurones in the cat’s striate cortex. The Journal of physiology. 1959 Oct;148:574–591.
- Hubel DH, Wiesel TN. Receptive fields and functional architecture of monkey striate cortex. The Journal of physiology. 1968 Mar;195:215–243.
- How does the brain solve visual object recognition? Neuron. 2012 Feb;73:415–434.
- Peterhans E, von der Heydt R. Subjective contours–bridging the gap between psychophysics and physiology. Trends in neurosciences. 1991 Mar;14:112–119.
- Van Essen DC. Visual areas of the mammalian cerebral cortex. Annual review of neuroscience. 1979;2:227–263.
- Ziemba CM, Freeman J. Representing ”stuff” in visual cortex. Proceedings of the National Academy of Sciences of the United States of America. 2015 Jan;112:942–943.
- Image statistics underlying natural texture selectivity of neurons in macaque V4. Proceedings of the National Academy of Sciences of the United States of America. 2015 Jan;112:E351–E360.
- Fast readout of object identity from macaque inferior temporal cortex. Science (New York, NY). 2005 Nov;310:863–866.
- Stimulus-selective properties of inferior temporal neurons in the macaque. The Journal of neuroscience : the official journal of the Society for Neuroscience. 1984 Aug;4:2051–2062.
- Shape representation in the inferior temporal cortex of monkeys. Current biology : CB. 1995 May;5:552–563.
- Kaas JH, Lyon DC. Visual cortex organization in primates: theories of V3 and adjoining visual areas. Progress in brain research. 2001;134:285–295.
- Nassi JJ, Callaway EM. Parallel processing strategies of the primate visual system. Nature reviews Neuroscience. 2009 May;10:360–372.
- Neuroimaging weighs in: humans meet macaques in ”primate” visual cortex. The Journal of neuroscience : the official journal of the Society for Neuroscience. 2003 May;23:3981–3989.
- Koulakov AA, Chklovskii DB. Orientation Preference Patterns in Mammalian Visual Cortex: A Wire Length Minimization Approach. Neuron. 2001;29(2):519–527. Available from: \urlhttps://www.sciencedirect.com/science/article/pii/S0896627301002239.
- Orientation selectivity without orientation maps in visual cortex of a highly visual mammal. The Journal of neuroscience : the official journal of the Society for Neuroscience. 2005 Jan;25:19–28.
- Causal Influence of Visual Cues on Hippocampal Directional Selectivity. Cell. 2016 Jan;164:197–207.
- Brette R. Philosophy of the Spike: Rate-Based vs. Spike-Based Theories of the Brain. Frontiers in Systems Neuroscience. 2015;9. Available from: \urlhttps://www.frontiersin.org/articles/10.3389/fnsys.2015.00151.
- Information redundancy across spatial scales modulates early visual cortical processing. NeuroImage. 2021;244:118613. Available from: \urlhttps://www.sciencedirect.com/science/article/pii/S1053811921008867.
- Functional magnetic resonance imaging (FMRI) of the human brain. Journal of neuroscience methods. 1994 Oct;54:171–187.
- Reconstructing visual experiences from brain activity evoked by natural movies. Current biology : CB. 2011 Oct;21:1641–1646.
- In vivo two-photon calcium imaging of neuronal networks. Proceedings of the National Academy of Sciences of the United States of America. 2003 Jun;100:7319–7324.
- A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nature neuroscience. 2020 Jan;23:138–151.
- Electroencephalography. Current biology : CB. 2019 Feb;29:R80–R85.
- Survey of spiking in the mouse visual system reveals functional hierarchy. Nature. 2021 Apr;592:86–92.
- Deep learning. Nature. 2015;521(7553):436–444.
- Shah NP, Chichilnisky EJ. Computational challenges and opportunities for a bi-directional artificial retina. Journal of Neural Engineering. 2020 Oct;17(5):055002. Publisher: IOP Publishing.
- High-Fidelity Reproduction of Visual Signals by Electrical Stimulation in the Central Primate Retina. The Journal of Neuroscience. 2023 may;43(25):4625–4641.
- Neural Decoding of Visual Information Across Different Neural Recording Modalities and Approaches. Machine Intelligence Research. 2022 jul;19(5):350–365.
- Revealing Fine Structures of the Retinal Receptive Field by Deep-Learning Networks. IEEE Transactions on Cybernetics. 2022 jan;52(1):39–50.
- Unraveling neural coding of dynamic natural visual scenes via convolutional recurrent neural networks. Patterns. 2021 Oct;2(10):100350. Publisher: Elsevier BV.
- Horikawa T, Kamitani Y. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications. 2017;8(1):15037.
- Convolutional neural network-based encoding and decoding of visual object recognition in space and time. NeuroImage. 2018;180:253–266.
- Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral cortex. 2018;28(12):4136–4160.
- Category decoding of visual stimuli from human brain activity using a bidirectional recurrent neural network to simulate bidirectional information flows in human visual cortices. Frontiers in neuroscience. 2019;13:692.
- Decoding pixel-level image features from two-photon calcium signals of macaque visual cortex. Neural Computation. 2022;34(6):1369–1397.
- Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior. Nature neuroscience. 2019;22(6):974–983.
- Recurrence is required to capture the representational dynamics of the human visual system. Proceedings of the National Academy of Sciences. 2019;116(43):21854–21863.
- Bar M. A cortical mechanism for triggering top-down facilitation in visual object recognition. Journal of cognitive neuroscience. 2003;15(4):600–609.
- Decoding visual motions from EEG using attention-based RNN. Applied Sciences. 2020;10(16):5662.
- Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research. 2014;15(1):1929–1958.
- Reconstruction of natural visual scenes from neural spikes with deep neural networks. Neural Networks. 2020;125:19–30.
- Variational autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex. NeuroImage. 2019;198:125–136.
- Reconstructing perceived images from human brain activities with Bayesian deep multiview learning. IEEE transactions on neural networks and learning systems. 2018;30(8):2310–2323.
- Neural Encoding and Decoding With a Flow-Based Invertible Generative Model. IEEE Transactions on Cognitive and Developmental Systems. 2023 jun;15(2):724–736.
- Nanowire arrays restore vision in blind mice. Nature Communications. 2018;9(1):786. Publisher: Nature Publishing Group.
- An in-silico analysis of retinal electric field distribution induced by different electrode design of trans-corneal electrical stimulation. Journal of Neural Engineering. 2022 sep;19(5):055004.
- St-Yves G, Naselaris T. Generative adversarial networks conditioned on brain activity reconstruct seen images. In: 2018 IEEE international conference on systems, man, and cybernetics (SMC). IEEE; 2018. p. 1054–1061.
- Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage. 2018;181:775–785.
- Reconstructing perceived faces from brain activations with deep adversarial neural decoding. Advances in neural information processing systems. 2017;30.
- VanRullen R, Reddy L. Reconstructing faces from fMRI patterns using deep generative neural networks. Communications biology. 2019;2(1):193.
- Brain decoding of viewed image categories via semi-supervised multi-view Bayesian generative model. IEEE Transactions on Signal Processing. 2020;68:5769–5781.
- From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI. Advances in Neural Information Processing Systems. 2019;32.
- Peter Beech (1 paper)
- Shanshan Jia (10 papers)
- Zhaofei Yu (61 papers)
- Jian K. Liu (17 papers)