Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
140 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Efficient Representation of Natural Image Patches (2210.13004v3)

Published 24 Oct 2022 in cs.CV, cs.LG, eess.IV, and q-bio.NC

Abstract: Utilizing an abstract information processing model based on minimal yet realistic assumptions inspired by biological systems, we study how to achieve the early visual system's two ultimate objectives: efficient information transmission and accurate sensor probability distribution modeling. We prove that optimizing for information transmission does not guarantee optimal probability distribution modeling in general. We illustrate, using a two-pixel (2D) system and image patches, that an efficient representation can be realized through a nonlinear population code driven by two types of biologically plausible loss functions that depend solely on output. After unsupervised learning, our abstract information processing model bears remarkable resemblances to biological systems, despite not mimicking many features of real neurons, such as spiking activity. A preliminary comparison with a contemporary deep learning model suggests that our model offers a significant efficiency advantage. Our model provides novel insights into the computational theory of early visual systems as well as a potential new approach to enhance the efficiency of deep learning models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (89)
  1. Joseph J. Atick. Could information theory provide an ecological theory of sensory processing? Network: Computation in neural systems, 3(2):213–251, 1992.
  2. Fred Attneave. Some informational aspects of visual perception. Psychological review, 61(3):183, 1954.
  3. An energy budget for signaling in the grey matter of the brain. Journal of Cerebral Blood Flow & Metabolism, 21(10):1133–1145, 2001.
  4. The functional diversity of retinal ganglion cells in the mouse. Nature, 529(7586):345–350, 2016.
  5. Temporal precision of spike trains in extrastriate cortex of the behaving macaque monkey. Neural computation, 8(6):1185–1202, 1996.
  6. Learning in linear neural networks: A survey. IEEE Transactions on neural networks, 6(4):837–858, 1995.
  7. Horace B Barlow. Unsupervised learning. Neural computation, 1(3):295–311, 1989.
  8. Horace B Barlow et al. Possible principles underlying the transformation of sensory messages. Sensory communication, 1(01):217–233, 1961.
  9. Intensity coding in primate visual system. Experimental Brain Research, 31:163–177, 1978.
  10. An information-maximization approach to blind separation and blind deconvolution. Neural Computation, 7:1129–1159, 11 1995.
  11. The “independent components” of natural scenes are edge filters. Vision research, 37(23):3327–3338, 1997.
  12. The structure and precision of retinal spike trains. Proceedings of the National Academy of Sciences of the United States of America, 94(10):5411–5416, may 1997.
  13. Neural correlates of sparse coding and dimensionality reduction. PLOS Computational Biology, 15(6):e1006908, jun 2019.
  14. Bhutajata. Visible spectrum on a linear scale, 2017.
  15. Temporal precision in the neural code and the timescales of natural vision. Nature 2007 449:7158, 449(7158):92–95, sep 2007.
  16. John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6):679–698, 1986.
  17. Do We Know What the Early Visual System Does? Journal of Neuroscience, 25(46):10577–10597, nov 2005.
  18. Jean François Cardoso. Infomax and maximum likelihood for blind source separation. IEEE Signal Processing Letters, 4:112–114, 1997.
  19. Theoretical neuroscience: computational and mathematical modeling of neural systems. MIT press, 2005.
  20. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  21. Decorrelated neuronal firing in cortical microcircuits. Science, 327(5965):584–587, 2010.
  22. David J. Field. What Is the Goal of Sensory Coding? Neural Computation, 6(4):559–601, jul 1994.
  23. Peter Foldiak. Sparse coding in the primate cortex. The handbook of brain theory and neural networks, 2003.
  24. The coding of uniform colour figures in monkey visual cortex. The Journal of physiology, 548(2):593–613, 2003.
  25. Efficient sensory encoding and bayesian inference with heterogeneous neural populations. Neural computation, 26(10):2103–2134, 2014.
  26. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Transactions on pattern analysis and machine intelligence, (6):721–741, 1984.
  27. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
  28. Entity embeddings of categorical variables. arXiv preprint arXiv:1604.06737, 2016.
  29. Microstructure of a spatial map in the entorhinal cortex. Nature 2005 436:7052, 436(7052):801–806, jun 2005.
  30. The luminance and response range of monkey retinal ganglion cells to white light. Vision Research, 22(2):271–277, 1982.
  31. The principal components of natural images. Network: computation in neural systems, 3(1):61, 1992.
  32. V1 mechanisms underlying chromatic contrast detection. Journal of Neurophysiology, 109(10):2483–2494, 2013.
  33. Neuroscience-Inspired Artificial Intelligence. Neuron, 95(2):245–258, 2017.
  34. Reducing the Dimensionality of Data with Neural Networks. Science, 313(5786):504–507, 2006.
  35. Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771–1800, 2002.
  36. Natural image statistics: A probabilistic approach to early computational vision., volume 39. Springer Science & Business Media, 2009.
  37. Eugene M Izhikevich. Which model to use for cortical spiking neurons? IEEE transactions on neural networks, 15(5):1063–1070, 2004.
  38. Bayesian networks and decision graphs, volume 2. Springer, 2007.
  39. The spatial transformation of color in the primary visual cortex of the macaque monkey. Nature neuroscience, 4(4):409–416, 2001.
  40. Emergence of complex cell properties by learning to generalize in natural scenes. Nature, 457(7225):83–86, jan 2009.
  41. Efficient coding of natural images with a population of noisy linear-nonlinear neurons. Advances in neural information processing systems, 24, 2011.
  42. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  43. Simon Laughlin. A Simple Coding Procedure Enhances a Neuron’s Information Capacity. Zeitschrift für Naturforschung C, 36(9-10):910–912, oct 1981.
  44. Revisiting edge detection in convolutional neural networks. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–9. IEEE, 2021.
  45. Deep learning. Nature, 521(7553):436–444, 2015.
  46. The Nonlinear Statistics of High-Contrast Patches in Natural Images. International Journal of Computer Vision, 54(5413):83–103, 2003.
  47. Learning the parts of objects by non-negative matrix factorization. Nature, 401:788, oct 1999.
  48. Efficient sparse coding algorithms. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006.
  49. William R Levick. Receptive fields and trigger features of ganglion cells in the visual streak of the rabbit’s retina. The Journal of physiology, 188(3):285, 1967.
  50. Mixing of chromatic and luminance retinal signals in primate area v1. Cerebral Cortex, 25(7):1920–1937, 2015.
  51. Microsoft coco: Common objects in context, 2014.
  52. Ralph Linsker. Self-organization in a perceptual network. Computer, 21(3):105–117, 1988.
  53. Reliability of spike timing in neocortical neurons. Science, 268(5216):1503–1506, 1995.
  54. Independence of luminance and contrast in natural scenes and in the early visual system. Nature neuroscience, 8(12):1690–1697, 2005.
  55. Multiple models to capture the variability in biological neurons and networks. Nature neuroscience, 14(2):133–138, 2011.
  56. David Marr. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. The MIT Press, 07 2010.
  57. Richard H. Masland. The Neuronal Organization of the Retina, oct 2012.
  58. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.
  59. Elliott Scott Milner and Michael Tri Hoang Do. A population representation of absolute light intensity in the mammalian retina. Cell, 171(4):865–876, 2017.
  60. On the Number of Linear Regions of Deep Neural Networks. Advances in Neural Information Processing Systems, 4(January):2924–2932, feb 2014.
  61. Nonlinear neurons in the low-noise limit: a factorial code maximizes information transfer. http://dx.doi.org/10.1088/0954-898X_5_4_008, 5(4):565–581, 2009.
  62. Encoding of Luminance and Contrast by Linear and Nonlinear Synapses in the Retina. Neuron, 73:758–773, 2012.
  63. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–609, oct 1996.
  64. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311–3325, 1997.
  65. Sparse coding of sensory inputs. Current opinion in neurobiology, 14(4):481–487, 2004.
  66. Modeling image patches with a directed hierarchy of Markov random fields. In Advances in Neural Information Processing Systems 20 (NIPS’07), pages 1121–1128. 2008.
  67. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  68. Factored 3-Way Restricted Boltzmann Machines For Modeling Natural Images. In AISTATS ’10, volume 9, pages 621–628, 2010.
  69. Efficient learning of sparse representations with an energy-based model. Advances in neural information processing systems, 19, 2006.
  70. The representation of brightness in primary visual cortex. Science, 273:1104–1107, 8 1996.
  71. Fields of experts: A framework for learning image priors. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 2, pages 860–867. IEEE, 2005.
  72. Design Principles of Insect and Vertebrate Visual Systems, 2010.
  73. Natural signal statistics and sensory gain control. Nature neuroscience, 4(8):819–825, 2001.
  74. C. E. Shannon. A Mathematical Theory of Communication. Bell System Technical Journal, 27(3):379–423, 1948.
  75. Eero P Simoncelli. Vision and the statistics of the visual environment. Current Opinion in Neurobiology, 13(2):144–149, 2003.
  76. Natural image statistics and neural representation. Annual review of neuroscience, 24(1):1193–1216, 2001.
  77. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  78. Spatial mapping of monkey vi cells with pure color and luminance stimuli. Vision research, 24(7):751–769, 1984.
  79. Probabilistic principal component analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology, 61(3):611–622, 1999.
  80. J Hans van Hateren and Dan L Ruderman. Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex. Proceedings of the Royal Society of London. Series B: Biological Sciences, 265(1412):2315–2320, 1998.
  81. JH van Hateren. A theory of maximizing sensory information. Biol. Cybern, 68:23–29, 1992.
  82. Synaptic energy efficiency in retinal processing. Vision research, 43(11):1285–1292, 2003.
  83. Luminance potentiates human visuocortical responses. Journal of Neurophysiology, 123(2):473–483, 2020.
  84. Convergence and segregation of the multiple rod pathways in mammalian retina. Journal of Neuroscience, 24(49):11182–11192, 2004.
  85. Representation of color stimuli in awake macaque primary visual cortex. Neuron, 37(4):681–691, 2003.
  86. Mutual information, fisher information, and efficient coding. Neural computation, 28(2):305–326, 2016.
  87. The dynamic receptive fields of retinal ganglion cells. Progress in retinal and eye research, 67:102–117, 2018.
  88. Efficient sensory coding of multidimensional stimuli. PLoS computational biology, 16(9):e1008146, 2020.
  89. From learning models of natural image patches to whole image restoration. In 2011 international conference on computer vision, pages 479–486. IEEE, 2011.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now