Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
164 tokens/sec
GPT-4o
10 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Deep Oscillatory Neural Network (2405.03725v2)

Published 6 May 2024 in cs.NE, cs.AI, and cs.LG

Abstract: We propose a novel, brain-inspired deep neural network model known as the Deep Oscillatory Neural Network (DONN). Deep neural networks like the Recurrent Neural Networks indeed possess sequence processing capabilities but the internal states of the network are not designed to exhibit brain-like oscillatory activity. With this motivation, the DONN is designed to have oscillatory internal dynamics. Neurons of the DONN are either nonlinear neural oscillators or traditional neurons with sigmoidal or ReLU activation. The neural oscillator used in the model is the Hopf oscillator, with the dynamics described in the complex domain. Input can be presented to the neural oscillator in three possible modes. The sigmoid and ReLU neurons also use complex-valued extensions. All the weight stages are also complex-valued. Training follows the general principle of weight change by minimizing the output error and therefore has an overall resemblance to complex backpropagation. A generalization of DONN to convolutional networks known as the Oscillatory Convolutional Neural Network is also proposed. The two proposed oscillatory networks are applied to a variety of benchmark problems in signal and image/video processing. The performance of the proposed models is either comparable or superior to published results on the same data sets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (76)
  1. A. J. Kell, D. L. Yamins, E. N. Shook, S. V. Norman-Haignere, and J. H. McDermott, “A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy,” Neuron, vol. 98, no. 3, pp. 630–644.e16, 2018.
  2. Y. Huang, S. Dai, T. Nguyen, P. Bao, D. Y. Tsao, R. G. Baraniuk, and A. Anandkumar, “Brain-inspired robust vision using convolutional neural networks with feedback,” in Real Neurons & Hidden Units: Future directions at the intersection of neuroscience and artificial intelligence @ NeurIPS 2019, 2019.
  3. S. G. Wardle, J. Taubert, L. Teichmann, and C. I. Baker, “Rapid and dynamic processing of face pareidolia in the human brain,” Nature communications, vol. 11, no. 1, p. 4518, 2020.
  4. S. G. Wardle and C. I. Baker, “Recent advances in understanding object recognition in the human brain: deep neural networks, temporal dynamics, and context,” F1000Research, vol. 9, 2020.
  5. R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, “Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness,” 2022.
  6. R. M. Cichy, A. Khosla, D. Pantazis, A. Torralba, and A. Oliva, “Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence,” Scientific reports, vol. 6, no. 1, p. 27755, 2016.
  7. C. Bick, M. Goodfellow, C. R. Laing, and E. A. Martens, “Understanding the dynamics of biological and neural oscillator networks through exact mean-field reductions: a review,” The Journal of Mathematical Neuroscience, vol. 10, no. 1, p. 9, 2020.
  8. N. Chinichian, M. Lindner, S. Yanchuk, T. Schwalger, E. Schöll, and R. Berner, “Modeling brain network flexibility in networks of coupled oscillators: a feasibility study,” Scientific Reports, vol. 14, no. 1, p. 5713, 2024.
  9. C. R. Laing, “Phase oscillator network models of brain dynamics,” Computational models of brain and behavior, pp. 505–517, 2017.
  10. S. Kumari, V. Chandrasekaran, and V. Chakravarthy, “The flip-flop neuron: a memory efficient alternative for solving challenging sequence processing and decision-making problems,” Neural Computing and Applications, 2023.
  11. K. Gopalakrishnan and F. M. Salem, “Sentiment analysis using simplified long short-term memory recurrent neural networks,” 2020.
  12. A. Graves, “Generating sequences with recurrent neural networks,” 2014.
  13. Z. Tan, S. Wang, Z. Yang, G. Chen, X. Huang, M. Sun, and Y. Liu, “Neural machine translation: A review of methods, resources, and tools,” AI Open, vol. 1, pp. 5–21, 2020.
  14. X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W. kin Wong, and W. chun Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” 2015.
  15. H. A. Atabay, “Deep residual learning for tomato plant leaf disease identification.” Journal of Theoretical & Applied Information Technology, vol. 95, no. 24, 2017.
  16. B. Rajalingam, R. Priya, and R. Scholar, “Review of multimodality medical image fusion using combined transform techniques for clinical application,” Intl Journal of Sci Res in Comp Sci App., vol. 7, no. 3, pp. 1–8, 2018.
  17. E. W. Large, I. Roman, J. C. Kim, J. Cannon, J. K. Pazdera, L. J. Trainor, J. Rinzel, and A. Bose, “Dynamic models for musical rhythm perception and coordination,” Frontiers in Computational Neuroscience, vol. 17, 2023.
  18. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath et al., “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” IEEE Signal processing magazine, vol. 29, no. 6, pp. 82–97, 2012.
  19. S.-Y. Chou, K.-H. Cheng, J.-S. R. Jang, and Y.-H. Yang, “Learning to match transient sound events using attentional similarity for few-shot sound recognition,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2019, pp. 26–30.
  20. M. J. Kahana, “The cognitive correlates of human brain oscillations,” Journal of Neuroscience, vol. 26, no. 6, pp. 1669–1672, 2006.
  21. O. Akeju and E. N. Brown, “Neural oscillations demonstrate that general anesthesia and sedative states are neurophysiologically distinct from sleep,” Current opinion in neurobiology, vol. 44, pp. 178–185, 2017.
  22. J. A. Blanco, M. Stead, A. Krieger, W. Stacey, D. Maus, E. Marsh, J. Viventi, K. H. Lee, R. Marsh, B. Litt et al., “Data mining neocortical high-frequency oscillations in epilepsy and controls,” Brain, vol. 134, no. 10, pp. 2948–2959, 2011.
  23. G. Buzsáki and A. Draguhn, “Neuronal oscillations in cortical networks,” Science, vol. 304, no. 5679, pp. 1926–1929, 2004.
  24. D. Pinotsis, P. Robinson, P. beim Graben, and K. Friston, “Neural masses and fields: modeling the dynamics of brain activity,” p. 149, 2014.
  25. W. M. Cowan, D. Gottlieb, A. E. Hendrickson, J. Price, and T. Woolsey, “The autoradiographic demonstration of axonal connections in the central nervous system,” Brain research, vol. 37, no. 1, pp. 21–51, 1972.
  26. J. Nagumo, S. Arimoto, and S. Yoshizawa, “An active pulse transmission line simulating nerve axon,” Proceedings of the IRE, vol. 50, no. 10, pp. 2061–2070, 1962.
  27. V. Jirsa, M. M. Woodman, and L. Domide, “The virtual brain (tvb): Simulation environment for large-scale brain networks,” in Encyclopedia of Computational Neuroscience.   Springer, 2022, pp. 3397–3407.
  28. A. Bandyopadhyay, S. Ghosh, D. Biswas, and et al., “A phenomenological model of whole brain dynamics using a network of neural oscillators with power-coupling,” Scientific Reports, vol. 13, p. 16935, 2023.
  29. J. Cabral, E. Hugues, O. Sporns, and G. Deco, “Role of local network oscillations in resting-state functional connectivity,” Neuroimage, vol. 57, no. 1, pp. 130–139, 2011.
  30. P. Bey, K. Dhindsa, A. Kashyap, M. Schirner, J. Feldheim, M. Bönstrup, R. Schulz, B. Cheng, G. Thomalla, C. Gerloff et al., “Lesion aware automated processing pipeline for multimodal neuroimaging stroke data and the virtual brain (tvb),” bioRxiv, pp. 2023–08, 2023.
  31. P. Sanz Leon, S. A. Knock, M. M. Woodman, L. Domide, J. Mersmann, A. R. McIntosh, and V. Jirsa, “The virtual brain: a simulator of primate brain network dynamics,” Frontiers in neuroinformatics, vol. 7, p. 10, 2013.
  32. G. Deco, V. Jirsa, O. Sporns, and R. Kötter, “Key role of coupling, delay, and noise in resting brain fluctuations,” Proceedings of the National Academy of Sciences, vol. 106, no. 25, pp. 10 302–10 307, 2009.
  33. M. Dutra, A. de Pina Filho, and V. Romano, “Modeling of a bipedal locomotor using coupled nonlinear oscillators of van der pol,” Biol Cybern, vol. 88, pp. 286–292, 2003.
  34. D. Lakatos, F. Petit, and A. Albu-Schäffer, “Nonlinear oscillations for cyclic movements in human and robotic arms,” IEEE Transactions on Robotics, vol. 30, no. 4, pp. 865–879, 2014.
  35. Y. Cao, S. Bi, Y. Cai, and L. Zhang, “Applying coupled nonlinear oscillators to imitate swimming modes of cow-nosed rays,” in 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO), 2013, pp. 552–557.
  36. J. C. Kim and E. Large, “Signal processing in periodically forced gradient frequency neural networks,” Frontiers in Computational Neuroscience, vol. 9, 12 2015.
  37. S. Kumari and V. S. Chakravarthy, “Biologically inspired image classifier based on saccadic eye movement design for convolutional neural networks,” Neurocomputing, vol. 513, pp. 294–317, 2022.
  38. K. Soman, V. Muralidharan, and V. S. Chakravarthy, “A model of multisensory integration and its influence on hippocampal spatial cell responses,” IEEE Transactions on Cognitive and Developmental Systems, vol. 10, no. 3, pp. 637–646, 2018.
  39. A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, “Learning word vectors for sentiment analysis,” in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies.   Association for Computational Linguistics, June 2011, pp. 142–150.
  40. J. Liu, J. Luo, and M. Shah, “Recognizing realistic actions from videos “in the wild”,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 1996–2003.
  41. Z. Chang, X. Zhang, S. Wang, S. Ma, Y. Ye, X. Xinguang, and W. Gao, “Mau: A motion-aware unit for video prediction and beyond,” Advances in Neural Information Processing Systems, vol. 34, pp. 26 950–26 962, 2021.
  42. R. M. Alexander, “The gaits of bipedal and quadrupedal animals,” International Journal of Robotics Research, vol. 3, no. 2, pp. 49–59, Jun 1984.
  43. C. Maufroy, H. Kimura, and K. Takase, “Towards a general neural controller for quadrupedal locomotion,” Neural Networks, vol. 21, no. 4, pp. 642–653, 2008.
  44. S. Grillner, “Neural networks for vertebrate locomotion,” Scientific American, vol. 274, no. 1, pp. 64–69, 1996.
  45. D. G. Ivashko, B. I. Prilutsky, S. N. Markin, J. K. Chapin, and I. A. Rybak, “Modeling the spinal cord neural circuitry controlling cat hindlimb movement during locomotion,” Neurocomputing, vol. 52–54, pp. 331–338, 2003.
  46. A. Bonagiri, D. Biswas, and S. Chakravarthy, “Coupled memristor oscillators for neuromorphic locomotion control: Modeling and analysis,” IEEE Transactions on Neural Networks and Learning Systems, 2022.
  47. A. J. Ijspeert, “Central pattern generators for locomotion control in animals and robots: A review,” Neural Networks, vol. 21, no. 4, pp. 642–653, 2008.
  48. C. Gray, P. König, A. Engel, and W. Singer, “Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties,” Nature, vol. 338, no. 6213, pp. 334–337, 1989.
  49. T. Isokawa, H. Nishimura, N. Kamiura, and N. Matsui, “Perceptual binding by coupled oscillatory neural network,” in Artificial Neural Networks: Biological Inspirations – ICANN 2005, 2005, pp. 336–344.
  50. A. R. Rao, “An oscillatory neural network model that demonstrates the benefits of multisensory learning,” Cognitive Neurodynamics, vol. 12, no. 5, pp. 481–499, 2018.
  51. J. E. Pina, M. Bodner, and B. Ermentrout, “Oscillations in working memory and neural binding: A mechanism for multiple memories and their interactions,” PLoS Comput Biol, vol. 14, no. 11, p. e1006517, 2018.
  52. V. Sip, M. Hashemi, T. Dickscheid, K. Amunts, S. Petkoski, and V. Jirsa, “Characterization of regional differences in resting-state fmri with a data-driven network model of brain dynamics,” Science Advances, vol. 9, no. 11, p. eabq7547, 2023.
  53. S. Ghosh, D. Biswas, S. Vijayan, and V. S. Chakravarthy, “Modeling whole brain electroencephalogram (eeg) in a spatially organized oscillatory neural network,” bioRxiv, vol. 2023.07.16.549247, 2023.
  54. A. Al-Hossenat, P. Wen, and Y. Li, “Large-scale brain network model and multi-band electroencephalogram rhythm simulations,” International Journal of Biomedical Engineering and Technology, vol. 38, no. 4, pp. 395–409, 2022.
  55. M. Schirner, A. R. McIntosh, V. Jirsa, G. Deco, and P. Ritter, “Inferring multi-scale neural mechanisms with brain network modelling,” elife, vol. 7, p. e28927, 2018.
  56. C. B. Schäfer, B. R. Morgan, A. X. Ye, M. J. Taylor, and S. M. Doesburg, “Oscillations, networks, and their development: Meg connectivity changes with age,” Human Brain Mapping, vol. 35, no. 10, pp. 5249–5261, 2014.
  57. L. Yang, J. Lu, D. Li, J. Xiang, T. Yan, J. Sun, and B. Wang, “Alzheimer’s disease: Insights from large-scale brain dynamics models,” Brain Sciences, vol. 13, p. 1133, 07 2023.
  58. S. Idesis, C. Favaretto, N. V. Metcalf, J. C. Griffis, G. L. Shulman, M. Corbetta, and G. Deco, “Inferring the dynamical effects of stroke lesions through whole-brain modeling,” NeuroImage: Clinical, vol. 36, p. 103233, 2022.
  59. J. Kim, M. Criaud, S. S. Cho, M. Díez-Cirarda, A. Mihaescu, S. Coakeley, C. Ghadery, M. Valli, M. F. Jacobs, S. Houle, and A. P. Strafella, “Abnormal intrinsic brain functional network dynamics in Parkinson’s disease,” Brain, vol. 140, no. 11, pp. 2955–2967, 10 2017.
  60. J. J. Hopfield, “Neural networks and physical systems with emergent collective computational abilities,” Proceedings of the National Academy of Sciences, vol. 79, no. 8, pp. 2554–2558, 1982.
  61. ——, “Neurons with graded response have collective computational properties like those of two-state neurons,” Proceedings of the National Academy of Sciences, vol. 81, no. 10, pp. 3088–3092, 1984.
  62. W. J. Freeman and J. M. Barrie, “Analysis of spatial patterns of phase in neocortical gamma eegs in rabbit,” Journal of Neurophysiology, vol. 84, no. 3, pp. 1266–1278, 2000.
  63. Y. Imai and T. Taniguchi, “Associative memory by virtual oscillator network based on single spin-torque oscillator,” Scientific Reports, vol. 13, p. 15809, 2023.
  64. S. Chakravarthy and J. Ghosh, “A complex-valued associative memory for storing patterns as oscillatory states,” Biological Cybernetics, vol. 75, pp. 229–238, 1996.
  65. T. Aonishi, “Phase transitions of an oscillator neural network with a standard hebb learning rule,” Phys. Rev. E, vol. 58, pp. 4865–4871, Oct 1998.
  66. T. Nishikawa, F. C. Hoppensteadt, and Y.-C. Lai, “Oscillatory associative memory network with perfect retrieval,” Physica D: Nonlinear Phenomena, vol. 197, no. 1, pp. 134–148, 2004.
  67. R. Follmann, E. E. N. Macau, E. Rosa, and J. R. C. Piqueira, “Phase oscillatory network and visual pattern recognition,” IEEE Transactions on Neural Networks and Learning Systems, vol. 26, no. 7, pp. 1539–1544, 2015.
  68. D. Wang and D. Terman, “Locally excitatory globally inhibitory oscillator networks,” IEEE Transactions on Neural Networks, vol. 6, no. 1, pp. 283–286, 1995.
  69. M. G. Kuzmina, E. A. Manykin, and I. Surina, “Oscillatory network with self-organized dynamical connections for synchronization-based image segmentation,” Biosystems, vol. 76, p. 43, 2004.
  70. M. G. Kuzmina and E. A. Manykin, “Oscillatory neural network for adaptive dynamical image processing,” in Proc. CIMCA’05, 2005, p. 301.
  71. G. Csaba and W. Porod, “Noise immunity of oscillatory computing devices,” IEEE Journal on Exploratory Solid-State Computational Devices and Circuits, vol. 6, no. 2, pp. 164–169, 2020.
  72. R. L. Wigington, “A new concept in computing,” Proceedings of the IRE, vol. 47, no. 4, pp. 516–523, 1959.
  73. N. Rathi, I. Chakraborty, A. Kosta, A. Sengupta, A. Ankit, P. Panda, and K. Roy, “Exploring neuromorphic computing based on spiking neural networks: Algorithms to hardware,” ACM Computing Surveys, vol. 55, no. 12, p. 243, 2023.
  74. D. E. Nikonov and et al., “Coupled-oscillator associative memory array operation for pattern recognition,” IEEE J. Exploratory Solid-State Comput. Devices Circuits, vol. 1, pp. 85–93, 2015.
  75. T. Rudner, W. Porod, and G. Csaba, “Design of oscillatory neural networks by machine learning,” Frontiers in Neuroscience, vol. 18, 2024.
  76. M. Abernot, T. Gil, M. Jiménez, J. Núñez, M. J. Avellido, B. Linares-Barranco, T. Gonos, T. Hardelin, and A. Todri-Sanial, “Digital implementation of oscillatory neural network for image recognition applications,” Frontiers in Neuroscience, vol. 15, 2021.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now