Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
173 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Towards Brain Inspired Design for Addressing the Shortcomings of ANNs (2307.00039v1)

Published 30 Jun 2023 in cs.NE, cs.AI, cs.CV, and cs.LG

Abstract: As our understanding of the mechanisms of brain function is enhanced, the value of insights gained from neuroscience to the development of AI algorithms deserves further consideration. Here, we draw parallels with an existing tree-based ANN architecture and a recent neuroscience study[27] arguing that the error-based organization of neurons in the cerebellum that share a preference for a personalized view of the entire error space, may account for several desirable features of behavior and learning. We then analyze the learning behavior and characteristics of the model under varying scenarios to gauge the potential benefits of a similar mechanism in ANN. Our empirical results suggest that having separate populations of neurons with personalized error views can enable efficient learning under class imbalance and limited data, and reduce the susceptibility to unintended shortcut strategies, leading to improved generalization. This work highlights the potential of translating the learning machinery of the brain into the design of a new generation of ANNs and provides further credence to the argument that biologically inspired AI may hold the key to overcoming the shortcomings of ANNs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124018, 2019.
  2. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  3. Imbalanced deep learning by minority class incremental rectification. IEEE transactions on pattern analysis and machine intelligence, 41(6):1367–1381, 2018.
  4. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020.
  5. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231, 2018.
  6. Neuroscience-inspired artificial intelligence. Neuron, 95(2):245–258, 2017.
  7. Jeff Hawkins. A thousand brains: A new theory of intelligence. Hachette UK, 2021.
  8. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340–8349, 2021.
  9. Using pre-training can improve model robustness and uncertainty. arXiv preprint arXiv:1901.09960, 2019.
  10. Natural adversarial examples.(2019). arXiv preprint cs.LG/1907.07174, 2019.
  11. Principles of operation of a cerebellar learning circuit. Elife, 9:e55217, 2020.
  12. Encoding of error and learning to correct that error by the purkinje cells of the cerebellum. Nature neuroscience, 21(5):736–743, 2018.
  13. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision, pages 1501–1510, 2017.
  14. Combining diverse feature priors. arXiv preprint arXiv:2110.08220, 2021.
  15. Survey on deep learning with class imbalance. Journal of Big Data, 6(1):1–54, 2019.
  16. On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
  17. Splitnet: Learning to semantically split deep networks for parameter reduction and model parallelization. In International Conference on Machine Learning, pages 1866–1874. PMLR, 2017.
  18. Cerebellar complex spikes encode both destinations and errors in arm movements. Nature, 392(6675):494–497, 1998.
  19. Changes in simple spike activity of some purkinje cells in the oculomotor vermis during saccade adaptation are appropriate to participate in motor learning. Journal of Neuroscience, 30(10):3715–3727, 2010.
  20. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4(3):196–210, 2022.
  21. Interpolated adversarial training: Achieving robust neural networks without sacrificing too much accuracy. In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security, pages 95–103. ACM, 2019.
  22. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730–3738, 2015.
  23. Natural and artificial intelligence: A brief introduction to the interplay between ai and neuroscience research. Neural Networks, 144:603–613, 2021.
  24. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1406–1415, 2019.
  25. The role of the cerebellum in cognition and behavior: a selective review. The Journal of neuropsychiatry and clinical neurosciences, 12(2):193–198, 2000.
  26. Grad-cam: Visual explanations from deep networks via gradient-based localization. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 618–626. IEEE, 2017.
  27. Reza Shadmehr. Population coding in the cerebellum: a machine learning perspective. Journal of Neurophysiology, 124(6):2022–2051, 2020.
  28. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810, 2017.
  29. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1–5. IEEE, 2015.
  30. Fundamental tradeoffs between invariance and sensitivity to adversarial perturbations. In International Conference on Machine Learning, pages 9561–9571. PMLR, 2020.
  31. Robustness may be at odds with accuracy. In International Conference on Learning Representations, number 2019, 2019.
  32. Larry Vandervert. The prominent role of the cerebellum in the learning, origin and advancement of culture. Cerebellum & Ataxias, 3(1):1–13, 2016.
  33. Fupin Yao. Machine learning with limited data. arXiv preprint arXiv:2101.11461, 2021.
  34. Anthony M Zador. A critique of pure learning and what artificial neural networks can learn from animal brains. Nature communications, 10(1):1–7, 2019.
  35. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
  36. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4320–4328, 2018.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now