Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Synaptic Weight Distributions Depend on the Geometry of Plasticity (2305.19394v2)

Published 30 May 2023 in q-bio.NC, cs.LG, and cs.NE

Abstract: A growing literature in computational neuroscience leverages gradient descent and learning algorithms that approximate it to study synaptic plasticity in the brain. However, the vast majority of this work ignores a critical underlying assumption: the choice of distance for synaptic changes - i.e. the geometry of synaptic plasticity. Gradient descent assumes that the distance is Euclidean, but many other distances are possible, and there is no reason that biology necessarily uses Euclidean geometry. Here, using the theoretical tools provided by mirror descent, we show that the distribution of synaptic weights will depend on the geometry of synaptic plasticity. We use these results to show that experimentally-observed log-normal weight distributions found in several brain areas are not consistent with standard gradient descent (i.e. a Euclidean geometry), but rather with non-Euclidean distances. Finally, we show that it should be possible to experimentally test for different synaptic geometries by comparing synaptic weight distributions before and after learning. Overall, our work shows that the current paradigm in theoretical work on synaptic plasticity that assumes Euclidean synaptic geometry may be misguided and that it should be possible to experimentally determine the true geometry of synaptic plasticity in the brain.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Deep learning without weight transport. Advances in neural information processing systems, 32, 2019.
  2. Shun-ichi Amari. Differential-geometrical methods in statistics. Lecture Notes on Statistics, 1985.
  3. Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 1998.
  4. Stochastic gradient/mirror descent: Minimax optimality and implicit regularization. arXiv preprint arXiv:1806.00952, 2018.
  5. Stochastic mirror descent on overparameterized nonlinear models. IEEE Transactions on Neural Networks and Learning Systems, 2021.
  6. The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning. Advances in Neural Information Processing Systems, 34:25164–25178, 2021.
  7. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167–175, 2003.
  8. Patrick Billingsley. Convergence of probability measures. John Wiley & Sons, 1999.
  9. Central limit theorems for interchangeable processes. Canadian Journal of Mathematics, 10:222–229, 1958.
  10. Self-consistent dynamical field theory of kernel evolution in wide neural networks. arXiv preprint arXiv:2205.09653, 2022.
  11. Lev M Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR computational mathematics and mathematical physics, 7(3):200–217, 1967.
  12. Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning, 8(3-4):231–357, 2015.
  13. The log-dynamic brain: how skewed distributions affect network operations. Nature Reviews Neuroscience, 15(4):264–278, 2014.
  14. On the choice of metric in gradient-based theories of brain function. arXiv e-prints, pp.  arXiv–1805, 2018.
  15. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  15750–15758, 2021.
  16. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
  17. Credit assignment through broadcasting a global error vector. Advances in Neural Information Processing Systems, 34:10053–10066, 2021.
  18. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.  248–255. Ieee, 2009.
  19. Binary and analog variation of synapses between cortical pyramidal neurons. Elife, 11:e76120, 2022.
  20. Composite objective mirror descent. In COLT, volume 10, pp.  14–26. Citeseer, 2010.
  21. Orthogonal representations for robust context-dependent task performance in brains and neural networks. Neuron, 110(7):1258–1270, 2022.
  22. Neuromodulated spike-timing-dependent plasticity, and theory of three-factor learning rules. Frontiers in neural circuits, 9:85, 2016.
  23. Exponentiated gradient meets gradient descent. In Algorithmic learning theory, pp.  386–407. PMLR, 2020.
  24. Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems, 30, 2017.
  25. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning, pp. 1832–1841. PMLR, 2018.
  26. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  27. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
  28. Exponentiated gradient versus gradient descent for linear predictors. information and computation, 132(1):1–63, 1997.
  29. Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs. In Neural Information Processing Systems (NeurIPS). Curran Associates, Inc., 2019.
  30. Learning with three factors: modulating hebbian plasticity with errors. Current opinion in neurobiology, 46:170–177, 2017.
  31. Mirror descent and the information ratio. In Conference on Learning Theory, pp.  2965–2992. PMLR, 2021.
  32. Bandit algorithms. Cambridge University Press, 2020.
  33. FFCV: Accelerating training by removing data bottlenecks. https://github.com/libffcv/ffcv/, 2022. commit 720cc2e.
  34. Mnist handwritten digit database, 2010.
  35. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
  36. Spatial profile of excitatory and inhibitory synaptic connectivity in mouse primary auditory cortex. Journal of Neuroscience, 32(16):5609–5619, 2012.
  37. How important is weight symmetry in backpropagation? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
  38. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7(1):13276, 2016.
  39. Multiplicative dynamics underlie the emergence of the log-normal distribution of spine sizes in the neocortex in vivo. Journal of Neuroscience, 31(26):9481–9488, 2011.
  40. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp.  116–131, 2018.
  41. James Martens. New insights and perspectives on the natural gradient method. The Journal of Machine Learning Research, 21(1):5776–5851, 2020.
  42. Gaussian process behaviour in wide deep neural networks. arXiv preprint arXiv:1804.11271, 2018.
  43. Distinct in vivo dynamics of excitatory synapses onto cortical pyramidal neurons and parvalbumin-positive interneurons. Cell reports, 37(6):109972, 2021.
  44. Making look-ahead active learning strategies feasible with neural tangent kernels. arXiv preprint arXiv:2206.12569, 2022.
  45. Task-driven convolutional recurrent models of the visual system. Advances in neural information processing systems, 31, 2018.
  46. Problem complexity and method efficiency in optimization. Wiley-Interscience, 1983.
  47. Information-geometric optimization algorithms: A unifying picture via invariance principles. The Journal of Machine Learning Research, 2017.
  48. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  49. Biological credit assignment through dynamic inversion of feedforward networks. Advances in Neural Information Processing Systems, 33:10065–10076, 2020.
  50. Kernelized information bottleneck leads to biologically plausible 3-factor hebbian learning in deep networks. Advances in Neural Information Processing Systems, 33:7296–7307, 2020.
  51. How to prepare your task head for finetuning. arXiv preprint arXiv:2302.05779, 2023.
  52. A deep learning framework for neuroscience. Nature neuroscience, 22(11):1761–1770, 2019.
  53. The study of plasticity has always been about gradients. The Journal of Physiology, 2023.
  54. Daniel L Ruderman. The statistics of natural images. Network: computation in neural systems, 5(4):517, 1994.
  55. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, pp.  407007, 2018.
  56. Five key factors determining pairwise correlations in visual cortex. Journal of neurophysiology, 114(2):1022–1033, 2015.
  57. Powerpropagation: A sparsity inducing weight reparameterisation. Advances in Neural Information Processing Systems, 34:28889–28903, 2021.
  58. Shai Shalev-Shwartz et al. Online learning and online convex optimization. Foundations and Trends® in Machine Learning, 4(2):107–194, 2012.
  59. Highly nonrandom features of synaptic connectivity in local cortical circuits. PLoS biology, 3(3):e68, 2005.
  60. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105–6114. PMLR, 2019.
  61. Filopodia are a structural substrate for silent synapses in adult neocortex. Nature, pp.  1–5, 2022.
  62. Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
  63. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111(23):8619–8624, 2014.
  64. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–115, 2021.
  65. A theory of learning with constrained weight-distribution. arXiv preprint arXiv:2206.08933, 2022.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now