Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Probing Biological and Artificial Neural Networks with Task-dependent Neural Manifolds (2312.14285v1)

Published 21 Dec 2023 in q-bio.NC, cs.LG, and cs.NE

Abstract: Recently, growth in our understanding of the computations performed in both biological and artificial neural networks has largely been driven by either low-level mechanistic studies or global normative approaches. However, concrete methodologies for bridging the gap between these levels of abstraction remain elusive. In this work, we investigate the internal mechanisms of neural networks through the lens of neural population geometry, aiming to provide understanding at an intermediate level of abstraction, as a way to bridge that gap. Utilizing manifold capacity theory (MCT) from statistical physics and manifold alignment analysis (MAA) from high-dimensional statistics, we probe the underlying organization of task-dependent manifolds in deep neural networks and macaque neural recordings. Specifically, we quantitatively characterize how different learning objectives lead to differences in the organizational strategies of these models and demonstrate how these geometric analyses are connected to the decodability of task-relevant information. These analyses present a strong direction for bridging mechanistic and normative theories in neural networks through neural population geometry, potentially opening up many future research avenues in both machine learning and neuroscience.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Unsupervised pre-training of image features on non-curated data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2959–2968, 2019.
  2. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  3. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862, 2019.
  4. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3):e2014196118, 2021. doi: 10.1073/pnas.2014196118. URL https://www.pnas.org/doi/abs/10.1073/pnas.2014196118.
  5. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23):8619–8624, 2014.
  6. Learning efficient coding of natural images with maximum manifold capacity representations. arXiv preprint arXiv:2303.03307, 2023.
  7. A deep learning framework for neuroscience. Nature neuroscience, 22(11):1761–1770, 2019.
  8. SueYeon Chung and LF Abbott. Neural population geometry: An approach for understanding biological and artificial neural networks. Current opinion in neurobiology, 70:137–144, 2021.
  9. Intrinsic dimension of data representations in deep neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/cfcce0621b49c983991ead4c3d4d3b6b-Paper.pdf.
  10. Separability and geometry of object manifolds in deep neural networks. Nature Communications, 11(1):746, Feb 2020. ISSN 2041-1723. doi: 10.1038/s41467-020-14578-5. URL https://doi.org/10.1038/s41467-020-14578-5.
  11. Classification and geometry of general perceptual manifolds. Phys. Rev. X, 8:031003, Jul 2018. doi: 10.1103/PhysRevX.8.031003. URL https://link.aps.org/doi/10.1103/PhysRevX.8.031003.
  12. Object manifold geometry across the mouse cortical visual hierarchy. BioRxiv, pages 2020–08, 2020.
  13. Transformation of acoustic information to sensory decision variables in the parietal cortex. Proceedings of the National Academy of Sciences, 120(2):e2212120120, 2023.
  14. On the geometry of generalization and memorization in deep neural networks. arXiv preprint arXiv:2105.14602, 2021.
  15. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance. Journal of Neuroscience, 35(39):13402–13418, 2015.
  16. Explicit information for category-orthogonal object properties increases along the ventral stream. Nature Neuroscience, 19(4):613–622, Apr 2016. ISSN 1546-1726. doi: 10.1038/nn.4247. URL https://doi.org/10.1038/nn.4247.
  17. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  18. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13001–13008, 2020.
  19. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03230, 2021.
  20. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pages 132–149, 2018.
  21. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912–9924, 2020.
  22. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  23. Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6707–6717, 2020.
  24. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271–21284, 2020.
  25. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021.
  26. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  27. Aligning model and macaque inferior temporal cortex representations improves model-to-human behavioral alignment and adversarial robustness. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=SMYdcXjJh1q.
  28. Linear classification of neural manifolds with correlated variability, 2022.
  29. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007, 2018.
  30. Representational similarity analysis - connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2, 2008. ISSN 1662-5137. doi: 10.3389/neuro.06.004.2008. URL https://www.frontiersin.org/articles/10.3389/neuro.06.004.2008.
  31. System identification of neural systems: If we got it right, would we know?, 2023.
  32. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2661–2671, 2019.
  33. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512–523, 2020.
  34. Why do better loss functions lead to less transferable features? Advances in Neural Information Processing Systems, 34:28648–28662, 2021.
  35. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229, 2019.
  36. A mathematical exploration of why language models help solve downstream tasks. arXiv preprint arXiv:2010.03648, 2020.
  37. On the provable advantage of unsupervised pretraining. arXiv preprint arXiv:2303.01566, 2023.
  38. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2017.
  39. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
  40. When is unsupervised disentanglement possible? Advances in Neural Information Processing Systems, 34:5150–5161, 2021.
  41. Disentanglement with biological constraints: A theory of functional cell types. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=9Z_GfhZnGH.
  42. Abstract representations emerge naturally in neural networks trained to perform multiple tasks. Nature Communications, 14(1):1040, 2023.
  43. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 4114–4124. PMLR, 2019.
  44. On the bures–wasserstein distance between positive definite matrices. Expositiones Mathematicae, 37(2):165–191, 2019.
  45. Representational dissimilarity metric spaces for stochastic neural networks. arXiv preprint arXiv:2211.11665, 2022.
  46. Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2019.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now