Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Topological Data Analysis for Neural Network Analysis: A Comprehensive Survey (2312.05840v2)

Published 10 Dec 2023 in cs.LG and math.AT

Abstract: This survey provides a comprehensive exploration of applications of Topological Data Analysis (TDA) within neural network analysis. Using TDA tools such as persistent homology and Mapper, we delve into the intricate structures and behaviors of neural networks and their datasets. We discuss different strategies to obtain topological information from data and neural networks by means of TDA. Additionally, we review how topological information can be leveraged to analyze properties of neural networks, such as their generalization capacity or expressivity. We explore practical implications of deep learning, specifically focusing on areas like adversarial detection and model selection. Our survey organizes the examined works into four broad domains: 1. Characterization of neural network architectures; 2. Analysis of decision regions and boundaries; 3. Study of internal representations, activations, and parameters; 4. Exploration of training dynamics and loss functions. Within each category, we discuss several articles, offering background information to aid in understanding the various methodologies. We conclude with a synthesis of key insights gained from our study, accompanied by a discussion of challenges and potential advancements in the field.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (132)
  1. Credit Card Fraud Detection: Anonymized credit card transactions labeled as fraudulent or genuine. https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud/data. Accessed: 2023-12-01.
  2. Persistence images: A stable vector representation of persistent homology. Journal of Machine Learning Research, 18(8):1–35, 2017. URL http://jmlr.org/papers/v18/16-337.html.
  3. M. Agueh and G. Carlier. Barycenters in the Wasserstein space. SIAM Journal on Mathematical Analysis, 43(2):904–924, 2011. doi: 10.1137/100805741. URL https://doi.org/10.1137/100805741.
  4. Experimental stability analysis of neural networks in classification problems with confidence sets for persistence diagrams. Neural Networks, 143:42–51, 2021. ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2021.05.007. URL https://www.sciencedirect.com/science/article/pii/S0893608021001994.
  5. Metric space magnitude and generalisation in neural networks, 2023.
  6. Wasserstein generative adversarial networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 214–223. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.press/v70/arjovsky17a.html.
  7. On the stability of persistent entropy and new summary functions for topological data analysis. Pattern Recognition, 107:107509, 2020. ISSN 0031-3203. doi: https://doi.org/10.1016/j.patcog.2020.107509. URL https://www.sciencedirect.com/science/article/pii/S0031320320303125.
  8. Decorrelating neurons using persistence. In NeurIPS 2023 Workshop on Symmetry and Geometry in Neural Representations, 2023a. URL https://openreview.net/forum?id=NJS6568y79.
  9. Predicting the generalization gap in neural networks using topological data analysis, 2023b.
  10. Representation topology divergence: A method for comparing neural network representations. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1607–1626. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/barannikov22a.html.
  11. On the use of persistent homology to control the generalization capacity of a neural network. In B. Luo, L. Cheng, Z.-G. Wu, H. Li, and C. Li, editors, Neural Information Processing, pages 274–286, Singapore, 2024. Springer Nature Singapore. ISBN 978-981-99-8132-8.
  12. U. Bauer. Ripser: Efficient computation of Vietoris–Rips persistence barcodes. Journal of Applied and Computational Topology, 5(3):391–423, Sep 2021. ISSN 2367-1734. doi: 10.1007/s41468-021-00071-5. URL https://doi.org/10.1007/s41468-021-00071-5.
  13. M. Bianchini and F. Scarselli. On the complexity of neural network classifiers: A comparison between shallow and deep architectures. IEEE Transactions on Neural Networks and Learning Systems, 25(8):1553–1565, 2014. doi: 10.1109/TNNLS.2013.2293637.
  14. Intrinsic dimension, persistent homology and generalization in neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 6776–6789. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/35a12c43227f217207d4e06ffefe39d3-Paper.pdf.
  15. What is the state of neural network pruning? In I. Dhillon, D. Papailiopoulos, and V. Sze, editors, Proceedings of Machine Learning and Systems, volume 2, pages 129–146, 2020. URL https://proceedings.mlsys.org/paper_files/paper/2020/file/6c44dc73014d66ba49b28d483a8f8b0d-Paper.pdf.
  16. Persistence-based pooling for shape pose recognition. In A. Bac and J.-L. Mari, editors, Computational Topology in Image Context, pages 19–29, Cham, 2016. Springer International Publishing. ISBN 978-3-319-39441-1.
  17. M. Botnan and M. Lesnick. Algebraic stability of zigzag persistence modules. Algebraic & Geometric Topology, 18(6):3133–3204, oct 2018. doi: 10.2140/agt.2018.18.3133. URL https://doi.org/10.2140%2Fagt.2018.18.3133.
  18. R. Brüel Gabrielsson and G. Carlsson. Exposition and interpretation of the topology of neural networks. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pages 1069–1076, 2019. doi: 10.1109/ICMLA.2019.00180.
  19. P. Bubenik. Statistical topological data analysis using persistence landscapes. Journal of Machine Learning Research, 16(3):77–102, 2015. URL http://jmlr.org/papers/v16/bubenik15a.html.
  20. G. Carlsson and R. Brüel Gabrielsson. Topological approaches to deep learning. In N. A. Baas, G. E. Carlsson, G. Quick, M. Szymik, and M. Thaule, editors, Topological Data Analysis, pages 119–146, Cham, 2020. Springer International Publishing. ISBN 978-3-030-43408-3.
  21. G. Carlsson and V. de Silva. Zigzag persistence. Foundations of Computational Mathematics, 10(4):367–405, Aug 2010. ISSN 1615-3383. doi: 10.1007/s10208-010-9066-0. URL https://doi.org/10.1007/s10208-010-9066-0.
  22. On the local behavior of spaces of natural images. International Journal of Computer Vision, 76(1):1–12, Jan 2008. ISSN 1573-1405. doi: 10.1007/s11263-007-0056-x. URL https://doi.org/10.1007/s11263-007-0056-x.
  23. M. Carrière and S. Oudot. Structure and stability of the one-dimensional Mapper. Foundations of Computational Mathematics, 18(6):1333–1396, Dec 2018. ISSN 1615-3383. doi: 10.1007/s10208-017-9370-z. URL https://doi.org/10.1007/s10208-017-9370-z.
  24. Sliced Wasserstein kernel for persistence diagrams. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 664–673. JMLR.org, 2017.
  25. Optimizing persistent homology based functions. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1294–1303. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/carriere21a.html.
  26. Phom-gem: Persistent homology for generative models. In The 6th Swiss Conference on Data Science (SDS), 2019 IEEE International Conference. IEEE, 2019.
  27. Persistence stability for geometric complexes. Geometriae Dedicata, 173(1):193–214, Dec 2014. ISSN 1572-9168. doi: 10.1007/s10711-013-9937-z. URL https://doi.org/10.1007/s10711-013-9937-z.
  28. The Structure and Stability of Persistence Modules, volume 10 of SpringerBriefs in Mathematics. Springer, 2016.
  29. B. Chazelle. A minimum spanning tree algorithm with inverse-Ackermann type complexity. J. ACM, 47(6):1028–1047, nov 2000. ISSN 0004-5411. doi: 10.1145/355541.355562. URL https://doi.org/10.1145/355541.355562.
  30. A topological regularizer for classifiers via persistent homology. In K. Chaudhuri and M. Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 2573–2582. PMLR, 16–18 Apr 2019. URL https://proceedings.mlr.press/v89/chen19g.html.
  31. S. Chowdhury and F. Mémoli. Persistent path homology of directed networks. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA ’18, page 1152–1169, USA, 2018. Society for Industrial and Applied Mathematics. ISBN 9781611975031.
  32. Path homologies of deep feedforward networks. In 2019 18th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 1077–1082, 2019. doi: 10.1109/ICMLA.2019.00181.
  33. Stability of persistence diagrams. Discrete & Computational Geometry, 37(1):103–120, Jan 2007. ISSN 1432-0444. doi: 10.1007/s00454-006-1276-5. URL https://doi.org/10.1007/s00454-006-1276-5.
  34. What does it mean to learn in deep networks? And, how does one detect adversarial attacks? In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4752–4761, 2019. doi: 10.1109/CVPR.2019.00489.
  35. Computing the testing error without a testing set. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2674–2682, 2020. doi: 10.1109/CVPR42600.2020.00275.
  36. V. de Silva and G. Carlsson. Topological estimation using witness complexes. In M. Gross, H. Pfister, M. Alexa, and S. Rusinkiewicz, editors, SPBG’04 Symposium on Point-Based Graphics 2004. The Eurographics Association, 2004. ISBN 3-905673-09-6. doi: 10.2312/SPBG/SPBG04/157-166.
  37. Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity. Journal of Medicinal Chemistry, 34(2):786–797, 1991. doi: 10.1021/jm00106a046. URL https://doi.org/10.1021/jm00106a046.
  38. L. Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141–142, 2012. doi: 10.1109/MSP.2012.2211477.
  39. BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1423. URL https://doi.org/10.18653/v1/n19-1423.
  40. B. Di Fabio and M. Ferri. Comparing persistence diagrams through complex vectors. In V. Murino and E. Puppo, editors, Image Analysis and Processing – ICIAP 2015, pages 294–305, Cham, 2015. Springer International Publishing. ISBN 978-3-319-23231-7.
  41. Grounding representation similarity through statistical testing. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=_kwj6V53ZqB.
  42. Generalization bounds using data-dependent fractal dimensions. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 8922–8968. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/dupuis23a.html.
  43. H. Edelsbrunner and J. L. Harer. Computational Topology: An Introduction. American Mathematical Society, 2022.
  44. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD’96, page 226–231. AAAI Press, 1996.
  45. M. Gabella. Topology of learning in feedforward neural networks. IEEE Transactions on Neural Networks and Learning Systems, 32(8):3588–3592, 2021. doi: 10.1109/TNNLS.2020.3015790.
  46. Characterizing the shape of activation space in deep neural networks. In 2019 18th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 1537–1542, 2019. doi: 10.1109/ICMLA.2019.00254.
  47. M. Gidea and Y. Katz. Topological data analysis of financial time series: Landscapes of crashes. Physica A: Statistical Mechanics and its Applications, 491:820–834, 2018. ISSN 0378-4371. doi: https://doi.org/10.1016/j.physa.2017.09.028. URL https://www.sciencedirect.com/science/article/pii/S0378437117309202.
  48. Caveats of neural persistence in deep neural networks. In 2nd Annual TAG in Machine Learning, 2023.
  49. An adversarial robustness perspective on the topology of neural networks. In NeurIPS ML Safety Workshop, 2022. URL https://openreview.net/forum?id=EtGd7pF237i.
  50. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672–2680, 2014. URL http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf.
  51. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.
  52. Explaining and harnessing adversarial examples, 2015.
  53. Homotopy theory for digraphs, 2014.
  54. P. Grohs and G. Kutyniok. Mathematical Aspects of Deep Learning. Cambridge University Press, 2022.
  55. Improved training of Wasserstein GANs. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 5769–5779, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
  56. W. H. Guss and R. Salakhutdinov. On characterizing the capacity of neural networks using algebraic topology, 2018. URL https://openreview.net/forum?id=H11lAfbCW.
  57. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. doi: 10.1109/CVPR.2016.90.
  58. A survey of topological machine learning methods. Frontiers in Artificial Intelligence, 4, 2021. ISSN 2624-8212. doi: 10.3389/frai.2021.681108. URL https://www.frontiersin.org/articles/10.3389/frai.2021.681108.
  59. Towards a definition of disentangled representations, 2018.
  60. Neurips 2020 Competition: Predicting generalization in deep learning, 2020a.
  61. Fantastic generalization measures and where to find them. In International Conference on Learning Representations, 2020b. URL https://openreview.net/forum?id=SJgIPJBFvH.
  62. Methods and analysis of the first competition in predicting generalization of deep learning. In H. J. Escalante and K. Hofmann, editors, Proceedings of the NeurIPS 2020 Competition and Demonstration Track, volume 133 of Proceedings of Machine Learning Research, pages 170–190. PMLR, 06–12 Dec 2021. URL https://proceedings.mlr.press/v133/jiang21a.html.
  63. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=Hk99zCeAb.
  64. The UCI Machine Learning Repository. https://archive.ics.uci.edu.
  65. A. Khovanskiĭ. Fewnomials. Translations of Mathematical Monographs, 88. American Mathematical Society, Providence, 1991. ISBN 0821845470.
  66. V. Khrulkov and I. Oseledets. Geometry score: A method for comparing generative adversarial networks. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2621–2629. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/khrulkov18a.html.
  67. D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In Y. Bengio and Y. LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http://arxiv.org/abs/1312.6114.
  68. NAS-Bench-NLP: Neural architecture search benchmark for natural language processing, 2020.
  69. The minimal spanning tree and the upper box dimension. Proceedings of the American Mathematical Society, 134(4):1183–1187, 2006. ISSN 00029939, 10886826.
  70. A. Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL http://www.cs.toronto.edu/~kriz/cifar.html.
  71. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf.
  72. S. V. Kurochkin. Neural network with smooth activation functions and without bottlenecks is almost surely a Morse function. Computational Mathematics and Mathematical Physics, 61(7):1162–1168, Jul 2021. ISSN 1555-6662. doi: 10.1134/S0965542521070101. URL https://doi.org/10.1134/S0965542521070101.
  73. Topological uncertainty: Monitoring trained neural networks through persistence of activation graphs. In Z.-H. Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2666–2672. International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 10.24963/ijcai.2021/367. URL https://doi.org/10.24963/ijcai.2021/367. Main Track.
  74. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. doi: 10.1109/5.726791.
  75. A framework for differential calculus on persistence barcodes. Foundations of Computational Mathematics, 22(4):1069–1131, Aug 2022. ISSN 1615-3383. doi: 10.1007/s10208-021-09522-y. URL https://doi.org/10.1007/s10208-021-09522-y.
  76. Finding the homology of decision boundaries with active learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 8355–8365. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/5f14615696649541a025d3d0f8e0447f-Paper.pdf.
  77. ReLU neural networks, polyhedral decompositions, and persistent homology. In 2nd Annual TAG in Machine Learning, 2023.
  78. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  79. V. Lohweg. Banknote Authentication. UCI Machine Learning Repository, 2013. DOI: https://doi.org/10.24432/C55P57.
  80. Topological convolutional neural networks. In TDA & Beyond, 2020. URL https://openreview.net/forum?id=hntbh8Zo1V.
  81. Computing persistent homology of directed flag complexes. Algorithms, 13(1), 2020. ISSN 1999-4893. doi: 10.3390/a13010019. URL https://www.mdpi.com/1999-4893/13/1/19.
  82. dsprites: Disentanglement testing sprites dataset. https://github.com/deepmind/dsprites-dataset/, 2017.
  83. Can neural networks learn persistent homology features? In TDA & Beyond, 2020. URL https://openreview.net/forum?id=pqpXM1Wjsxe.
  84. Topological dynamics of functional neural network graphs during reinforcement learning. In B. Luo, L. Cheng, Z.-G. Wu, H. Li, and C. Li, editors, Neural Information Processing, pages 190–204, Singapore, 2024. Springer Nature Singapore. ISBN 978-981-99-8138-0.
  85. J. R. Munkres. Topology. Prentice Hall, Inc., Upper Saddle River, NJ, 2nd edition, 2000. ISBN 0131816292; 9780131816299; 0131784498; 9780131784499.
  86. Topology of deep neural networks. J. Mach. Learn. Res., 21(1), Jan 2020. ISSN 1532-4435.
  87. Q. Nguyen. On connected sublevel sets in deep learning. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4790–4799. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/nguyen19a.html.
  88. A region-of-interest-reweight 3D convolutional neural network for the analytics of brain information processing. In A. F. Frangi, J. A. Schnabel, C. Davatzikos, C. Alberola-López, and G. Fichtinger, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2018, pages 302–310, Cham, 2018. Springer International Publishing. ISBN 978-3-030-00931-1.
  89. N. I. of Standards and Technology. NIST TrojAI Competition Dataset. https://pages.nist.gov/trojai/docs/index.html#round-1.
  90. Feature visualization. Distill, 2017. doi: 10.23915/distill.00007. https://distill.pub/2017/feature-visualization.
  91. The building blocks of interpretability. Distill, 2018. doi: 10.23915/distill.00010. https://distill.pub/2018/building-blocks.
  92. Architectures of topological deep learning: A survey on topological neural networks, 2023.
  93. G. Petri and A. Leitão. On the topological expressive power of neural networks. In TDA & Beyond, 2020. URL https://openreview.net/forum?id=I44kJPuvqPD.
  94. L. Prechelt. Early Stopping - But When?, pages 55–69. Springer Berlin Heidelberg, Berlin, Heidelberg, 1998. ISBN 978-3-540-49430-0. doi: 10.1007/3-540-49430-8˙3. URL https://doi.org/10.1007/3-540-49430-8_3.
  95. Experimental observations of the topology of convolutional neural network activations. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’23/IAAI’23/EAAI’23. AAAI Press, 2023. ISBN 978-1-57735-880-0. doi: 10.1609/aaai.v37i8.26134. URL https://doi.org/10.1609/aaai.v37i8.26134.
  96. Unsupervised representation learning with deep convolutional generative adversarial networks. In Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxiv.org/abs/1511.06434.
  97. Topological data analysis of decision boundaries with application to model selection. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5351–5360. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/ramamurthy19a.html.
  98. TopoAct: Visually exploring the shape of activations in deep learning. Computer Graphics Forum, 40(1):382–397, 2021. doi: https://doi.org/10.1111/cgf.14195. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.14195.
  99. Topobert: Exploring the topology of fine-tuned word representations. Information Visualization, 22(3):186–208, 2023. doi: 10.1177/14738716231168671. URL https://doi.org/10.1177/14738716231168671.
  100. G. Reeb. Sur les points singuliers d’une forme de Pfaff complètement intégrable ou d’une fonction numérique [On the singular points of a completely integrable Pfaff form or of a numerical function]. Comptes Rendus Acad. Sciences Paris, 222:847–849, 1946.
  101. Cliques of neurons bound into cavities provide a missing link between structure and function. Frontiers in Computational Neuroscience, 11, 2017. ISSN 1662-5188. doi: 10.3389/fncom.2017.00048. URL https://www.frontiersin.org/articles/10.3389/fncom.2017.00048.
  102. B. Rieck. On the expressivity of persistent homology in graph learning, 2023.
  103. Neural persistence: A complexity measure for deep neural networks using algebraic topology. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=ByxkijC5FQ.
  104. B. Schweinhart. Persistent homology and the upper box dimension. Discrete & Computational Geometry, 65(2):331–364, Mar 2021. ISSN 1432-0444. doi: 10.1007/s00454-019-00145-3. URL https://doi.org/10.1007/s00454-019-00145-3.
  105. B. Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin–Madison, 2009. URL http://axon.cs.byu.edu/~martinez/classes/778/Papers/settles.activelearning.pdf.
  106. D. R. Sheehy. Linear-size approximations to the Vietoris–Rips filtration. Discrete & Computational Geometry, 49(4):778–796, Jun 2013. ISSN 1432-0444. doi: 10.1007/s00454-013-9513-1. URL https://doi.org/10.1007/s00454-013-9513-1.
  107. R. Sibson. SLINK: An optimally efficient algorithm for the single-link cluster method. The Computer Journal, 16(1):30–34, 01 1973. ISSN 0010-4620. doi: 10.1093/comjnl/16.1.30. URL https://doi.org/10.1093/comjnl/16.1.30.
  108. Hausdorff dimension, heavy tails, and generalization in neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5138–5151. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/37693cfc748049e45d87b8c7d8b9aacd-Paper.pdf.
  109. Topological Methods for the Analysis of High Dimensional Data Sets and 3D Object Recognition. In M. Botsch, R. Pajarola, B. Chen, and M. Zwicker, editors, Eurographics Symposium on Point-Based Graphics. The Eurographics Association, 2007. ISBN 978-3-905673-51-7. doi: 10.2312/SPBG/SPBG07/091-100.
  110. Y. Skaf and R. Laubenbacher. Topological data analysis in biomedicine: A review. Journal of Biomedical Informatics, 130:104082, 2022. ISSN 1532-0464. doi: https://doi.org/10.1016/j.jbi.2022.104082. URL https://www.sciencedirect.com/science/article/pii/S1532046422000983.
  111. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–9, Los Alamitos, CA, USA, Jun 2015. IEEE Computer Society. doi: 10.1109/CVPR.2015.7298594. URL https://doi.ieeecomputersociety.org/10.1109/CVPR.2015.7298594.
  112. R. E. Tarjan. Efficiency of a good but not linear set union algorithm. J. ACM, 22(2):215–225, apr 1975. ISSN 0004-5411. doi: 10.1145/321879.321884. URL https://doi.org/10.1145/321879.321884.
  113. The GUDHI Project. GUDHI User and Reference Manual. GUDHI Editorial Board, 2015. URL http://gudhi.gforge.inria.fr/doc/latest/.
  114. Wasserstein auto-encoders. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=HkL7n1-0b.
  115. Kepler Mapper: A flexible Python implementation of the Mapper algorithm. Journal of Open Source Software, 4(42):1315, 2019. doi: 10.21105/joss.01315. URL https://doi.org/10.21105/joss.01315.
  116. J. Von Rohrscheidt and B. Rieck. Topological singularity detection at multiple scales. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research. PMLR, 2023.
  117. S. Watanabe and H. Yamana. Deep neural network pruning using persistent homology. In 2020 IEEE Third International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), pages 153–156, 2020. doi: 10.1109/AIKE48582.2020.00030.
  118. S. Watanabe and H. Yamana. Overfitting measurement of convolutional neural networks using trained network weights. International Journal of Data Science and Analytics, 14(3):261–278, Sep 2022a. ISSN 2364-4168. doi: 10.1007/s41060-022-00332-1. URL https://doi.org/10.1007/s41060-022-00332-1.
  119. S. Watanabe and H. Yamana. Topological measurement of deep neural networks using persistent homology. Annals of Mathematics and Artificial Intelligence, 90(1):75–92, Jan 2022b. ISSN 1573-7470. doi: 10.1007/s10472-021-09761-3. URL https://doi.org/10.1007/s10472-021-09761-3.
  120. Activation landscapes as a topological summary of neural network performance. In 2021 IEEE International Conference on Big Data (Big Data), pages 3865–3870, Los Alamitos, CA, USA, dec 2021. IEEE Computer Society. doi: 10.1109/BigData52589.2021.9671368. URL https://doi.ieeecomputersociety.org/10.1109/BigData52589.2021.9671368.
  121. S. Willerton. Spread: A measure of the size of metric spaces. International Journal of Computational Geometry & Applications, 25(03):207–225, 2015. doi: 10.1142/S0218195915500120. URL https://doi.org/10.1142/S0218195915500120.
  122. A review on AI Safety in highly automated driving. Frontiers in Artificial Intelligence, 5, 2022. ISSN 2624-8212. doi: 10.3389/frai.2022.952773. URL https://www.frontiersin.org/articles/10.3389/frai.2022.952773.
  123. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms, 2017.
  124. M. Yacoub and Y. Bennani. HVS: A heuristic for variable selection in multilayer artificial neural network classifier. HAL, 1997, 1997.
  125. Intelligent health care: Applications of deep learning in computational medicine. Frontiers in Genetics, 12, 2021. ISSN 1664-8021. doi: 10.3389/fgene.2021.607471. URL https://www.frontiersin.org/articles/10.3389/fgene.2021.607471.
  126. Assessing the clinical utility of cancer genomic and proteomic data across tumor types. Nat Biotechnol, 32(7):644–652, June 2014.
  127. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64(3):107–115, feb 2021. ISSN 0001-0782. doi: 10.1145/3446776. URL https://doi.org/10.1145/3446776.
  128. GPU-Accelerated computation of Vietoris-Rips persistence barcodes. In 36th International Symposium on Computational Geometry (SoCG 2020). Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2020.
  129. Y. Zhao and H. Zhang. Quantitative performance assessment of CNN units via topological entropy calculation. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=xFOyMwWPkz.
  130. Topological detection of trojaned neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=1r2EannVuIA.
  131. Evaluating the disentanglement of deep generative models through manifold topology. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=djwS0m4Ft_A.
  132. Visualizing and analyzing the topology of neuron activations in deep adversarial training. In Topological, Algebraic and Geometric Learning Workshops 2023, pages 134–145. PMLR, 2023.
Citations (3)
View on Semantic Scholar

Summary

  • The paper presents TDA as a novel approach to unveil neural network structures and clarify decision boundaries.
  • It applies persistent homology and Mapper to trace feature lifecycles and visualize high-dimensional data.
  • The study demonstrates TDA's potential for improving generalization and guiding efficient training while addressing computational challenges.

Deep Insight into Neural Networks with Topological Data Analysis

Understanding Neural Network Structure and Function

Neural networks have the ability to solve complex problems but understanding their intricate structures is a challenge. The application of Topological Data Analysis (TDA), a field that studies the shape of data, offers a novel perspective. By analyzing neural network architectures, the relationships between network components, the boundaries for decision-making in classification problems, and the evolving internal representations during training, TDA provides insights beyond traditional techniques.

TDA Tools: Persistent Homology and Mapper

Key tools in TDA, like persistent homology and Mapper, delve into the complexities of neural networks. Persistent homology traces the "lifecycle" of features across different scales, revealing patterns not immediately apparent. Mapper, on the other hand, creates simplified graphs that capture the high-level structure of data, allowing us to visualize and understand the high-dimensional space that neural networks operate in.

Practical Applications and Implications

Researchers have leveraged TDA in multiple ways. It has been used to characterize decision boundaries in classification tasks and to analyze the weights within networks layer-by-layer. These methods have helped in model selection, identifying and preventing overfitting, and even in detecting adversarial or corrupted data inputs. Generative neural networks have also benefited, with TDA aiding the assessment of model quality and the analysis of disentanglement.

TDA's Role in Training and Generalization

Homology-based techniques have also been used to predict the generalization ability of networks. By regressing the generalization gap based on topological summaries obtained from persistence diagrams, researchers have been able to propose early stopping criteria for training and recommend regularization strategies. This has significant implications for improving the training efficiency and robustness of neural networks.

Challenges and Future Research

Despite TDA's promise, it's not without challenges. The computational demands of TDA tools are significant, which can limit their application to larger or state-of-the-art neural network models. There's also a need for more theoretical support linking TDA characteristics to neural network behavior. Yet, the field is ripe for growth, with potential in creating faster computation algorithms, applying TDA to cutting-edge network architectures, and providing more theoretical grounding.

Conclusion

TDA offers a unique lens through which to examine neural networks, providing researchers and practitioners with the means to explore these computational models' topological landscapes. As the field of neural networks continues to grow, so too does the potential for TDA to play a crucial role in understanding and harnessing the power of these systems. With more research and development, TDA could significantly enhance our ability to design and analyze neural networks across various applications.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown