Papers
Topics
Authors
Recent
Search
2000 character limit reached

Homological Neural Networks: A Sparse Architecture for Multivariate Complexity

Published 27 Jun 2023 in cs.LG and cs.AI | (2306.15337v1)

Abstract: The rapid progress of Artificial Intelligence research came with the development of increasingly complex deep learning models, leading to growing challenges in terms of computational complexity, energy efficiency and interpretability. In this study, we apply advanced network-based information filtering techniques to design a novel deep neural network unit characterized by a sparse higher-order graphical architecture built over the homological structure of underlying data. We demonstrate its effectiveness in two application domains which are traditionally challenging for deep learning: tabular data and time series regression problems. Results demonstrate the advantages of this novel design which can tie or overcome the results of state-of-the-art machine learning and deep learning models using only a fraction of parameters.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC), 13:1 – 18, 2015.
  2. Tabnet: Attentive interpretable tabular learning. ArXiv, abs/1908.07442, 2019.
  3. Aste, T. Topological regularization with information filtering networks. Information Sciences, arXiv:2005.04692, 2022.
  4. Sparse causality network retrieval from short time series. Complex., 2017:4518429:1–4518429:13, 2017.
  5. Parsimonious modeling with information filtering networks. Physical Review E, 94(6), Dec 2016. ISSN 2470-0053. doi: 10.1103/physreve.94.062306. URL http://dx.doi.org/10.1103/PhysRevE.94.062306.
  6. Deep neural networks and tabular data: A survey. IEEE transactions on neural networks and learning systems, PP, 2021.
  7. Dependency structures in cryptocurrency market from high to low frequency. 2022.
  8. Topological feature selection: A graph-based filter feature selection approach. ArXiv, abs/2302.09543, 2023.
  9. Anatomy of a stablecoin’s failure: the terra-luna case. ArXiv, abs/2207.13914, 2022.
  10. Improved large-scale graph learning through ridge spectral sparsification. In ICML, 2018.
  11. Spectral sparsification in spectral clustering. 2016 23rd International Conference on Pattern Recognition (ICPR), pp.  2301–2306, 2016.
  12. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp.  785–794, 2016.
  13. Prevalence and characteristics of autism spectrum disorder among children aged 8 years — autism and developmental disabilities monitoring network, 11 sites, united states, 2012. MMWR Surveillance Summaries, 65:1 – 23, 2016.
  14. Complexity and categorical analysis may improve the interpretation of agreement studies using continuous variables. Journal of evaluation in clinical practice, 17 3:511–4, 2011.
  15. Simplicial neural networks. arXiv preprint arXiv:2010.03633, 2020.
  16. Friedman, J. H. Greedy function approximation: a gradient boosting machine. Annals of statistics, pp.  1189–1232, 2001.
  17. Why do tree-based models still outperform deep learning on tabular data? ArXiv, abs/2207.08815, 2022.
  18. Bayesian graph neural networks with adaptive connection sampling. ArXiv, abs/2006.04064, 2020.
  19. A neuroevolution approach to general atari game playing. IEEE Transactions on Computational Intelligence and AI in Games, 6:355–366, 2014.
  20. Tabpfn: A transformer that solves small tabular classification problems in a second. 2022.
  21. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. ArXiv, abs/1607.03250, 2016.
  22. Lightgbm: A highly efficient gradient boosting decision tree. In NIPS, 2017.
  23. How to find your friendly neighborhood: Graph attention design with self-supervision. In ICLR, 2021.
  24. Kruskal, J. B. On the shortest spanning subtree of a graph and the traveling salesman problem. 1956.
  25. Modeling long- and short-term temporal patterns with deep neural networks. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, 2017.
  26. Snip: Single-shot network pruning based on connection sensitivity. ArXiv, abs/1810.02340, 2018.
  27. Sparse convolutional neural networks. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.  806–814, 2015.
  28. Learning efficient convolutional networks through network slimming. 2017 IEEE International Conference on Computer Vision (ICCV), pp.  2755–2763, 2017.
  29. Learning sparse neural networks through l0 regularization. ArXiv, abs/1712.01312, 2017.
  30. A cnn-lstm-based model to forecast stock prices. Complex., 2020:6622927:1–6622927:10, 2020.
  31. Learning to drop: Robust graph neural network via topological denoising. Proceedings of the 14th ACM International Conference on Web Search and Data Mining, 2021.
  32. Mantegna, R. N. Hierarchical structure in financial markets. The European Physical Journal B - Condensed Matter and Complex Systems, 11:193–197, 1998.
  33. Learning clique forests. ArXiv, abs/1905.02266, 2019.
  34. Network filtering for big data: Triangulated maximally filtered graph. J. Complex Networks, 5:161–178, 2017.
  35. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9, 2017.
  36. Variational dropout sparsifies deep neural networks. ArXiv, abs/1701.05369, 2017.
  37. Pruning convolutional neural networks for resource efficient transfer learning. ArXiv, abs/1611.06440, 2016.
  38. Moyano, L. G. Learning network representations. The European Physical Journal Special Topics, 226:499–518, 2017.
  39. Otakar boruvka on minimum spanning tree problem translation of both the 1926 papers, comments, history. Discrete mathematics, 233(1-3):3–36, 2001.
  40. Collaborative channel pruning for deep networks. In International Conference on Machine Learning, 2019.
  41. The dual information bottleneck. arXiv preprint arXiv:2006.04641, 2020.
  42. Theoretical issues in deep networks. Proceedings of the National Academy of Sciences, 117(48):30039–30045, 2020.
  43. Prim, R. C. Shortest connection networks and some generalizations. Bell System Technical Journal, 36:1389–1401, 1957.
  44. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018.
  45. Gaussian processes for timeseries modelling. 2012.
  46. Pmlb v1.0: an open source dataset collection for benchmarking machine learning methods. arXiv preprint arXiv:2012.00058v2, 2021.
  47. Dropedge: Towards deep graph convolutional networks on node classification. In ICLR, 2020.
  48. Explaining deep neural networks and beyond: A review of methods and applications. Proceedings of the IEEE, 109:247–278, 2021.
  49. Pre-training enhanced spatial-temporal graph neural network for multivariate time series forecasting. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022a.
  50. Decoupled dynamic spatial-temporal graph neural network for traffic forecasting. ArXiv, abs/2206.09112, 2022b.
  51. Temporal pattern attention for multivariate time series forecasting. Machine Learning, pp.  1–21, 2018.
  52. Tabular data: Deep learning is not all you need. Inf. Fusion, 81:84–90, 2022.
  53. Representation compression and generalization in deep neural networks, 2018.
  54. Evolving neural networks through augmenting topologies. Evolutionary Computation, 10:99–127, 2002.
  55. The brain as a complex system: Using network science as a tool for understanding the brain. Brain connectivity, 1 (4):295–308, 2011.
  56. Simplicial complexes: higher-order spectral dimension and dynamics. Journal of Physics: Complexity, 1, 2020.
  57. A tool for filtering information in complex systems. Proceedings of the National Academy of Sciences, 102(30):10421–10426, 2005. ISSN 0027-8424. doi: 10.1073/pnas.0500298102. URL https://www.pnas.org/content/102/30/10421.
  58. Ftx’s downfall and binance’s consolidation: The fragility of centralized digital finance. SSRN Electronic Journal, 2023.
  59. Multivariate temporal convolutional network: A deep neural networks approach for multivariate time series forecasting. Electronics, 2019.
  60. Exploring linear relationship in feature map subspace for convnets compression. ArXiv, abs/1803.05729, 2018.
  61. Network filtering of spatial-temporal gnn for multivariate time-series prediction. Proceedings of the Third ACM International Conference on AI in Finance, 2022.
  62. Learning structured sparsity in deep neural networks. ArXiv, abs/1608.03665, 2016.
  63. Connecting the dots: Multivariate time series forecasting with graph neural networks. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
  64. Simplicial convolutional neural networks. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.  8847–8851. IEEE, 2022.
  65. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. ArXiv, abs/1802.00124, 2018.
  66. Nisp: Pruning networks using neuron importance score propagation. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  9194–9203, 2017.
  67. Zhang, G. P. Time series forecasting using a hybrid arima and neural network model. Neurocomputing, 50:159–175, 2003.
  68. Gman: A graph multi-attention network for traffic prediction. ArXiv, abs/1911.08415, 2020.
  69. Discrimination-aware channel pruning for deep neural networks. In Neural Information Processing Systems, 2018.
  70. Scsp: Spectral clustering filter pruning with soft self-adaption manners. ArXiv, abs/1806.05320, 2018.
  71. Vector autoregressive models for multivariate time series. 2003.
Citations (4)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up