Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
173 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Joint Diffusion Processes as an Inductive Bias in Sheaf Neural Networks (2407.20597v1)

Published 30 Jul 2024 in cs.LG and math.DS

Abstract: Sheaf Neural Networks (SNNs) naturally extend Graph Neural Networks (GNNs) by endowing a cellular sheaf over the graph, equipping nodes and edges with vector spaces and defining linear mappings between them. While the attached geometric structure has proven to be useful in analyzing heterophily and oversmoothing, so far the methods by which the sheaf is computed do not always guarantee a good performance in such settings. In this work, drawing inspiration from opinion dynamics concepts, we propose two novel sheaf learning approaches that (i) provide a more intuitive understanding of the involved structure maps, (ii) introduce a useful inductive bias for heterophily and oversmoothing, and (iii) infer the sheaf in a way that does not scale with the number of features, thus using fewer learnable parameters than existing methods. In our evaluation, we show the limitations of the real-world benchmarks used so far on SNNs, and design a new synthetic task -- leveraging the symmetries of n-dimensional ellipsoids -- that enables us to better assess the strengths and weaknesses of sheaf-based models. Our extensive experimentation on these novel datasets reveals valuable insights into the scenarios and contexts where SNNs in general -- and our proposed approaches in particular -- can be beneficial.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (26)
  1. Sheaf neural networks with connection laplacians. In Topological, Algebraic and Geometric Learning Workshops 2022, pp.  28–36. PMLR, 2022.
  2. Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in gnns. Advances in Neural Information Processing Systems, 35:18527–18541, 2022.
  3. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.  3438–3445, 2020.
  4. Curry, J. M. Sheaves, cosheaves and applications. University of Pennsylvania, 2014.
  5. Sheaf hypergraph networks. Advances in Neural Information Processing Systems, 36, 2024.
  6. A hitchhiker’s guide to geometric gnns for 3d atomic systems. arXiv preprint arXiv:2312.07511, 2023.
  7. A survey on decentralized federated learning. arXiv preprint arXiv:2308.04604, 2023.
  8. Sheaf neural networks. arXiv preprint arXiv:2012.06333, 2020.
  9. Toward a spectral theory of cellular sheaves. Journal of Applied and Computational Topology, 3(4):315–358, 2019.
  10. Opinion dynamics on discourse sheaves. SIAM Journal on Applied Mathematics, 81(5):2033–2060, 2021.
  11. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR, 2017.
  12. Revisiting heterophily for graph neural networks. Advances in neural information processing systems, 35:1362–1375, 2022.
  13. Revisiting graph neural networks: All we have is low-pass filters. arXiv preprint arXiv:1905.09550, 2019.
  14. Graph neural networks exponentially lose expressive power for node classification. arXiv preprint arXiv:1905.10947, 2019.
  15. Characterizing graph datasets for node classification: Homophily-heterophily dichotomy and beyond. Advances in Neural Information Processing Systems, 36, 2023.
  16. Deepinf: Social influence prediction with deep learning. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp.  2110–2119, 2018.
  17. Unveiling the potential of graph neural networks for network modeling and optimization in sdn. In Proceedings of the 2019 ACM Symposium on SDN Research, pp.  140–151, 2019.
  18. The graph neural network model. IEEE transactions on neural networks, 20(1):61–80, 2008.
  19. Surfing on the neural sheaf. In NeurIPS 2022 Workshop on Symmetry and Geometry in Neural Representations, 2022.
  20. Graph Attention Networks. In ICLR, 2018.
  21. Collective dynamics of ‘small-world’networks. nature, 393(6684):440–442, 1998.
  22. Opinion dynamics: A multidisciplinary review and perspective on future research. International Journal of Knowledge and Systems Science (IJKSS), 2(4):72–91, 2011.
  23. Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks. In 2022 IEEE International Conference on Data Mining (ICDM), pp.  1287–1292. IEEE, 2022.
  24. Zaghen, O. Nonlinear sheaf diffusion in graph neural networks. arXiv preprint arXiv:2403.00337, 2024.
  25. Graph neural networks: A review of methods and applications. AI open, 1:57–81, 2020.
  26. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in neural information processing systems, 33:7793–7804, 2020.

Summary

We haven't generated a summary for this paper yet.