Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning Invariant Representations of Graph Neural Networks via Cluster Generalization (2403.03599v1)

Published 6 Mar 2024 in cs.LG

Abstract: Graph neural networks (GNNs) have become increasingly popular in modeling graph-structured data due to their ability to learn node representations by aggregating local structure information. However, it is widely acknowledged that the test graph structure may differ from the training graph structure, resulting in a structure shift. In this paper, we experimentally find that the performance of GNNs drops significantly when the structure shift happens, suggesting that the learned models may be biased towards specific structure patterns. To address this challenge, we propose the Cluster Information Transfer (CIT) mechanism (Code available at https://github.com/BUPT-GAMMA/CITGNN), which can learn invariant representations for GNNs, thereby improving their generalization ability to various and unknown test graphs with structure shift. The CIT mechanism achieves this by combining different cluster information with the nodes while preserving their cluster-independent information. By generating nodes across different clusters, the mechanism significantly enhances the diversity of the nodes and helps GNNs learn the invariant representations. We provide a theoretical analysis of the CIT mechanism, showing that the impact of changing clusters during structure shift can be mitigated after transfer. Additionally, the proposed mechanism is a plug-in that can be easily used to improve existing GNNs. We comprehensively evaluate our proposed method on three typical structure shift scenarios, demonstrating its effectiveness in enhancing GNNs' performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. Theodore Wilbur Anderson. An introduction to multivariate statistical analysis. Technical report, Wiley New York, 1962.
  2. Spectral clustering with graph neural networks for graph pooling. In International Conference on Machine Learning, pages 874–883. PMLR, 2020.
  3. A convex optimization approach to high-dimensional sparse quadratic discriminant analysis. The Annals of Statistics, 49(3):1537–1568, 2021.
  4. Exploiting centrality information with graph convolutions for network representation learning. In 2019 IEEE 35th International Conference on Data Engineering (ICDE), pages 590–601. IEEE, 2019.
  5. Simple and deep graph convolutional networks. In International Conference on Machine Learning, pages 1725–1735. PMLR, 2020.
  6. A simple and powerful framework for stable dynamic network embedding. arXiv preprint arXiv:2311.09251, 2023.
  7. A fair comparison of graph neural networks for graph classification. arXiv preprint arXiv:1912.09893, 2019.
  8. Collective spammer detection in evolving multi-relational social networks. In Proceedings of the 21th acm sigkdd international conference on knowledge discovery and data mining, pages 1769–1778, 2015.
  9. Debiased graph neural networks with agnostic label selection bias. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  10. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997, 2018.
  11. Good: A graph out-of-distribution benchmark. arXiv preprint arXiv:2206.08452, 2022.
  12. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118–22133, 2020.
  13. Graph invariant learning with subgraph co-mixup for out-of-distribution generalization. arXiv preprint arXiv:2312.10988, 2023.
  14. Stochastic blockmodels and community structure in networks. Physical review E, 83(1):016107, 2011.
  15. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
  16. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308, 2016.
  17. Learning invariant graph representations for out-of-distribution generalization. In Advances in Neural Information Processing Systems, 2022.
  18. Uncertainty modeling for out-of-distribution generalization. arXiv preprint arXiv:2202.03958, 2022.
  19. New benchmarks for learning on non-homophilous graphs. arXiv preprint arXiv:2104.01404, 2021.
  20. Interpretable and generalizable graph learning via stochastic attention mechanism. In International Conference on Machine Learning, pages 15524–15543. PMLR, 2022.
  21. Emmanuel Müller. Graph clustering with graph neural networks. Journal of Machine Learning Research, 24:1–21, 2023.
  22. Collective classification in network data. AI magazine, 29(3):93–93, 2008.
  23. Probabilistic face embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6902–6911, 2019.
  24. Graph-structured representations for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2017.
  25. T Tony Cai and Linjun Zhang. High dimensional linear discriminant analysis: optimality, adaptive algorithm and missing data. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 81(4):675–705, 2019.
  26. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.
  27. Attributed graph clustering: A deep attentional embedding approach. arXiv preprint arXiv:1906.06532, 2019.
  28. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022.
  29. Community preserving network embedding. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
  30. Heterogeneous graph attention network. In The world wide web conference, pages 2022–2032, 2019.
  31. Handling distribution shifts on graphs: An invariance perspective. arXiv preprint arXiv:2202.02466, 2022.
  32. Discovering invariant rationales for graph neural networks. arXiv preprint arXiv:2201.12872, 2022.
  33. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning, pages 40–48. PMLR, 2016.
  34. Improving out-of-distribution robustness via selective augmentation. In International Conference on Machine Learning, pages 25407–25437. PMLR, 2022.
  35. Design space for graph neural networks. Advances in Neural Information Processing Systems, 33:17009–17021, 2020.
  36. Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018.
  37. An end-to-end deep learning architecture for graph classification. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  38. Domain generalization in vision: A survey. arXiv preprint arXiv:2103.02503, 2021.
  39. Interpreting and unifying graph neural networks with an optimization framework. In Proceedings of the Web Conference 2021, pages 1215–1226, 2021.
  40. Shift-robust gnns: Overcoming the limitations of localized graph training data. Advances in Neural Information Processing Systems, 34:27965–27977, 2021.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now