Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Graph Elimination Networks (2401.01233v1)

Published 2 Jan 2024 in cs.LG

Abstract: Graph Neural Networks (GNNs) are widely applied across various domains, yet they perform poorly in deep layers. Existing research typically attributes this problem to node over-smoothing, where node representations become indistinguishable after multiple rounds of propagation. In this paper, we delve into the neighborhood propagation mechanism of GNNs and discover that the real root cause of GNNs' performance degradation in deep layers lies in ineffective neighborhood feature propagation. This propagation leads to an exponential growth of a node's current representation at every propagation step, making it extremely challenging to capture valuable dependencies between long-distance nodes. To address this issue, we introduce Graph Elimination Networks (GENs), which employ a specific algorithm to eliminate redundancies during neighborhood propagation. We demonstrate that GENs can enhance nodes' perception of distant neighborhoods and extend the depth of network propagation. Extensive experiments show that GENs outperform the state-of-the-art methods on various graph-level and node-level datasets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Rumor Detection on Social Media with Bi-Directional Graph Convolutional Networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 549–556.
  2. Measuring and Relieving the Over-Smoothing Problem for Graph Neural Networks from the Topological View. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 3438–3445.
  3. Structure-Aware Transformer for Graph Representation Learning. In International Conference on Machine Learning. PMLR, 3469–3489.
  4. Scalable Graph Neural Networks via Bidirectional Propagation. Advances in Neural Information Processing Systems 33 (2020), 14556–14566.
  5. Simple and Deep Graph Convolutional Networks. In International Conference on Machine Learning. PMLR, 1725–1735.
  6. Graph Transformation Policy Network for Chemical Reaction Prediction. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 750–760.
  7. Graph Neural Networks for Social Recommendation. In The World Wide Web Conference. 417–426.
  8. Graph Random Neural Networks for Semi-Supervised Learning on Graphs. Advances in Neural Information Processing Systems 33 (2020), 22092–22103.
  9. Predict then Propagate: Graph Neural Networks Meet Personalized PageRank. arXiv preprint arXiv:1810.05997 (2018).
  10. Neural Message Passing for Quantum Chemistry. In International Conference on Machine Learning. PMLR, 1263–1272.
  11. Orthogonal Graph Neural Networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 3996–4004.
  12. Inductive Representation Learning on Large Graphs. Advances in Neural Information Processing Systems 30 (2017).
  13. Open Graph Benchmark: Datasets for Machine Learning on Graphs. Advances in Neural Information Processing Systems 33 (2020), 22118–22133.
  14. Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective. arXiv preprint arXiv:2206.07743 (2022).
  15. Thomas N Kipf and Max Welling. 2016. Semi-Supervised Classification with Graph Convolutional Networks. arXiv preprint arXiv:1609.02907 (2016).
  16. GemNet: Universal Directional Graph Neural Networks for Molecules. arXiv e-prints (2021), arXiv–2106.
  17. DeepGCNs: Can GCNs Go as Deep as CNNs?. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9267–9276.
  18. DeeperGCN: All You Need to Train Deeper GCNs. arXiv preprint arXiv:2006.07739 (2020).
  19. Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.
  20. GeomGCL: Geometric Graph Contrastive Learning for Molecular Property Prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 4541–4549.
  21. Towards Deeper Graph Neural Networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 338–348.
  22. TUDataset: A Collection of Benchmark Datasets for Learning with Graphs. arXiv preprint arXiv:2007.08663 (2020).
  23. Kenta Oono and Taiji Suzuki. 2019. Graph Neural Networks Exponentially Lose Expressive Power for Node Classification. arXiv preprint arXiv:1905.10947 (2019).
  24. DropEdge: Towards Deep Graph Convolutional Networks on Node Classification. arXiv preprint arXiv:1907.10903 (2019).
  25. Graph-Coupled Oscillator Networks. In International Conference on Machine Learning. PMLR, 18888–18909.
  26. Learning to Simulate Complex Physics with Graph Networks. In International Conference on Machine Learning. PMLR, 8459–8468.
  27. Ordered GNN: Ordering Message Passing to Deal with Heterophily and Over-smoothing. arXiv preprint arXiv:2302.01524 (2023).
  28. Adaptive Propagation Graph Convolutional Network. IEEE Transactions on Neural Networks and Learning Systems 32, 10 (2020), 4755–4760.
  29. Attention Is All You Need. Advances in Neural Information Processing Systems 30 (2017).
  30. Graph Attention Networks. arXiv preprint arXiv:1710.10903 (2017).
  31. A Comprehensive Survey on Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems 32, 1 (2020), 4–24.
  32. How Powerful are Graph Neural Networks? arXiv preprint arXiv:1810.00826 (2018).
  33. Predicting Chemical Shifts with Graph Neural Networks. Chemical Science 12, 32 (2021), 10802–10809.
  34. Do Transformers Really Perform Bad for Graph Representation? URL https://arxiv. org/abs/2106.05234 (2021).
  35. Evaluating Deep Graph Neural Networks. arXiv preprint arXiv:2108.00955 (2021).
  36. Model Degradation Hinders Deep Graph Neural Networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2493–2503.
  37. Node Dependent Local Smoothing for Scalable Graph Learning. Advances in Neural Information Processing Systems 34 (2021), 20321–20332.
  38. CSGNN: Contrastive Self-Supervised Graph Neural Network for Molecular Interaction Prediction.. In IJCAI. 3756–3763.
  39. Lingxiao Zhao and Leman Akoglu. 2019. PairNorm: Tackling Oversmoothing in GNNs. arXiv preprint arXiv:1909.12223 (2019).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Shuo Wang (382 papers)
  2. Ge Cheng (5 papers)
  3. Yun Zhang (103 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now