Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Structure-Aware DropEdge Towards Deep Graph Convolutional Networks (2306.12091v1)

Published 21 Jun 2023 in cs.LG

Abstract: It has been discovered that Graph Convolutional Networks (GCNs) encounter a remarkable drop in performance when multiple layers are piled up. The main factor that accounts for why deep GCNs fail lies in over-smoothing, which isolates the network output from the input with the increase of network depth, weakening expressivity and trainability. In this paper, we start by investigating refined measures upon DropEdge -- an existing simple yet effective technique to relieve over-smoothing. We term our method as DropEdge++ for its two structure-aware samplers in contrast to DropEdge: layer-dependent sampler and feature-dependent sampler. Regarding the layer-dependent sampler, we interestingly find that increasingly sampling edges from the bottom layer yields superior performance than the decreasing counterpart as well as DropEdge. We theoretically reveal this phenomenon with Mean-Edge-Number (MEN), a metric closely related to over-smoothing. For the feature-dependent sampler, we associate the edge sampling probability with the feature similarity of node pairs, and prove that it further correlates the convergence subspace of the output layer with the input features. Extensive experiments on several node classification benchmarks, including both full- and semi- supervised tasks, illustrate the efficacy of DropEdge++ and its compatibility with a variety of backbones by achieving generally better performance over DropEdge and the no-drop version.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in ICLR, 2017.
  2. K. Xu, C. Li, Y. Tian, T. Sonobe, K.-i. Kawarabayashi, and S. Jegelka, “Representation learning on graphs with jumping knowledge networks,” arXiv preprint arXiv:1806.03536, 2018.
  3. T. Zhang, Q. Wu, and J. Yan, “Learning high-order graph convolutional networks via adaptive layerwise aggregation combination,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–12, 2021.
  4. K. Xu, H. Huang, P. Deng, and Y. Li, “Deep feature aggregation framework driven by graph convolutional network for scene classification in remote sensing,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 10, pp. 5751–5765, 2022.
  5. J. Chen, S. Chen, M. Bai, J. Pu, J. Zhang, and J. Gao, “Graph decoupling attention markov networks for semisupervised graph node classification,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–15, 2022.
  6. Y. Qian, P. Expert, T. Rieu, P. Panzarasa, and M. Barahona, “Quantifying the alignment of graph and features in deep learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 4, pp. 1663–1672, 2022.
  7. M. Gong, H. Zhou, A. Qin, W. Liu, and Z. Zhao, “Self-paced co-training of graph neural networks for semi-supervised node classification,” IEEE Transactions on Neural Networks and Learning Systems, 2022.
  8. F. Fouss, L. Yen, A. Pirotte, and M. Saerens, “An experimental investigation of graph kernels on a collaborative recommendation task,” in Sixth International Conference on Data Mining (ICDM’06).   IEEE, 2006, pp. 863–868.
  9. B. Perozzi, R. Al-Rfou, and S. Skiena, “Deepwalk: Online learning of social representations,” in Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining.   ACM, 2014, pp. 701–710.
  10. A. Grover and J. Leskovec, “node2vec: Scalable feature learning for networks,” in Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining.   ACM, 2016, pp. 855–864.
  11. Z. Liu, Y. Yi, and X. Luo, “A high-order proximity-incorporated nonnegative matrix factorization-based community detector,” IEEE Transactions on Emerging Topics in Computational Intelligence, pp. 1–15, 2023.
  12. Z. Liu, G. Yuan, and X. Luo, “Symmetry and nonnegativity-constrained matrix factorization for community detection,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 9, pp. 1691–1693, 2022.
  13. Z. Liu, X. Luo, and Z. Wang, “Convergence analysis of single latent factor-dependent, nonnegative, and multiplicative update-based nonnegative latent factor models,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 4, pp. 1737–1749, 2021.
  14. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  15. Q. Li, Z. Han, and X.-M. Wu, “Deeper insights into graph convolutional networks for semi-supervised learning,” in Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
  16. K. Oono and T. Suzuki, “Graph neural networks exponentially lose expressive power for node classification,” in ICLR, 2020.
  17. J. Klicpera, A. Bojchevski, and S. Günnemann, “Predict then propagate: Graph neural networks meet personalized pagerank,” 2018.
  18. M. Chen, Z. Wei, Z. Huang, B. Ding, and Y. Li, “Simple and deep graph convolutional networks,” in International Conference on Machine Learning.   PMLR, 2020, pp. 1725–1735.
  19. D. Chen, Y. Lin, W. Li, P. Li, J. Zhou, and X. Sun, “Measuring and relieving the over-smoothing problem for graph neural networks from the topological view,” arXiv preprint arXiv:1909.03211, 2019.
  20. Y. Rong, W. Huang, T. Xu, and J. Huang, “Dropedge: Towards deep graph convolutional networks on node classification,” in ICLR, 2020.
  21. L. Zhao and L. Akoglu, “Pairnorm: Tackling oversmoothing in gnns,” in ICLR, 2020.
  22. K. Zhou, X. Huang, Y. Li, D. Zha, R. Chen, and X. Hu, “Towards deeper graph neural networks with differentiable group normalization,” arXiv preprint arXiv:2006.06972, 2020.
  23. Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu, “A comprehensive survey on graph neural networks,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 1, pp. 4–24, 2021.
  24. J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun, “Spectral networks and locally connected networks on graphs,” in Proceedings of International Conference on Learning Representations, 2013.
  25. M. Defferrard, X. Bresson, and P. Vandergheynst, “Convolutional neural networks on graphs with fast localized spectral filtering,” in Advances in Neural Information Processing Systems, 2016, pp. 3844–3852.
  26. M. Henaff, J. Bruna, and Y. LeCun, “Deep convolutional networks on graph-structured data,” arXiv preprint arXiv:1506.05163, 2015.
  27. R. Li, S. Wang, F. Zhu, and J. Huang, “Adaptive graph convolutional neural networks,” in Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
  28. R. Levie, F. Monti, X. Bresson, and M. M. Bronstein, “Cayleynets: Graph convolutional neural networks with complex rational spectral filters,” IEEE Transactions on Signal Processing, vol. 67, no. 1, pp. 97–109, 2017.
  29. X. He, B. Wang, Y. Hu, J. Gao, Y. Sun, and B. Yin, “Parallelly adaptive graph convolutional clustering model,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–14, 2022.
  30. P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in ICLR, 2018.
  31. A. Feng, C. You, S. Wang, and L. Tassiulas, “Kergnns: Interpretable graph neural networks with graph kernels,” arXiv preprint arXiv:2201.00491, 2022.
  32. W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in Advances in Neural Information Processing Systems, 2017, pp. 1025–1035.
  33. F. Monti, D. Boscaini, J. Masci, E. Rodola, J. Svoboda, and M. M. Bronstein, “Geometric deep learning on graphs and manifolds using mixture model cnns,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 5115–5124.
  34. M. Niepert, M. Ahmed, and K. Kutzkov, “Learning convolutional neural networks for graphs,” in International conference on machine learning, 2016, pp. 2014–2023.
  35. H. Gao, Z. Wang, and S. Ji, “Large-scale learnable graph convolutional networks,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.   ACM, 2018, pp. 1416–1424.
  36. J. Chen, T. Ma, and C. Xiao, “Fastgcn: Fast learning with graph convolutional networks via importance sampling,” in Proceedings of the 6th International Conference on Learning Representations, 2018.
  37. W. Huang, T. Zhang, Y. Rong, and J. Huang, “Adaptive sampling towards fast graph representation learning,” in Advances in Neural Information Processing Systems, 2018, pp. 4558–4567.
  38. H. Zeng, H. Zhou, A. Srivastava, R. Kannan, and V. Prasanna, “Graphsaint: Graph sampling based inductive learning method,” arXiv preprint arXiv:1907.04931, 2019.
  39. W. Huang, Y. Rong, T. Xu, F. Sun, and J. Huang, “Tackling over-smoothing for general graph convolutional networks,” arXiv e-prints, pp. arXiv–2008, 2020.
  40. P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Galligher, and T. Eliassi-Rad, “Collective classification in network data,” AI magazine, vol. 29, no. 3, p. 93, 2008.
  41. Z. Yang, W. Cohen, and R. Salakhudinov, “Revisiting semi-supervised learning with graph embeddings,” in International conference on machine learning.   PMLR, 2016, pp. 40–48.
  42. O. Shchur, M. Mumme, A. Bojchevski, and S. Günnemann, “Pitfalls of graph neural network evaluation,” arXiv preprint arXiv:1811.05868, 2018.
  43. H. Chang, Y. Rong, T. Xu, W. Huang, S. Sojoudi, J. Huang, and W. Zhu, “Spectral graph attention network,” 2020.
  44. A. Fout, J. Byrd, B. Shariat, and A. Ben-Hur, “Protein interface prediction using graph convolutional networks,” in Advances in Neural Information Processing Systems, 2017, pp. 6530–6539.
  45. J. Klicpera, S. Weißenberger, and S. Günnemann, “Diffusion improves graph learning,” 2019.
  46. E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” in International Conference on Learning Representations, 2017.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now