Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Theoretical and Empirical Insights into the Origins of Degree Bias in Graph Neural Networks (2404.03139v2)

Published 4 Apr 2024 in cs.LG and cs.SI

Abstract: Graph Neural Networks (GNNs) often perform better for high-degree nodes than low-degree nodes on node classification tasks. This degree bias can reinforce social marginalization by, e.g., privileging celebrities and other high-degree actors in social networks during social and content recommendation. While researchers have proposed numerous hypotheses for why GNN degree bias occurs, we find via a survey of 38 degree bias papers that these hypotheses are often not rigorously validated, and can even be contradictory. Thus, we provide an analysis of the origins of degree bias in message-passing GNNs with different graph filters. We prove that high-degree test nodes tend to have a lower probability of misclassification regardless of how GNNs are trained. Moreover, we show that degree bias arises from a variety of factors that are associated with a node's degree (e.g., homophily of neighbors, diversity of neighbors). Furthermore, we show that during training, some GNNs may adjust their loss on low-degree nodes more slowly than on high-degree nodes; however, with sufficiently many epochs of training, message-passing GNNs can achieve their maximum possible training accuracy, which is not significantly limited by their expressive power. Throughout our analysis, we connect our findings to previously-proposed hypotheses for the origins of degree bias, supporting and unifying some while drawing doubt to others. We validate our theoretical findings on 8 common real-world networks, and based on our theoretical and empirical insights, describe a roadmap to alleviate degree bias.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Towards a unified framework for fair and stable graph representation learning. 2021.
  2. Albert-László Barabási. Network science. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371, 2016.
  3. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. In International Conference on Learning Representations, 2018.
  4. Implications of topological imbalance for representation learning on biomedical knowledge graphs. Briefings in Bioinformatics, 23(5):bbac279, 07 2022.
  5. How graph convolutions amplify popularity bias for recommendation? Frontiers of Computer Science, 18(5), December 2023.
  6. Simple and deep graph convolutional networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1725–1735. PMLR, 13–18 Jul 2020.
  7. Anchor-enhanced geographical entity representation learning. IEEE Transactions on Neural Networks and Learning Systems, pages 1–15, 2023.
  8. Ba-gnn: On learning bias-aware graph neural network. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 3012–3024, 2022.
  9. Contextual stochastic block models. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  10. Inductive dummy-based homogeneous neighborhood augmentation for graph collaborative filtering. In 2023 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2023.
  11. Studying the effect of GNN spatial convolutions on the embedding space’s geometry. In Robin J. Evans and Ilya Shpitser, editors, Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pages 539–548. PMLR, 31 Jul–04 Aug 2023.
  12. Brian Everitt. The Cambridge dictionary of statistics. Cambridge University Press, Cambridge, UK; New York, 2002.
  13. Should graph convolution trust neighbors? a simple causal inference method. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, page 1208–1218, New York, NY, USA, 2021. Association for Computing Machinery.
  14. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
  15. Understanding convolution on graphs via energies. Transactions on Machine Learning Research, 2023.
  16. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 1025–1035, Red Hook, NY, USA, 2017. Curran Associates Inc.
  17. Towards label position bias in graph neural networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  18. Marginal nodes matter: Towards structure fairness in graphs, 2023.
  19. Mitigating degree biases in message passing mechanism by utilizing community structures, 2023.
  20. Graphpatcher: Mitigating degree bias for graph neural networks via test-time augmentation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  21. Rawlsgcn: Towards rawlsian difference principle on graph convolutional network. In Proceedings of the ACM Web Conference 2022, WWW ’22, page 1214–1225, New York, NY, USA, 2022. Association for Computing Machinery.
  22. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
  23. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
  24. On dyadic fairness: Exploring and mitigating bias in graph connections. In International Conference on Learning Representations, 2021.
  25. A metadata-driven approach to understand graph neural networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  26. An explicitly weighted gcn aggregator based on temporal and popularity features for recommendation. ACM Trans. Recomm. Syst., 1(2), apr 2023.
  27. Tackling long-tailed distribution issue in graph neural networks via normalization. IEEE Transactions on Knowledge and Data Engineering, pages 1–11, 2023.
  28. Sailor: Structural augmentation based tail node representation learning. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM ’23, page 1389–1399, New York, NY, USA, 2023. Association for Computing Machinery.
  29. Local augmentation for graph neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 14054–14072. PMLR, 17–23 Jul 2022.
  30. Trade less accuracy for fairness and trade-off explanation for gnn. In 2022 IEEE International Conference on Big Data (Big Data), pages 4681–4690, 2022.
  31. Locality-aware tail node embeddings on homogeneous and heterogeneous networks. IEEE Transactions on Knowledge and Data Engineering, pages 1–16, 2023.
  32. Tail-gnn: Tail-node graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD ’21, page 1109–1119, New York, NY, USA, 2021. Association for Computing Machinery.
  33. On generalized degree fairness in graph neural networks. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’23/IAAI’23/EAAI’23. AAAI Press, 2023.
  34. Is homophily a necessity for graph neural networks? In International Conference on Learning Representations, 2022.
  35. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Curran Associates Inc., Red Hook, NY, USA, 2019.
  36. Multi-Scale attributed node embedding. Journal of Complex Networks, 9(2):cnab014, 05 2021.
  37. Pitfalls of graph neural network evaluation. ArXiv, abs/1811.05868, 2018.
  38. Toward degree bias in embedding-based knowledge graph completion. In Proceedings of the ACM Web Conference 2023, WWW ’23, page 705–715, New York, NY, USA, 2023. Association for Computing Machinery.
  39. Networked inequality: Preferential attachment bias in graph neural network link prediction. In NeurIPS 2023 Workshop: New Frontiers in Graph Learning, 2023.
  40. Investigating and mitigating degree-related biases in graph convoltuional networks. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM ’20, page 1435–1444, New York, NY, USA, 2020. Association for Computing Machinery.
  41. Graph attention networks. In International Conference on Learning Representations, 2018.
  42. Blade: Biased neighborhood sampling based graph neural network for directed graphs. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM ’23, page 42–50, New York, NY, USA, 2023. Association for Computing Machinery.
  43. Uncovering the structural fairness in graph contrastive learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  44. Meta graph learning for long-tail recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’23, page 2512–2522, New York, NY, USA, 2023. Association for Computing Machinery.
  45. Simplifying graph convolutional networks, 2019.
  46. Self-supervised graph learning for recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, page 726–735, New York, NY, USA, 2021. Association for Computing Machinery.
  47. Demo-net: Degree-specific graph neural networks for node and graph classification. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, page 406–415, New York, NY, USA, 2019. Association for Computing Machinery.
  48. Learning how to propagate messages in graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD ’21, page 1894–1903, New York, NY, USA, 2021. Association for Computing Machinery.
  49. Grace: Graph self-distillation and completion to mitigate degree-related biases. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’23, page 2813–2824, New York, NY, USA, 2023. Association for Computing Machinery.
  50. Hyncf: A hybrid normalization strategy via feature statistics for collaborative filtering. Expert Systems with Applications, 238:121875, 2024.
  51. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
  52. Representation learning on graphs with jumping knowledge networks. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5453–5462. PMLR, 10–15 Jul 2018.
  53. Lte4g: Long-tail experts for graph neural networks. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM ’22, page 2434–2443, New York, NY, USA, 2022. Association for Computing Machinery.
  54. Incorporating bias-aware margins into contrastive loss for collaborative filtering. Advances in Neural Information Processing Systems, 35:7866–7878, 2022.
  55. Learning fair representations via rebalancing graph structure. Information Processing & Management, 61(1):103570, 2024.
  56. Rumour detection on social media with long-tail strategy. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2022.
  57. Bayesian graph local extrema convolution with long-tail strategy for misinformation detection. ACM Trans. Knowl. Discov. Data, jan 2024. Just Accepted.
  58. Bilateral filtering graph convolutional network for multi-relational social recommendation in the power-law networks. ACM Trans. Inf. Syst., 40(2), sep 2021.
  59. Dahgn: Degree-aware heterogeneous graph neural network. Knowledge-Based Systems, 285:111355, 2024.
  60. Self-supervised graph attention collaborative filtering for recommendation. Electronics, 12(4), 2023.
  61. Markus Zopf. 1-wl expressiveness is (almost) all you need. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Arjun Subramonian (22 papers)
  2. Jian Kang (142 papers)
  3. Yizhou Sun (149 papers)
Citations (1)
View on Semantic Scholar

Summary

Unveiling the Roots of Degree Bias in Graph Neural Networks

Introduction

Graph Neural Networks (GNNs) have emerged as the de facto standard for learning representations on graphs, with applications spanning across node classification, link prediction, and graph classification tasks. A peculiar phenomenon observed in these networks, however, is their better performance on high-degree nodes as opposed to their low-degree counterparts in node classification tasks. This bias not only skews the performance metrics but also has broader implications for fairness, potentially marginalizing already under-represented communities in social or citation networks. In the paper by Arjun Subramonian, Jian Kang, and Yizhou Sun, the authors explore the underlying reasons for degree bias in GNNs via a methodical investigation, combining theoretical analysis with empirical validation on real-world datasets.

Theoretical Insights into Degree Bias

The authors begin their exploration by establishing a theoretical framework to analyze degree bias in message-passing GNNs, particularly focusing on Graph Convolutional Networks (GCNs) with different normalization filters such as RW (Random Walk-normalized filter), SYM (Symmetric-normalized filter), and ATT (Attention-based filter). They prove that high-degree test nodes inherently have a lower probability of misclassification across these diverse architectures. Furthermore, they identify that factors such as homophily of neighbors and the diversity of neighbors' degrees contribute to the emergence of degree bias. Crucially, during training, it is shown that GNNs may adjust their loss on low-degree nodes more slowly compared to high-degree nodes. This finding implies that degree bias is not merely a by-product of model expressiveness but also a consequence of the training dynamics.

Empirical Investigations

The theoretical claims are substantiated through extensive experiments on eight real-world networks, covering a spectrum from citation to social and product co-purchasing networks. The empirical results align with the theoretical insights, showing a consistent pattern of degree bias across different GNN architectures and datasets. Specifically, it is observed that high-degree nodes generally incur a lower test loss than low-degree nodes. Additionally, the authors demonstrate that message-passing GNNs, irrespective of their normalization filters, can achieve their maximum possible training accuracy, suggesting that the expressive power of GNNs is not significantly restricted by degree bias.

Implications and Future Directions

The investigation into the origins of degree bias in GNNs by Subramonian et al. holds significant implications for the design of graph neural networks. By identifying the factors contributing to degree bias, the paper paves the way for developing more equitable and effective graph representation learning models. Specifically, it highlights the need for mechanisms that can enhance the representational quality of low-degree nodes, such as through neighborhood augmentation strategies or normalizing graph filters to minimize distributional discrepancies between node degrees. Moreover, addressing training discrepancies for nodes with varying degrees is critical for ensuring that no subset of nodes is systematically disadvantaged.

Looking ahead, the findings encourage a broader consideration of fairness in graph neural networks, urging future research to devise strategies that mitigate degree bias. Such efforts could involve designing novel architectural innovations, regularization techniques, or even rethinking the training algorithms to promote equitable learning outcomes across all nodes. Furthermore, as the paper primarily focuses on transductive learning settings, extending the analysis to inductive settings presents an interesting avenue for future research, potentially uncovering new dimensions of degree bias and opportunities for intervention.

Conclusion

The exploration into degree bias in GNNs by Subramonian and colleagues offers valuable insights into the mechanisms that give rise to this phenomenon. By marrying theoretical analysis with empirical validation, the paper enhances our understanding of why GNNs perform better on high-degree nodes and lays a foundation for future work aimed at developing fairer and more robust graph representation learning models. As the field of graph neural networks continues to evolve, addressing the challenges posed by degree bias will be paramount for ensuring that these powerful models can serve diverse and equitable applications across all domains.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets