Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Comprehensive Survey on Distributed Training of Graph Neural Networks (2211.05368v3)

Published 10 Nov 2022 in cs.DC and cs.LG

Abstract: Graph neural networks (GNNs) have been demonstrated to be a powerful algorithmic model in broad application fields for their effectiveness in learning over graphs. To scale GNN training up for large-scale and ever-growing graphs, the most promising solution is distributed training which distributes the workload of training across multiple computing nodes. At present, the volume of related research on distributed GNN training is exceptionally vast, accompanied by an extraordinarily rapid pace of publication. Moreover, the approaches reported in these studies exhibit significant divergence. This situation poses a considerable challenge for newcomers, hindering their ability to grasp a comprehensive understanding of the workflows, computational patterns, communication strategies, and optimization techniques employed in distributed GNN training. As a result, there is a pressing need for a survey to provide correct recognition, analysis, and comparisons in this field. In this paper, we provide a comprehensive survey of distributed GNN training by investigating various optimization techniques used in distributed GNN training. First, distributed GNN training is classified into several categories according to their workflows. In addition, their computational patterns and communication patterns, as well as the optimization techniques proposed by recent work are introduced. Second, the software frameworks and hardware platforms of distributed GNN training are also introduced for a deeper understanding. Third, distributed GNN training is compared with distributed training of deep neural networks, emphasizing the uniqueness of distributed GNN training. Finally, interesting issues and opportunities in this field are discussed.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (138)
  1. S. Wasserman, K. Faust et al., “Social network analysis: Methods and applications,” 1994.
  2. H. Kwak, C. Lee, H. Park, and S. Moon, “What is twitter, a social network or a news media?” in Proceedings of the 19th International Conference on World Wide Web, ser. WWW ’10.   New York, NY, USA: Association for Computing Machinery, 2010, p. 591–600. [Online]. Available: https://doi.org/10.1145/1772690.1772751
  3. J. J. McAuley and J. Leskovec, “Learning to discover social circles in ego networks,” in Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, P. L. Bartlett, F. C. N. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds., 2012, pp. 548–556. [Online]. Available: https://proceedings.neurips.cc/paper/2012/hash/7a614fd06c325499f1680b9896beedeb-Abstract.html
  4. K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor, “Freebase: a collaboratively created graph database for structuring human knowledge,” in Proceedings of the 2008 ACM SIGMOD international conference on Management of data, 2008, pp. 1247–1250.
  5. M. Ashburner, C. A. Ball, J. A. Blake, D. Botstein, H. Butler, J. M. Cherry, A. P. Davis, K. Dolinski, S. S. Dwight, J. T. Eppig et al., “Gene ontology: tool for the unification of biology,” Nature genetics, vol. 25, no. 1, pp. 25–29, 2000.
  6. G. A. Miller, “Wordnet: a lexical database for english,” Communications of the ACM, vol. 38, no. 11, pp. 39–41, 1995.
  7. Z. Zhang, P. Cui, and W. Zhu, “Deep learning on graphs: A survey,” IEEE Trans. Knowl. Data Eng., vol. 34, no. 1, pp. 249–270, 2022. [Online]. Available: https://doi.org/10.1109/TKDE.2020.2981333
  8. Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu, “A comprehensive survey on graph neural networks,” IEEE Trans. Neural Networks Learn. Syst., vol. 32, no. 1, pp. 4–24, 2021. [Online]. Available: https://doi.org/10.1109/TNNLS.2020.2978386
  9. J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun, “Graph neural networks: A review of methods and applications,” AI Open, vol. 1, pp. 57–81, 2020. [Online]. Available: https://doi.org/10.1016/j.aiopen.2021.01.001
  10. Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, pp. 436–444, 2015.
  11. W. L. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., 2017, pp. 1024–1034. [Online]. Available: https://proceedings.neurips.cc/paper/2017/hash/5dd9db5e033da9c6fb5ba83c7a7ebea9-Abstract.html
  12. M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst, “Geometric deep learning: going beyond euclidean data,” IEEE Signal Processing Magazine, vol. 34, no. 4, pp. 18–42, 2017.
  13. T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.   OpenReview.net, 2017. [Online]. Available: https://openreview.net/forum?id=SJU4ayYgl
  14. M. Zhang and Y. Chen, “Link prediction based on graph neural networks,” in Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., 2018, pp. 5171–5181. [Online]. Available: https://proceedings.neurips.cc/paper/2018/hash/53f0d7c537d99b3824f0f99d62ea2428-Abstract.html
  15. M. Zhang, Z. Cui, M. Neumann, and Y. Chen, “An end-to-end deep learning architecture for graph classification,” in Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, S. A. McIlraith and K. Q. Weinberger, Eds.   AAAI Press, 2018, pp. 4438–4445. [Online]. Available: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17146
  16. M. Schlichtkrull, T. N. Kipf, P. Bloem, R. v. d. Berg, I. Titov, and M. Welling, “Modeling relational data with graph convolutional networks,” in European semantic web conference.   Springer, 2018, pp. 593–607.
  17. T. Young, D. Hazarika, S. Poria, and E. Cambria, “Recent trends in deep learning based natural language processing [review article],” IEEE Comput. Intell. Mag., vol. 13, no. 3, pp. 55–75, 2018. [Online]. Available: https://doi.org/10.1109/MCI.2018.2840738
  18. M. Wu, S. Pan, X. Zhu, C. Zhou, and L. Pan, “Domain-adversarial graph neural networks for text classification,” in 2019 IEEE International Conference on Data Mining, ICDM 2019, Beijing, China, November 8-11, 2019, J. Wang, K. Shim, and X. Wu, Eds.   IEEE, 2019, pp. 648–657. [Online]. Available: https://doi.org/10.1109/ICDM.2019.00075
  19. J. Bastings, I. Titov, W. Aziz, D. Marcheggiani, and K. Sima’an, “Graph convolutional encoders for syntax-aware neural machine translation,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.   Copenhagen, Denmark: Association for Computational Linguistics, Sep. 2017, pp. 1957–1967. [Online]. Available: https://www.aclweb.org/anthology/D17-1209
  20. R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec, “Graph convolutional neural networks for web-scale recommender systems,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, Y. Guo and F. Farooq, Eds.   ACM, 2018, pp. 974–983. [Online]. Available: https://doi.org/10.1145/3219819.3219890
  21. W. Fan, Y. Ma, Q. Li, Y. He, Y. E. Zhao, J. Tang, and D. Yin, “Graph neural networks for social recommendation,” in The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, L. Liu, R. W. White, A. Mantrach, F. Silvestri, J. J. McAuley, R. Baeza-Yates, and L. Zia, Eds.   ACM, 2019, pp. 417–426. [Online]. Available: https://doi.org/10.1145/3308558.3313488
  22. C. Ma, L. Ma, Y. Zhang, J. Sun, X. Liu, and M. Coates, “Memory augmented graph neural networks for sequential recommendation,” in The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020.   AAAI Press, 2020, pp. 5045–5052. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/5945
  23. X. Chen, L.-J. Li, L. Fei-Fei, and A. Gupta, “Iterative visual reasoning beyond convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7239–7248.
  24. A. Mirhoseini, A. Goldie, M. Yazgan, J. W. Jiang, E. Songhori, S. Wang, Y.-J. Lee, E. Johnson, O. Pathak, A. Nazi et al., “A graph placement methodology for fast chip design,” Nature, vol. 594, no. 7862, pp. 207–212, 2021.
  25. H. Wang, K. Wang, J. Yang, L. Shen, N. Sun, H.-S. Lee, and S. Han, “Gcn-rl circuit designer: Transferable transistor sizing with graph neural networks and reinforcement learning,” in 2020 57th ACM/IEEE Design Automation Conference (DAC).   IEEE, 2020, pp. 1–6.
  26. Y. Ma, H. Ren, B. Khailany, H. Sikka, L. Luo, K. Natarajan, and B. Yu, “High performance graph convolutional networks with applications in testability analysis,” in Proceedings of the 56th Annual Design Automation Conference 2019, 2019, pp. 1–6.
  27. L. Oliver et al. Traffic prediction with advanced graph neural networks. [Online]. Available: https://deepmind.com/blog/article/traffic-prediction-with-advanced-graph-neural-networks
  28. C. Chen, K. Li, S. G. Teo, X. Zou, K. Wang, J. Wang, and Z. Zeng, “Gated residual recurrent graph neural networks for traffic prediction,” in The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019.   AAAI Press, 2019, pp. 485–492. [Online]. Available: https://doi.org/10.1609/aaai.v33i01.3301485
  29. M. Li and Z. Zhu, “Spatial-temporal fusion graph neural networks for traffic flow forecasting,” in Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021.   AAAI Press, 2021, pp. 4189–4196. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/16542
  30. X. Geng, Y. Li, L. Wang, L. Zhang, Q. Yang, J. Ye, and Y. Liu, “Spatiotemporal multi-graph convolution network for ride-hailing demand forecasting,” in Proceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 3656–3663.
  31. A. Li, Z. Qin, R. Liu, Y. Yang, and D. Li, “Spam review detection with graph convolutional networks,” in Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019, pp. 2703–2711.
  32. A. Fout, J. Byrd, B. Shariat, and A. Ben-Hur, “Protein interface prediction using graph convolutional networks,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., 2017, pp. 6530–6539. [Online]. Available: https://proceedings.neurips.cc/paper/2017/hash/f507783927f2ec2737ba40afbd17efb5-Abstract.html
  33. R. Zhu, K. Zhao, H. Yang, W. Lin, C. Zhou, B. Ai, Y. Li, and J. Zhou, “Aligraph: A comprehensive graph neural network platform,” Proc. VLDB Endow., vol. 12, no. 12, pp. 2094–2105, 2019. [Online]. Available: http://www.vldb.org/pvldb/vol12/p2094-zhu.pdf
  34. D. Zhang, X. Huang, Z. Liu, J. Zhou, Z. Hu, X. Song, Z. Ge, L. Wang, Z. Zhang, and Y. Qi, “AGL: A scalable system for industrial-purpose graph machine learning,” Proc. VLDB Endow., vol. 13, no. 12, pp. 3125–3137, 2020. [Online]. Available: http://www.vldb.org/pvldb/vol13/p3125-zhang.pdf
  35. L. Wang, Q. Yin, C. Tian, J. Yang, R. Chen, W. Yu, Z. Yao, and J. Zhou, “Flexgraph: a flexible and efficient distributed framework for GNN training,” in EuroSys ’21: Sixteenth European Conference on Computer Systems, Online Event, United Kingdom, April 26-28, 2021, A. Barbalace, P. Bhatotia, L. Alvisi, and C. Cadar, Eds.   ACM, 2021, pp. 67–82. [Online]. Available: https://doi.org/10.1145/3447786.3456229
  36. D. Zheng, C. Ma, M. Wang, J. Zhou, Q. Su, X. Song, Q. Gan, Z. Zhang, and G. Karypis, “Distdgl: Distributed graph neural network training for billion-scale graphs,” in 10th IEEE/ACM Workshop on Irregular Applications: Architectures and Algorithms, IA3 2020, Atlanta, GA, USA, November 11, 2020.   IEEE, 2020, pp. 36–44. [Online]. Available: https://doi.org/10.1109/IA351965.2020.00011
  37. W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec, “Open graph benchmark: Datasets for machine learning on graphs,” in Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020. [Online]. Available: https://proceedings.neurips.cc/paper/2020/hash/fb60d411a5c5b72b2e7d3527cfc84fd0-Abstract.html
  38. A. Tripathy, K. A. Yelick, and A. Buluç, “Reducing communication in graph neural network training,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, C. Cuicchi, I. Qualters, and W. T. Kramer, Eds.   IEEE/ACM, 2020, p. 70. [Online]. Available: https://doi.org/10.1109/SC41405.2020.00074
  39. J. Huang, W. Qin, X. Wang, and W. Chen, “Survey of external memory large-scale graph processing on a multi-core system,” J. Supercomput., vol. 76, no. 1, pp. 549–579, 2020. [Online]. Available: https://doi.org/10.1007/s11227-019-03023-0
  40. J. Huang, H. Wang, X. Fei, X. Wang, and W. Chen, “$tc-stream$tc-stream: Large-scale graph triangle counting on a single machine using gpus,” IEEE Trans. Parallel Distributed Syst., vol. 33, no. 11, pp. 3067–3078, 2022. [Online]. Available: https://doi.org/10.1109/TPDS.2021.3135329
  41. “Global social media ranking— statistic.” [Online]. Available: https://www.statista.com/statistics
  42. L. Ma, Z. Yang, Y. Miao, J. Xue, M. Wu, L. Zhou, and Y. Dai, “Neugraph: Parallel deep neural network computation on large graphs,” in 2019 USENIX Annual Technical Conference, USENIX ATC 2019, Renton, WA, USA, July 10-12, 2019, D. Malkhi and D. Tsafrir, Eds.   USENIX Association, 2019, pp. 443–458. [Online]. Available: https://www.usenix.org/conference/atc19/presentation/ma
  43. Z. Jia, S. Lin, M. Gao, M. Zaharia, and A. Aiken, “Improving the accuracy, scalability, and performance of graph neural networks with roc,” in Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020, I. S. Dhillon, D. S. Papailiopoulos, and V. Sze, Eds.   mlsys.org, 2020. [Online]. Available: https://proceedings.mlsys.org/book/300.pdf
  44. Z. Cai, X. Yan, Y. Wu, K. Ma, J. Cheng, and F. Yu, “DGCL: an efficient communication library for distributed GNN training,” in EuroSys ’21: Sixteenth European Conference on Computer Systems, Online Event, United Kingdom, April 26-28, 2021, A. Barbalace, P. Bhatotia, L. Alvisi, and C. Cadar, Eds.   ACM, 2021, pp. 130–144. [Online]. Available: https://doi.org/10.1145/3447786.3456233
  45. V. Md, S. Misra, G. Ma, R. Mohanty, E. Georganas, A. Heinecke, D. D. Kalamkar, N. K. Ahmed, and S. Avancha, “Distgnn: scalable distributed training for large-scale graph neural networks,” in SC ’21: The International Conference for High Performance Computing, Networking, Storage and Analysis, St. Louis, Missouri, USA, November 14 - 19, 2021, B. R. de Supinski, M. W. Hall, and T. Gamblin, Eds.   ACM, 2021, pp. 76:1–76:14. [Online]. Available: https://doi.org/10.1145/3458817.3480856
  46. M. F. Balin, K. Sancak, and Ü. V. Çatalyürek, “MG-GCN: scalable multi-gpu GCN training framework,” CoRR, vol. abs/2110.08688, 2021. [Online]. Available: https://arxiv.org/abs/2110.08688
  47. J. Thorpe, Y. Qiao, J. Eyolfson, S. Teng, G. Hu, Z. Jia, J. Wei, K. Vora, R. Netravali, M. Kim, and G. H. Xu, “Dorylus: Affordable, scalable, and accurate GNN training with distributed CPU servers and serverless threads,” in 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021, July 14-16, 2021, A. D. Brown and J. R. Lorch, Eds.   USENIX Association, 2021, pp. 495–514. [Online]. Available: https://www.usenix.org/conference/osdi21/presentation/thorpe
  48. H. Mostafa, “Sequential aggregation and rematerialization: Distributed full-batch training of graph neural networks on large graphs,” CoRR, vol. abs/2111.06483, 2021. [Online]. Available: https://arxiv.org/abs/2111.06483
  49. Z. Lin, C. Li, Y. Miao, Y. Liu, and Y. Xu, “Pagraph: Scaling GNN training on large graphs via computation-aware caching,” in SoCC ’20: ACM Symposium on Cloud Computing, Virtual Event, USA, October 19-21, 2020, R. Fonseca, C. Delimitrou, and B. C. Ooi, Eds.   ACM, 2020, pp. 401–415. [Online]. Available: https://doi.org/10.1145/3419111.3421281
  50. L. Zhang, Z. Lai, S. Li, Y. Tang, F. Liu, and D. Li, “2pgraph: Accelerating GNN training over large graphs on GPU clusters,” in IEEE International Conference on Cluster Computing, CLUSTER 2021, Portland, OR, USA, September 7-10, 2021.   IEEE, 2021, pp. 103–113. [Online]. Available: https://doi.org/10.1109/Cluster48925.2021.00036
  51. M. Ramezani, W. Cong, M. Mahdavi, M. T. Kandemir, and A. Sivasubramaniam, “Learn locally, correct globally: A distributed algorithm for training graph neural networks,” CoRR, vol. abs/2111.08202, 2021. [Online]. Available: https://arxiv.org/abs/2111.08202
  52. D. Zheng, X. Song, C. Yang, D. LaSalle, and G. Karypis, “Distributed hybrid CPU and GPU training for graph neural networks on billion-scale heterogeneous graphs,” in KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022, A. Zhang and H. Rangwala, Eds.   ACM, 2022, pp. 4582–4591. [Online]. Available: https://doi.org/10.1145/3534678.3539177
  53. H. Li, Y. Liu, Y. Li, B. Huang, P. Zhang, G. Zhang, X. Zeng, K. Deng, W. Chen, and C. He, “Graphtheta: A distributed graph neural network learning system with flexible training strategy,” CoRR, vol. abs/2104.10569, 2021. [Online]. Available: https://arxiv.org/abs/2104.10569
  54. T. Kaler, N. Stathas, A. Ouyang, A. Iliopoulos, T. B. Schardl, C. E. Leiserson, and J. Chen, “Accelerating training and inference of graph neural networks with fast sampling and pipelining,” CoRR, vol. abs/2110.08450, 2021. [Online]. Available: https://arxiv.org/abs/2110.08450
  55. S. Gandhi and A. P. Iyer, “P3: distributed deep graph learning at scale,” in 15th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2021, July 14-16, 2021, A. D. Brown and J. R. Lorch, Eds.   USENIX Association, 2021, pp. 551–568. [Online]. Available: https://www.usenix.org/conference/osdi21/presentation/gandhi
  56. Z. Luo, Y. Bao, and C. Wu, “Optimizing task placement and online scheduling for distributed GNN training acceleration,” in IEEE INFOCOM 2022 - IEEE Conference on Computer Communications, London, United Kingdom, May 2-5, 2022.   IEEE, 2022, pp. 890–899. [Online]. Available: https://doi.org/10.1109/INFOCOM48880.2022.9796910
  57. S. Abadal, A. Jain, R. Guirado, J. López-Alonso, and E. Alarcón, “Computing graph neural networks: A survey from algorithms to accelerators,” ACM Comput. Surv., vol. 54, no. 9, pp. 191:1–191:38, 2022. [Online]. Available: https://doi.org/10.1145/3477141
  58. P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. F. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, Ç. Gülçehre, H. F. Song, A. J. Ballard, J. Gilmer, G. E. Dahl, A. Vaswani, K. R. Allen, C. Nash, V. Langston, C. Dyer, N. Heess, D. Wierstra, P. Kohli, M. Botvinick, O. Vinyals, Y. Li, and R. Pascanu, “Relational inductive biases, deep learning, and graph networks,” CoRR, vol. abs/1806.01261, 2018. [Online]. Available: http://arxiv.org/abs/1806.01261
  59. I. Chami, S. Abu-El-Haija, B. Perozzi, C. Ré, and K. Murphy, “Machine learning on graphs: A model and comprehensive taxonomy,” CoRR, vol. abs/2005.03675, 2020. [Online]. Available: https://arxiv.org/abs/2005.03675
  60. L. C. Lamb, A. S. d’Avila Garcez, M. Gori, M. O. R. Prates, P. H. C. Avelar, and M. Y. Vardi, “Graph neural networks meet neural-symbolic computing: A survey and perspective,” in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, C. Bessiere, Ed.   ijcai.org, 2020, pp. 4877–4884. [Online]. Available: https://doi.org/10.24963/ijcai.2020/679
  61. P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.   OpenReview.net, 2018. [Online]. Available: https://openreview.net/forum?id=rJXMpikCZ
  62. K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How powerful are graph neural networks?” arXiv preprint arXiv:1810.00826, 2018.
  63. S. Yan, Y. Xiong, and D. Lin, “Spatial temporal graph convolutional networks for skeleton-based action recognition,” in Thirty-second AAAI conference on artificial intelligence, 2018.
  64. Z. Wu, S. Pan, G. Long, J. Jiang, X. Chang, and C. Zhang, “Connecting the dots: Multivariate time series forecasting with graph neural networks,” in KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, R. Gupta, Y. Liu, J. Tang, and B. A. Prakash, Eds.   ACM, 2020, pp. 753–763. [Online]. Available: https://doi.org/10.1145/3394486.3403118
  65. M. Wu, S. Pan, L. Du, and X. Zhu, “Learning graph neural networks with positive and unlabeled nodes,” ACM Trans. Knowl. Discov. Data, vol. 15, no. 6, pp. 101:1–101:25, 2021. [Online]. Available: https://doi.org/10.1145/3450316
  66. S. Zhu, C. Zhou, S. Pan, X. Zhu, and B. Wang, “Relation structure-aware heterogeneous graph neural network,” in 2019 IEEE International Conference on Data Mining, ICDM 2019, Beijing, China, November 8-11, 2019, J. Wang, K. Shim, and X. Wu, Eds.   IEEE, 2019, pp. 1534–1539. [Online]. Available: https://doi.org/10.1109/ICDM.2019.00203
  67. Y. Liu, X. Qiao, L. Wang, and J. Lam, “EEGNN: edge enhanced graph neural network with a bayesian nonparametric graph model,” in International Conference on Artificial Intelligence and Statistics, 25-27 April 2023, Palau de Congressos, Valencia, Spain, ser. Proceedings of Machine Learning Research, F. J. R. Ruiz, J. G. Dy, and J. van de Meent, Eds., vol. 206.   PMLR, 2023, pp. 2132–2146. [Online]. Available: https://proceedings.mlr.press/v206/liu23a.html
  68. U. Alon and E. Yahav, “On the bottleneck of graph neural networks and its practical implications,” in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.   OpenReview.net, 2021. [Online]. Available: https://openreview.net/forum?id=i80OPhOCVH2
  69. Y. Rong, W. Huang, T. Xu, and J. Huang, “Dropedge: Towards deep graph convolutional networks on node classification,” in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.   OpenReview.net, 2020. [Online]. Available: https://openreview.net/forum?id=Hkx1qkrKPr
  70. W. Huang, Y. Rong, T. Xu, F. Sun, and J. Huang, “Tackling over-smoothing for general graph convolutional networks,” CoRR, vol. abs/2008.09864, 2020. [Online]. Available: https://arxiv.org/abs/2008.09864
  71. J. Chen, T. Ma, and C. Xiao, “Fastgcn: Fast learning with graph convolutional networks via importance sampling,” in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.   OpenReview.net, 2018. [Online]. Available: https://openreview.net/forum?id=rytstxWAW
  72. W. Huang, T. Zhang, Y. Rong, and J. Huang, “Adaptive sampling towards fast graph representation learning,” in Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., 2018, pp. 4563–4572. [Online]. Available: https://proceedings.neurips.cc/paper/2018/hash/01eee509ee2f68dc6014898c309e86bf-Abstract.html
  73. H. Zeng, H. Zhou, A. Srivastava, R. Kannan, and V. K. Prasanna, “Graphsaint: Graph sampling based inductive learning method,” in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.   OpenReview.net, 2020. [Online]. Available: https://openreview.net/forum?id=BJe8pkHFwS
  74. J. Dong, D. Zheng, L. F. Yang, and G. Karypis, “Global neighbor sampling for mixed CPU-GPU training on giant graphs,” in KDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021, F. Zhu, B. C. Ooi, and C. Miao, Eds.   ACM, 2021, pp. 289–299. [Online]. Available: https://doi.org/10.1145/3447548.3467437
  75. L. Bottou, “Large-scale machine learning with stochastic gradient descent,” in 19th International Conference on Computational Statistics, COMPSTAT 2010, Paris, France, August 22-27, 2010 - Keynote, Invited and Contributed Papers, Y. Lechevallier and G. Saporta, Eds.   Physica-Verlag, 2010, pp. 177–186. [Online]. Available: https://doi.org/10.1007/978-3-7908-2604-3\_16
  76. X. Liu, M. Yan, L. Deng, G. Li, X. Ye, and D. Fan, “Sampling methods for efficient training of graph convolutional networks: A survey,” IEEE CAA J. Autom. Sinica, vol. 9, no. 2, pp. 205–234, 2022. [Online]. Available: https://doi.org/10.1109/JAS.2021.1004311
  77. H. Dai, Z. Kozareva, B. Dai, A. J. Smola, and L. Song, “Learning steady-states of iterative algorithms over graphs,” in Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, ser. Proceedings of Machine Learning Research, J. G. Dy and A. Krause, Eds., vol. 80.   PMLR, 2018, pp. 1114–1122. [Online]. Available: http://proceedings.mlr.press/v80/dai18a.html
  78. J. Chen, J. Zhu, and L. Song, “Stochastic training of graph convolutional networks with variance reduction,” in Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, ser. Proceedings of Machine Learning Research, J. G. Dy and A. Krause, Eds., vol. 80.   PMLR, 2018, pp. 941–949. [Online]. Available: http://proceedings.mlr.press/v80/chen18p.html
  79. D. Zou, Z. Hu, Y. Wang, S. Jiang, Y. Sun, and Q. Gu, “Layer-dependent importance sampling for training deep and large graph convolutional networks,” in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, Eds., 2019, pp. 11 247–11 256. [Online]. Available: https://proceedings.neurips.cc/paper/2019/hash/91ba4a4478a66bee9812b0804b6f9d1b-Abstract.html
  80. W. Chiang, X. Liu, S. Si, Y. Li, S. Bengio, and C. Hsieh, “Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, A. Teredesai, V. Kumar, Y. Li, R. Rosales, E. Terzi, and G. Karypis, Eds.   ACM, 2019, pp. 257–266. [Online]. Available: https://doi.org/10.1145/3292500.3330925
  81. J. Bai, Y. Ren, and J. Zhang, “Ripple walk training: A subgraph-based training framework for large and deep graph neural network,” in International Joint Conference on Neural Networks, IJCNN 2021, Shenzhen, China, July 18-22, 2021.   IEEE, 2021, pp. 1–8. [Online]. Available: https://doi.org/10.1109/IJCNN52387.2021.9533429
  82. H. Zeng, H. Zhou, A. Srivastava, R. Kannan, and V. K. Prasanna, “Accurate, efficient and scalable graph embedding,” in 2019 IEEE International Parallel and Distributed Processing Symposium, IPDPS 2019, Rio de Janeiro, Brazil, May 20-24, 2019.   IEEE, 2019, pp. 462–471. [Online]. Available: https://doi.org/10.1109/IPDPS.2019.00056
  83. H. Lin, M. Yan, X. Yang, M. Zou, W. Li, X. Ye, and D. Fan, “Characterizing and understanding distributed GNN training on gpus,” IEEE Comput. Archit. Lett., vol. 21, no. 1, pp. 21–24, 2022. [Online]. Available: https://doi.org/10.1109/LCA.2022.3168067
  84. M. Fey and J. E. Lenssen, “Fast graph representation learning with pytorch geometric,” CoRR, vol. abs/1903.02428, 2019. [Online]. Available: http://arxiv.org/abs/1903.02428
  85. M. Wang, L. Yu, D. Zheng, Q. Gan, Y. Gai, Z. Ye, M. Li, J. Zhou, Q. Huang, C. Ma, Z. Huang, Q. Guo, H. Zhang, H. Lin, J. Zhao, J. Li, A. J. Smola, and Z. Zhang, “Deep graph library: Towards efficient and scalable deep learning on graphs,” CoRR, vol. abs/1909.01315, 2019. [Online]. Available: http://arxiv.org/abs/1909.01315
  86. M. Yan, L. Deng, X. Hu, L. Liang, Y. Feng, X. Ye, Z. Zhang, D. Fan, and Y. Xie, “Hygcn: A GCN accelerator with hybrid architecture,” in IEEE International Symposium on High Performance Computer Architecture, HPCA 2020, San Diego, CA, USA, February 22-26, 2020.   IEEE, 2020, pp. 15–29. [Online]. Available: https://doi.org/10.1109/HPCA47549.2020.00012
  87. T. J. Ham, L. Wu, N. Sundaram, N. Satish, and M. Martonosi, “Graphicionado: A high-performance and energy-efficient accelerator for graph analytics,” in 49th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2016, Taipei, Taiwan, October 15-19, 2016.   IEEE Computer Society, 2016, pp. 56:1–56:13. [Online]. Available: https://doi.org/10.1109/MICRO.2016.7783759
  88. M. Yan, X. Hu, S. Li, A. Basak, H. Li, X. Ma, I. Akgun, Y. Feng, P. Gu, L. Deng, X. Ye, Z. Zhang, D. Fan, and Y. Xie, “Alleviating irregularity in graph analytics acceleration: a hardware/software co-design approach,” in Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2019, Columbus, OH, USA, October 12-16, 2019.   ACM, 2019, pp. 615–628. [Online]. Available: https://doi.org/10.1145/3352460.3358318
  89. Z. Jia, Y. Kwon, G. M. Shipman, P. S. McCormick, M. Erez, and A. Aiken, “A distributed multi-gpu system for fast graph processing,” Proc. VLDB Endow., vol. 11, no. 3, pp. 297–310, 2017. [Online]. Available: http://www.vldb.org/pvldb/vol11/p297-jia.pdf
  90. “Cuda c++ programming guide,” https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#hardware-implementation, NVIDIA.
  91. Q. Fu, Y. Ji, and H. H. Huang, “TLPGNN: A lightweight two-level parallelism paradigm for graph neural network computation on GPU,” in HPDC ’22: The 31st International Symposium on High-Performance Parallel and Distributed Computing, Minneapolis, MN, USA, 27 June 2022 - 1 July 2022, J. B. Weissman, A. Chandra, A. Gavrilovska, and D. Tiwari, Eds.   ACM, 2022, pp. 122–134. [Online]. Available: https://doi.org/10.1145/3502181.3531467
  92. A. H. Nodehi Sabet, J. Qiu, and Z. Zhao, “Tigr: Transforming irregular graphs for gpu-friendly graph processing,” ACM SIGPLAN Notices, vol. 53, no. 2, pp. 622–636, 2018.
  93. G. Karypis and V. Kumar, “Metis – unstructured graph partitioning and sparse matrix ordering system, version 2.0,” Applied Physics Letters, Volume 97, Issue 12, id. 124101 (3 pages)(2010), 1995.
  94. X. Zhu, W. Chen, W. Zheng, and X. Ma, “Gemini: A computation-centric distributed graph processing system,” in 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, November 2-4, 2016, K. Keeton and T. Roscoe, Eds.   USENIX Association, 2016, pp. 301–316. [Online]. Available: https://www.usenix.org/conference/osdi16/technical-sessions/presentation/zhu
  95. H. Cui, J. Cipar, Q. Ho, J. K. Kim, S. Lee, A. Kumar, J. Wei, W. Dai, G. R. Ganger, P. B. Gibbons, G. A. Gibson, and E. P. Xing, “Exploiting bounded staleness to speed up big data analytics,” in 2014 USENIX Annual Technical Conference, USENIX ATC ’14, Philadelphia, PA, USA, June 19-20, 2014, G. Gibson and N. Zeldovich, Eds.   USENIX Association, 2014, pp. 37–48. [Online]. Available: https://www.usenix.org/conference/atc14/technical-sessions/presentation/cui
  96. B. Recht, C. Ré, S. J. Wright, and F. Niu, “Hogwild: A lock-free approach to parallelizing stochastic gradient descent,” in Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain, J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. C. N. Pereira, and K. Q. Weinberger, Eds., 2011, pp. 693–701. [Online]. Available: https://proceedings.neurips.cc/paper/2011/hash/218a0aefd1d1a4be65601cc6ddc1520e-Abstract.html
  97. X. Zhao, A. An, J. Liu, and B. X. Chen, “Dynamic stale synchronous parallel distributed training for deep learning,” in 39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019, Dallas, TX, USA, July 7-10, 2019.   IEEE, 2019, pp. 1507–1517. [Online]. Available: https://doi.org/10.1109/ICDCS.2019.00150
  98. Q. Ho, J. Cipar, H. Cui, S. Lee, J. K. Kim, P. B. Gibbons, G. A. Gibson, G. R. Ganger, and E. P. Xing, “More effective distributed ML via a stale synchronous parallel parameter server,” in Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, C. J. C. Burges, L. Bottou, Z. Ghahramani, and K. Q. Weinberger, Eds., 2013, pp. 1223–1231. [Online]. Available: https://proceedings.neurips.cc/paper/2013/hash/b7bb35b9c6ca2aee2df08cf09d7016c2-Abstract.html
  99. T. Chen, B. Xu, C. Zhang, and C. Guestrin, “Training deep nets with sublinear memory cost,” CoRR, vol. abs/1604.06174, 2016. [Online]. Available: http://arxiv.org/abs/1604.06174
  100. P. Jain, A. Jain, A. Nrusimha, A. Gholami, P. Abbeel, K. Keutzer, I. Stoica, and J. Gonzalez, “Checkmate: Breaking the memory wall with optimal tensor rematerialization,” in Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020, I. S. Dhillon, D. S. Papailiopoulos, and V. Sze, Eds.   mlsys.org, 2020. [Online]. Available: https://proceedings.mlsys.org/book/320.pdf
  101. J. Dean and S. Ghemawat, “Mapreduce: simplified data processing on large clusters,” Commun. ACM, vol. 51, no. 1, pp. 107–113, 2008. [Online]. Available: http://doi.acm.org/10.1145/1327452.1327492
  102. G. Karypis and V. Kumar, “A fast and high quality multilevel scheme for partitioning irregular graphs,” SIAM J. Sci. Comput., vol. 20, no. 1, pp. 359–392, 1998. [Online]. Available: https://doi.org/10.1137/S1064827595287997
  103. I. S. Dhillon, Y. Guan, and B. Kulis, “Weighted graph cuts without eigenvectors A multilevel approach,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 11, pp. 1944–1957, 2007. [Online]. Available: https://doi.org/10.1109/TPAMI.2007.1115
  104. J. E. Gonzalez, Y. Low, H. Gu, D. Bickson, and C. Guestrin, “Powergraph: Distributed graph-parallel computation on natural graphs,” in 10th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2012, Hollywood, CA, USA, October 8-10, 2012, C. Thekkath and A. Vahdat, Eds.   USENIX Association, 2012, pp. 17–30. [Online]. Available: https://www.usenix.org/conference/osdi12/technical-sessions/presentation/gonzalez
  105. E. G. Boman, K. D. Devine, and S. Rajamanickam, “Scalable matrix computations on large scale-free graphs using 2d graph partitioning,” in International Conference for High Performance Computing, Networking, Storage and Analysis, SC’13, Denver, CO, USA - November 17 - 21, 2013, W. Gropp and S. Matsuoka, Eds.   ACM, 2013, pp. 50:1–50:12. [Online]. Available: https://doi.org/10.1145/2503210.2503293
  106. I. Stanton and G. Kliot, “Streaming graph partitioning for large distributed graphs,” in The 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’12, Beijing, China, August 12-16, 2012, Q. Yang, D. Agarwal, and J. Pei, Eds.   ACM, 2012, pp. 1222–1230. [Online]. Available: https://doi.org/10.1145/2339530.2339722
  107. J. Leskovec, K. J. Lang, A. Dasgupta, and M. W. Mahoney, “Community structure in large networks: Natural cluster sizes and the absence of large well-defined clusters,” Internet Math., vol. 6, no. 1, pp. 29–123, 2009. [Online]. Available: https://doi.org/10.1080/15427951.2009.10129177
  108. H. Liu, S. Lu, X. Chen, and B. He, “G3: when graph neural networks meet parallel graph processing systems on gpus,” Proc. VLDB Endow., vol. 13, no. 12, pp. 2813–2816, 2020. [Online]. Available: http://www.vldb.org/pvldb/vol13/p2813-liu.pdf
  109. M. Garland and D. B. Kirk, “Understanding throughput-oriented architectures,” Communications of the ACM, vol. 53, no. 11, pp. 58–66, 2010.
  110. E. Lindholm, J. Nickolls, S. Oberman, and J. Montrym, “Nvidia tesla: A unified graphics and computing architecture,” IEEE micro, vol. 28, no. 2, pp. 39–55, 2008.
  111. J. D. Owens, M. Houston, D. Luebke, S. Green, J. E. Stone, and J. C. Phillips, “Gpu computing,” Proceedings of the IEEE, vol. 96, no. 5, pp. 879–899, 2008.
  112. K. S. Chahal, M. S. Grover, K. Dey, and R. R. Shah, “A hitchhiker’s guide on distributed training of deep neural networks,” J. Parallel Distributed Comput., vol. 137, pp. 65–76, 2020. [Online]. Available: https://doi.org/10.1016/j.jpdc.2019.10.004
  113. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, P. L. Bartlett, F. C. N. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds., 2012, pp. 1106–1114. [Online]. Available: https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html
  114. G. Sun, M. Yan, D. Wang, H. Li, W. Li, X. Ye, D. Fan, and Y. Xie, “Multi-node acceleration for large-scale gcns,” IEEE transactions on computers, 2022.
  115. M. Yan, Z. Chen, L. Deng, X. Ye, Z. Zhang, D. Fan, and Y. Xie, “Characterizing and understanding gcns on gpu,” IEEE Computer Architecture Letters, vol. 19, no. 1, pp. 22–25, 2020.
  116. Z. Zhang, J. Leng, L. Ma, Y. Miao, C. Li, and M. Guo, “Architectural implications of graph neural networks,” IEEE Computer Architecture Letters, vol. 19, no. 1, pp. 59–62, 2020.
  117. M. Yan, M. Zou, X. Yang, W. Li, X. Ye, D. Fan, and Y. Xie, “Characterizing and understanding hgnns on gpus,” IEEE Computer Architecture Letters, pp. 1–4, 2022.
  118. H. Lin, M. Yan, X. Yang, M. Zou, W. Li, X. Ye, and D. Fan, “Characterizing and understanding distributed gnn training on gpus,” IEEE Computer Architecture Letters, vol. 21, no. 1, pp. 21–24, 2022.
  119. K. Huang, J. Zhai, Z. Zheng, Y. Yi, and X. Shen, “Understanding and bridging the gaps in current GNN performance optimizations,” in PPoPP ’21: 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, Virtual Event, Republic of Korea, February 27- March 3, 2021, J. Lee and E. Petrank, Eds.   ACM, 2021, pp. 119–132. [Online]. Available: https://doi.org/10.1145/3437801.3441585
  120. H. Zhang, Z. Yu, G. Dai, G. Huang, Y. Ding, Y. Xie, and Y. Wang, “Understanding GNN computational graph: A coordinated computation, io, and memory perspective,” CoRR, vol. abs/2110.09524, 2021. [Online]. Available: https://arxiv.org/abs/2110.09524
  121. P. Mattson, V. J. Reddi, C. Cheng, C. Coleman, G. Diamos, D. Kanter, P. Micikevicius, D. Patterson, G. Schmuelling, H. Tang, G.-Y. Wei, and C.-J. Wu, “Mlperf: An industry standard benchmark suite for machine learning performance,” IEEE Micro, vol. 40, no. 2, pp. 8–16, 2020.
  122. H. Lin, X. Zhu, B. Yu, X. Tang, W. Xue, W. Chen, L. Zhang, T. Hoefler, X. Ma, X. Liu, W. Zheng, and J. Xu, “Shentu: processing multi-trillion edge graphs on millions of cores in seconds,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018, Dallas, TX, USA, November 11-16, 2018.   IEEE / ACM, 2018, pp. 56:1–56:11. [Online]. Available: http://dl.acm.org/citation.cfm?id=3291731
  123. J. Leskovec, D. Chakrabarti, J. M. Kleinberg, and C. Faloutsos, “Realistic, mathematically tractable graph generation and evolution, using kronecker multiplication,” in Knowledge Discovery in Databases: PKDD 2005, 9th European Conference on Principles and Practice of Knowledge Discovery in Databases, Porto, Portugal, October 3-7, 2005, Proceedings, ser. Lecture Notes in Computer Science, A. Jorge, L. Torgo, P. Brazdil, R. Camacho, and J. Gama, Eds., vol. 3721.   Springer, 2005, pp. 133–145. [Online]. Available: https://doi.org/10.1007/11564126\_17
  124. N. P. Jouppi, D. H. Yoon, G. Kurian, S. Li, N. Patil, J. Laudon, C. Young, and D. Patterson, “A domain-specific supercomputer for training deep neural networks,” Communications of the ACM, vol. 63, no. 7, pp. 67–78, 2020.
  125. C. Wang, D. Sun, and Y. Bai, “Pipad: Pipelined and parallel dynamic GNN training on gpus,” in Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, PPoPP 2023, Montreal, QC, Canada, 25 February 2023 - 1 March 2023, M. M. Dehnavi, M. Kulkarni, and S. Krishnamoorthy, Eds.   ACM, 2023, pp. 405–418. [Online]. Available: https://doi.org/10.1145/3572848.3577487
  126. Y. Xia, Z. Zhang, H. Wang, D. Yang, X. Zhou, and D. Cheng, “Redundancy-free high-performance dynamic GNN training with hierarchical pipeline parallelism,” in Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2023, Orlando, FL, USA, June 16-23, 2023, A. R. Butt, N. Mi, and K. Chard, Eds.   ACM, 2023, pp. 17–30. [Online]. Available: https://doi.org/10.1145/3588195.3592990
  127. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997. [Online]. Available: https://doi.org/10.1162/neco.1997.9.8.1735
  128. V. T. Chakaravarthy, S. S. Pandian, S. Raje, Y. Sabharwal, T. Suzumura, and S. Ubaru, “Efficient scaling of dynamic graph neural networks,” in International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2021, St. Louis, Missouri, USA, November 14-19, 2021, B. R. de Supinski, M. W. Hall, and T. Gamblin, Eds.   ACM, 2021, p. 77. [Online]. Available: https://doi.org/10.1145/3458817.3480858
  129. B. Rozemberczki, P. Scherer, Y. He, G. Panagopoulos, A. Riedel, M. S. Astefanoaei, O. Kiss, F. Béres, G. López, N. Collignon, and R. Sarkar, “Pytorch geometric temporal: Spatiotemporal signal processing with neural machine learning models,” in CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021, G. Demartini, G. Zuccon, J. S. Culpepper, Z. Huang, and H. Tong, Eds.   ACM, 2021, pp. 4564–4573. [Online]. Available: https://doi.org/10.1145/3459637.3482014
  130. H. Zhou, D. Zheng, I. Nisa, V. N. Ioannidis, X. Song, and G. Karypis, “TGL: A general framework for temporal GNN training onbillion-scale graphs,” Proc. VLDB Endow., vol. 15, no. 8, pp. 1572–1580, 2022. [Online]. Available: https://www.vldb.org/pvldb/vol15/p1572-zhou.pdf
  131. J. Klicpera, A. Bojchevski, and S. Günnemann, “Predict then propagate: Graph neural networks meet personalized pagerank,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.   OpenReview.net, 2019. [Online]. Available: https://openreview.net/forum?id=H1gL-2A9Ym
  132. G. Li, C. Xiong, A. K. Thabet, and B. Ghanem, “Deepergcn: All you need to train deeper gcns,” CoRR, vol. abs/2006.07739, 2020. [Online]. Available: https://arxiv.org/abs/2006.07739
  133. M. Chen, Z. Wei, Z. Huang, B. Ding, and Y. Li, “Simple and deep graph convolutional networks,” in Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, ser. Proceedings of Machine Learning Research, vol. 119.   PMLR, 2020, pp. 1725–1735. [Online]. Available: http://proceedings.mlr.press/v119/chen20v.html
  134. A. Singh, Q. Huang, S. L. Huang, O. Bhalerao, H. He, S. Lim, and A. R. Benson, “Edge proposal sets for link prediction,” CoRR, vol. abs/2106.15810, 2021. [Online]. Available: https://arxiv.org/abs/2106.15810
  135. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., 2017, pp. 5998–6008. [Online]. Available: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
  136. J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, “Neural message passing for quantum chemistry,” in Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, ser. Proceedings of Machine Learning Research, D. Precup and Y. W. Teh, Eds., vol. 70.   PMLR, 2017, pp. 1263–1272. [Online]. Available: http://proceedings.mlr.press/v70/gilmer17a.html
  137. F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, “The graph neural network model,” IEEE Trans. Neural Networks, vol. 20, no. 1, pp. 61–80, 2009. [Online]. Available: https://doi.org/10.1109/TNN.2008.2005605
  138. Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. X. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, and Z. Chen, “Gpipe: Efficient training of giant neural networks using pipeline parallelism,” in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, Eds., 2019, pp. 103–112. [Online]. Available: https://proceedings.neurips.cc/paper/2019/hash/093f65e080a295f8076b1c5722a46aa2-Abstract.html
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Haiyang Lin (3 papers)
  2. Mingyu Yan (39 papers)
  3. Xiaochun Ye (37 papers)
  4. Dongrui Fan (36 papers)
  5. Shirui Pan (198 papers)
  6. Wenguang Chen (21 papers)
  7. Yuan Xie (188 papers)
Citations (19)
View on Semantic Scholar