2000 character limit reached
Lumos: Heterogeneity-aware Federated Graph Learning over Decentralized Devices (2303.00492v3)
Published 1 Mar 2023 in cs.LG and cs.DC
Abstract: Graph neural networks (GNN) have been widely deployed in real-world networked applications and systems due to their capability to handle graph-structured data. However, the growing awareness of data privacy severely challenges the traditional centralized model training paradigm, where a server holds all the graph information. Federated learning is an emerging collaborative computing paradigm that allows model training without data centralization. Existing federated GNN studies mainly focus on systems where clients hold distinctive graphs or sub-graphs. The practical node-level federated situation, where each client is only aware of its direct neighbors, has yet to be studied. In this paper, we propose the first federated GNN framework called Lumos that supports supervised and unsupervised learning with feature and degree protection on node-level federated graphs. We first design a tree constructor to improve the representation capability given the limited structural information. We further present a Monte Carlo Markov Chain-based algorithm to mitigate the workload imbalance caused by degree heterogeneity with theoretically-guaranteed performance. Based on the constructed tree for each client, a decentralized tree-based GNN trainer is proposed to support versatile training. Extensive experiments demonstrate that Lumos outperforms the baseline with significantly higher accuracy and greatly reduced communication cost and training time.
- B. Ye, S. Yang, B. Hu, Z. Zhang, Y. He, K. Huang, J. Zhou, and Y. Fang, “Gaia: Graph neural network with temporal shift aware attention for gross merchandise value forecast in e-commerce,” in Proc. IEEE ICDE, 2022.
- S. Wang, H. Li, C. C. Cao, X.-H. Li, N. N. Fai, J. Liu, X. Xue, H. Song, J. Li, G. Gu, and L. Chen, “Tower bridge net (tb-net): Bidirectional knowledge graph aware embedding propagation for explainable recommender systems,” in Proc. IEEE ICDE, 2022.
- G. Li, X. Wang, G. S. Njoo, S. Zhong, S.-H. G. Chan, C.-C. Hung, and W.-C. Peng, “A data-driven spatial-temporal graph neural network for docked bike prediction,” in Proc, IEEE ICDE, 2022.
- European Commission, “2018 reform of eu data protection rules,” https://ec.europa.eu/commission/sites/beta-political/files/data-protection-factsheet-changes_en.pdf, 2018, accessed: 2020-02-06.
- B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics. PMLR, 2017.
- H. Xie, J. Ma, L. Xiong, and C. Yang, “Federated graph classification over non-iid graphs,” in NeuraIPS, 2021.
- C. Chen, W. Hu, Z. Xu, and Z. Zheng, “Fedgl: federated graph learning framework with global self-supervision,” arXiv preprint arXiv:2105.03170, 2021.
- K. Zhang, C. Yang, X. Li, L. Sun, and S. M. Yiu, “Subgraph federated learning with missing neighbor generation,” in NeuraIPS, 2021.
- F. Chen, P. Li, T. Miyazaki, and C. Wu, “Fedgraph: Federated graph learning with intelligent sampling,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 8, pp. 1775–1786, 2021.
- B. Du and C. Wu, “Federated graph learning with periodic neighbour sampling,” in Proc. IEEE/ACM IWQoS, 2022.
- G. Mei, Z. Guo, S. Liu, and L. Pan, “Sgnn: A graph neural network based federated learning approach by hiding structure,” in Proc. IEEE Conf. on Big Data, 2019, pp. 2560–2568.
- S. Sajadmanesh and D. Gatica-Perez, “Locally private graph neural networks,” in Proc. ACM CCS, 2021.
- ethereum.org, “Decentralized social networks,” https://ethereum.org/en/social-networks/, 2022, accessed: 2022-08-01.
- M. Welling and T. N. Kipf, “Semi-supervised classification with graph convolutional networks,” in Proc. ICLR, 2017.
- P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” in Proc. ICLR, 2018.
- J. Zeng, P. Wang, L. Lan, J. Zhao, F. Sun, J. Tao, J. Feng, M. Hu, and X. Guan, “Accurate and scalable graph neural networks for billion-scale graphs,” in Proc. IEEE ICDE, 2022.
- S. Hu, X. Zhang, J. Zhou, S. Ji, J. Yuan, Z. Li, Z. Wang, Q. Chen, Q. He, and L. Fang, “Turbo: Fraud detection in deposit-free leasing service via real-time behavior network mining,” in Proc. IEEE ICDE, 2021.
- Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, “Federated learning with non-iid data,” arXiv preprint arXiv:1806.00582, 2018.
- T. Nishio and R. Yonetani, “Client selection for federated learning with heterogeneous resources in mobile edge,” in Proc. IEEE ICC, 2019.
- X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of fedavg on non-iid data,” in Proc. ICLR, 2019.
- Y. Zhan, P. Li, and S. Guo, “Experience-driven computational resource allocation of federated learning by deep reinforcement learning,” in Proc. IEEE IPDPS, 2020.
- M. S. H. Abad, E. Ozfatura, D. Gunduz, and O. Ercetin, “Hierarchical federated learning across heterogeneous cellular networks,” in Proc. IEEE ICASSP, 2020.
- W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in NeuraIPS, 2017.
- K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How powerful are graph neural networks?” in Proc. ICLR, 2018.
- R. Sato, M. Yamada, and H. Kashima, “Approximation ratios of graph neural networks for combinatorial problems,” NeuraIPS, 2019.
- W. Jin, R. Barzilay, and T. Jaakkola, “Junction tree variational autoencoder for molecular graph generation,” in Proc. PMLR ICML, 2018.
- R. Talak, S. Hu, L. Peng, and L. Carlone, “Neural trees for learning on graphs,” NeuraIPS, 2021.
- Q. Pan and Y. Zhu, “Fedwalk: Communication efficient federated unsupervised node embedding with differential privacy,” in Proc. ACM SIGKDD, 2022.
- U. Erlingsson, V. Pihur, and A. Korolova, “Rappor: Randomized aggregatable privacy-preserving ordinal response,” in Proc. ACM SIGSAC, 2014.
- A. Clauset, C. R. Shalizi, and M. E. Newman, “Power-law distributions in empirical data,” SIAM review, vol. 51, no. 4, pp. 661–703, 2009.
- D. Jungnickel, “A hard problem: The tsp,” in Graphs, Networks and Algorithms. Springer, 1999, pp. 423–469.
- D. Rathee, M. Rathee, N. Kumar, N. Chandran, D. Gupta, A. Rastogi, and R. Sharma, “Cryptflow2: Practical 2-party secure inference,” in Proc. ACM CCS, 2020.
- K. Liu and E. Terzi, “Towards identity anonymization on graphs,” in Proc. ACM SIGMOD, 2008.
- S. Chib and E. Greenberg, “Understanding the metropolis-hastings algorithm,” The american statistician, vol. 49, no. 4, pp. 327–335, 1995.
- M. M. E. Upfal, “Probability and computing: Randomized algorithms and probabilistic analysis,” 2005.
- B. Ding, J. Kulkarni, and S. Yekhanin, “Collecting telemetry data privately,” in NeuraIPS, 2017.
- F. D. McSherry, “Privacy integrated queries: an extensible platform for privacy-preserving data analysis,” in Proc. ACM SIGMOD, 2009.
- G. Brassard, C. Crépeau, and J.-M. Robert, “All-or-nothing disclosure of secrets,” in Conference on the Theory and Application of Cryptographic Techniques. Springer, 1986, pp. 234–238.
- J. Kilian, “Founding crytpography on oblivious transfer,” in Proceedings of the twentieth annual ACM symposium on Theory of computing, 1988, pp. 20–31.
- B. Rozemberczki, C. Allen, and R. Sarkar, “Multi-scale attributed node embedding,” Journal of Complex Networks, vol. 9, no. 2, p. cnab014, 2021.
- B. Rozemberczki and R. Sarkar, “Characteristic functions on graphs: Birds of a feather, from statistical descriptors to parametric models,” in Proc. ACM CIKM, 2020, pp. 1325–1334.
- T. Fawcett, “An introduction to roc analysis,” Pattern recognition letters, vol. 27, no. 8, pp. 861–874, 2006.
- C. Dwork, A. Roth et al., “The algorithmic foundations of differential privacy,” Foundations and Trends in Theoretical Computer Science, vol. 9, no. 3–4, pp. 211–407, 2014.
- S. L. Warner, “Randomized response: A survey technique for eliminating evasive answer bias,” Journal of the American Statistical Association, vol. 60, no. 309, pp. 63–69, 1965.