Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FedGT: Federated Node Classification with Scalable Graph Transformer (2401.15203v1)

Published 26 Jan 2024 in cs.LG

Abstract: Graphs are widely used to model relational data. As graphs are getting larger and larger in real-world scenarios, there is a trend to store and compute subgraphs in multiple local systems. For example, recently proposed \emph{subgraph federated learning} methods train Graph Neural Networks (GNNs) distributively on local subgraphs and aggregate GNN parameters with a central server. However, existing methods have the following limitations: (1) The links between local subgraphs are missing in subgraph federated learning. This could severely damage the performance of GNNs that follow message-passing paradigms to update node/edge features. (2) Most existing methods overlook the subgraph heterogeneity issue, brought by subgraphs being from different parts of the whole graph. To address the aforementioned challenges, we propose a scalable \textbf{Fed}erated \textbf{G}raph \textbf{T}ransformer (\textbf{FedGT}) in the paper. Firstly, we design a hybrid attention scheme to reduce the complexity of the Graph Transformer to linear while ensuring a global receptive field with theoretical bounds. Specifically, each node attends to the sampled local neighbors and a set of curated global nodes to learn both local and global information and be robust to missing links. The global nodes are dynamically updated during training with an online clustering algorithm to capture the data distribution of the corresponding local subgraph. Secondly, FedGT computes clients' similarity based on the aligned global nodes with optimal transport. The similarity is then used to perform weighted averaging for personalized aggregation, which well addresses the data heterogeneity problem. Moreover, local differential privacy is applied to further protect the privacy of clients. Finally, extensive experimental results on 6 datasets and 2 subgraph settings demonstrate the superiority of FedGT.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. Local differential privacy for deep learning. IEEE Internet of Things Journal, 7(7):5827–5842, 2019.
  2. Federated learning with personalization layers, 2019.
  3. Personalized subgraph federated learning. In International Conference on Machine Learning, pp.  1396–1415. PMLR, 2023.
  4. Waffle: Weighted averaging for personalized federated learning. arXiv preprint arXiv:2110.06978, 2021.
  5. Richard Bellman. Dynamic programming. Science, 153(3731):34–37, 1966.
  6. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P10008, 2008.
  7. Scaling graph neural networks with approximate pagerank. In SIGKDD, pp.  2464–2473, 2020.
  8. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In AAAI, volume 34, pp.  3438–3445, 2020.
  9. Fedgraph: Federated graph learning with intelligent sampling. IEEE Transactions on Parallel and Distributed Systems, 33(8):1775–1786, 2021.
  10. Nagphormer: A tokenized graph transformer for node classification in large graphs. In The Eleventh International Conference on Learning Representations, 2023.
  11. Rethinking attention with performers. ICLR, 2021.
  12. Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. NeurIPS, 2013.
  13. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4):485–532, 2020.
  14. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699, 2020.
  15. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. NeurIPS, 33:3557–3568, 2020.
  16. A point set generation network for 3d object reconstruction from a single image. In CVPR, pp.  605–613, 2017.
  17. Graph neural networks for social recommendation. In The world wide web conference, pp.  417–426, 2019.
  18. Neural message passing for quantum chemistry. In ICML, pp.  1263–1272. PMLR, 2017.
  19. Inductive representation learning on large graphs. In NeurIPS, pp.  1025–1035, 2017a.
  20. Inductive representation learning on large graphs. In NeurIPS Long Beach, CA, USA, pp.  1024–1034, 2017b.
  21. Adaptive transfer learning on graph neural networks. In SIGKDD, pp.  565–574, 2021.
  22. Fedgraphnn: A federated learning system and benchmark for graph neural networks. arXiv preprint arXiv:2104.07145, 2021.
  23. Spreadgnn: Serverless multi-task federated learning for graph neural networks. AAAI, 2022.
  24. Open graph benchmark: Datasets for machine learning on graphs. NeurIPS, 33:22118–22133, 2020a.
  25. Open graph benchmark: Datasets for machine learning on graphs. NeurIPS, 33:22118–22133, 2020b.
  26. Graph structure learning for robust graph neural networks. In SIGKDD, pp.  66–74, 2020.
  27. A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing, 38(4):325–340, 1987.
  28. Metis – unstructured graph partitioning and sparse matrix ordering system, version 2.0. Technical report, 1995.
  29. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM Journal on scientific Computing, 20(1):359–392, 1998.
  30. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  31. Semi-supervised classification with graph convolutional networks. ICLR, 2017.
  32. Rethinking graph transformers with spectral attention. NeurIPS, 2021.
  33. Image-based recommendations on styles and substitutes. In SIGIR, pp.  43–52, 2015.
  34. Communication-efficient learning of deep networks from decentralized data. In AISTATS, 2017a.
  35. Communication-efficient learning of deep networks from decentralized data. In AISTATS, 2017b.
  36. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab, 1999.
  37. Fedni: Federated graph learning with network inpainting for population-based disease prediction. IEEE Transactions on Medical Imaging, 2022.
  38. Privacy-preserving news recommendation model learning. EMNLP, 2020.
  39. Recipe for a general, powerful, scalable graph transformer. NeurIPS, 2022.
  40. E (n) equivariant graph neural networks. In ICML, pp.  9323–9332. PMLR, 2021.
  41. Collective classification in network data. AI magazine, 29(3):93–93, 2008.
  42. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868, 2018.
  43. Personalized federated learning with moreau envelopes. NeurIPS, 33:21394–21405, 2020.
  44. Understanding over-squashing and bottlenecks on graphs via curvature. ICLR, 2021.
  45. Attention is all you need. In NeurIPS, pp.  5998–6008, 2017.
  46. Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
  47. Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10(3152676):10–5555, 2017.
  48. Federatedscope-gnn: Towards a unified, comprehensive and efficient package for federated graph learning. In SIGKDD, pp.  4110–4120, 2022.
  49. Fedgnn: Federated graph neural network for privacy-preserving recommendation. KDD, 2021.
  50. A federated graph neural network framework for privacy-preserving personalization. Nature Communications, 13(1):1–10, 2022a.
  51. Nodeformer: A scalable graph structure learning transformer for node classification. 2022b.
  52. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1):4–24, 2020.
  53. Federated graph classification over non-iid graphs. In NeurIPS, volume 34, pp.  18839–18852. Curran Associates, Inc., 2021a.
  54. Federated graph classification over non-iid graphs. NeurIPS, 34:18839–18852, 2021b.
  55. Federatedscope: A comprehensive and flexible federated learning platform via message passing. SIGKDD, 2022.
  56. On layer normalization in the transformer architecture. In ICML, pp.  10524–10533. PMLR, 2020.
  57. Towards consumer loan fraud detection: Graph neural networks with role-constrained conditional random field. In AAAI, volume 35, pp.  4537–4545, 2021.
  58. Topology attack and defense for graph neural networks: An optimization perspective. IJCAI, 2019a.
  59. How powerful are graph neural networks? ICLR, 2019b.
  60. Local differential privacy and its applications: A comprehensive survey. arXiv preprint arXiv:2008.03686, 2020.
  61. Federated machine learning: Concept and applications. TIST, 2019.
  62. Do transformers really perform bad for graph representation? NeurIPS, 2021.
  63. Online deep clustering for unsupervised representation learning. In CVPR, pp.  6688–6697, 2020.
  64. Heterogeneous graph neural network. In SIGKDD, pp.  793–803, 2019.
  65. Graph-bert: Only attention is needed for learning graph representations. arXiv preprint arXiv:2001.05140, 2020.
  66. Subgraph federated learning with missing neighbor generation. In NeurIPS, volume 34, pp.  6671–6682. Curran Associates, Inc., 2021.
  67. Hierarchical graph transformer with adaptive node sampling. NeurIPS, 2022.
  68. Gophormer: Ego-graph transformer for node classification. arXiv preprint arXiv:2110.13094, 2021.
  69. Graph neural networks: A review of methods and applications. AI open, 1:57–81, 2020.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets