Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On the Connection Between MPNN and Graph Transformer (2301.11956v4)

Published 27 Jan 2023 in cs.LG

Abstract: Graph Transformer (GT) recently has emerged as a new paradigm of graph learning algorithms, outperforming the previously popular Message Passing Neural Network (MPNN) on multiple benchmarks. Previous work (Kim et al., 2022) shows that with proper position embedding, GT can approximate MPNN arbitrarily well, implying that GT is at least as powerful as MPNN. In this paper, we study the inverse connection and show that MPNN with virtual node (VN), a commonly used heuristic with little theoretical understanding, is powerful enough to arbitrarily approximate the self-attention layer of GT. In particular, we first show that if we consider one type of linear transformer, the so-called Performer/Linear Transformer (Choromanski et al., 2020; Katharopoulos et al., 2020), then MPNN + VN with only O(1) depth and O(1) width can approximate a self-attention layer in Performer/Linear Transformer. Next, via a connection between MPNN + VN and DeepSets, we prove the MPNN + VN with O(nd) width and O(1) depth can approximate the self-attention layer arbitrarily well, where d is the input feature dimension. Lastly, under some assumptions, we provide an explicit construction of MPNN + VN with O(1) width and O(n) depth approximating the self-attention layer in GT arbitrarily well. On the empirical side, we demonstrate that 1) MPNN + VN is a surprisingly strong baseline, outperforming GT on the recently proposed Long Range Graph Benchmark (LRGB) dataset, 2) our MPNN + VN improves over early implementation on a wide range of OGB datasets and 3) MPNN + VN outperforms Linear Transformer and MPNN on the climate modeling task.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. On the bottleneck of graph neural networks and its practical implications. arXiv preprint arXiv:2006.05205, 2020.
  2. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.
  3. How attentive are graph attention networks? arXiv preprint arXiv:2105.14491, 2021.
  4. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006.13318, 2020.
  5. Structure-aware transformer for graph representation learning. In International Conference on Machine Learning, pp. 3469–3489. PMLR, 2022.
  6. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.
  7. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020.
  8. Cybenko, G. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303–314, 1989.
  9. Deep learning for physical processes: Incorporating prior scientific knowledge. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=By4HsfWAZ.
  10. Deep learning for physical processes: incorporating prior scientific knowledge. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124009, dec 2019. doi: 10.1088/1742-5468/ab3195. URL https://dx.doi.org/10.1088/1742-5468/ab3195.
  11. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  12. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699, 2020.
  13. Long range graph benchmark. arXiv preprint arXiv:2206.08164, 2022.
  14. Convit: Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning, pp. 2286–2296. PMLR, 2021.
  15. Transformers meet directed graphs. arXiv preprint arXiv:2302.00049, 2023.
  16. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263–1272. PMLR, 2017.
  17. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 2022.
  18. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118–22133, 2020.
  19. Ogb-lsc: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2103.09430, 2021.
  20. Improvements of the daily optimum interpolation sea surface temperature (doisst) version 2.1. Journal of Climate, 34(8):2923 – 2939, 2021. doi: 10.1175/JCLI-D-20-0166.1. URL https://journals.ametsoc.org/view/journals/clim/34/8/JCLI-D-20-0166.1.xml.
  21. Global self-attention as a replacement for graph convolution. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp.  655–665, 2022.
  22. An analysis of virtual nodes in graph neural networks for link prediction. In Learning on Graphs Conference, 2022.
  23. Ammus: A survey of transformer-based pretrained models in natural language processing. arXiv preprint arXiv:2108.05542, 2021.
  24. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the International Conference on Machine Learning (ICML), 2020a. URL https://arxiv.org/abs/2006.16236.
  25. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pp. 5156–5165. PMLR, 2020b.
  26. Pure transformers are powerful graph learners. arXiv preprint arXiv:2207.02505, 2022.
  27. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
  28. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems, 34:21618–21629, 2021.
  29. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI conference on artificial intelligence, 2018.
  30. Sign and basis invariant networks for spectral graph representation learning. arXiv preprint arXiv:2202.13013, 2022.
  31. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  10012–10022, 2021.
  32. Invariant and equivariant graph networks. arXiv preprint arXiv:1812.09902, 2018.
  33. Graphit: Encoding graph structure in transformers. arXiv preprint arXiv:2106.05667, 2021.
  34. Attending to graph transformers. arXiv preprint arXiv:2302.04181, 2023.
  35. Janossy pooling: Learning deep permutation-invariant functions for variable-size inputs. arXiv preprint arXiv:1811.01900, 2018.
  36. Graph neural networks exponentially lose expressive power for node classification. arXiv preprint arXiv:1905.10947, 2019.
  37. Grpe: Relative positional encoding for graph transformer. In ICLR2022 Machine Learning for Drug Discovery, 2022.
  38. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  652–660, 2017.
  39. Recipe for a general, powerful, scalable graph transformer. arXiv preprint arXiv:2205.12454, 2022.
  40. Daily high-resolution blended analyses for sea surface temperature. J. Climate, 20:5473–5496, 2007.
  41. A simple neural network module for relational reasoning. Advances in neural information processing systems, 30, 2017.
  42. On universal equivariant set networks. arXiv preprint arXiv:1910.02421, 2019.
  43. Benchmarking graphormer on large-scale molecular modeling datasets. arXiv preprint arXiv:2203.04810, 2022.
  44. Efficient transformers: A survey. ACM Computing Surveys (CSUR), 2020.
  45. Understanding over-squashing and bottlenecks on graphs via curvature. arXiv preprint arXiv:2111.14522, 2021.
  46. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  47. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.
  48. Universal approximation of functions on sets. Journal of Machine Learning Research, 23(151):1–56, 2022.
  49. Towards physics-informed deep learning for turbulent flow prediction. pp.  1457–1466, 08 2020a. doi: 10.1145/3394486.3403198.
  50. Meta-learning dynamics forecasting using task inference. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=BsSP7pZGFQO.
  51. Linformer: Self-attention with linear complexity, 2020b.
  52. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp.  38–45, 2020.
  53. Nodeformer: A scalable graph structure learning transformer for node classification. In Advances in Neural Information Processing Systems, 2022.
  54. Representing long-range context for graph neural networks with global attention. Advances in Neural Information Processing Systems, 34:13266–13279, 2021.
  55. Revisiting over-smoothing in deep gcns. arXiv preprint arXiv:2003.13663, 2020.
  56. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34:28877–28888, 2021.
  57. Deep sets. Advances in neural information processing systems, 30, 2017.
  58. Pairnorm: Tackling oversmoothing in gnns. arXiv preprint arXiv:1909.12223, 2019.
  59. Exponential separations in symmetric neural networks. arXiv preprint arXiv:2206.01266, 2022.
Citations (37)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now