Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Graph Neural Networks can Recover the Hidden Features Solely from the Graph Structure (2301.10956v4)

Published 26 Jan 2023 in cs.LG and cs.SI

Abstract: Graph Neural Networks (GNNs) are popular models for graph learning problems. GNNs show strong empirical performance in many practical tasks. However, the theoretical properties have not been completely elucidated. In this paper, we investigate whether GNNs can exploit the graph structure from the perspective of the expressive power of GNNs. In our analysis, we consider graph generation processes that are controlled by hidden (or latent) node features, which contain all information about the graph structure. A typical example of this framework is kNN graphs constructed from the hidden features. In our main results, we show that GNNs can recover the hidden node features from the input graph alone, even when all node features, including the hidden features themselves and any indirect hints, are unavailable. GNNs can further use the recovered node features for downstream tasks. These results show that GNNs can fully exploit the graph structure by themselves, and in effect, GNNs can use both the hidden and explicit node features for downstream tasks. In the experiments, we confirm the validity of our results by showing that GNNs can accurately recover the hidden features using a GNN architecture built based on our theoretical analysis.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. The surprising power of graph neural networks with random node initialization. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, IJCAI, pages 2112–2118, 2021.
  2. Morteza Alamgir and Ulrike von Luxburg. Shortest path distance in random k-nearest neighbor graphs. In Proceedings of the 29th International Conference on Machine Learning, ICML, 2012.
  3. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput., 15(6):1373–1396, 2003.
  4. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, AAAI, pages 3438–3445, 2020.
  5. FastGCN: Fast learning with graph convolutional networks via importance sampling. In Proceedings of the 6th International Conference on Learning Representations, ICLR, 2018.
  6. Simple and deep graph convolutional networks. In Proceedings of the 37th International Conference on Machine Learning, ICML, pages 1725–1735, 2020.
  7. Discovering symbolic models from deep learning with inductive biases. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS, 2020.
  8. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, NeurIPS, pages 3837–3845, 2016.
  9. Understanding the representation power of graph neural networks in learning graph topology. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS, pages 15387–15397, 2019.
  10. On random graphs i. Publicationes mathematicae, 6(1):290–297, 1959.
  11. A fair comparison of graph neural networks for graph classification. In Proceedings of the 8th International Conference on Learning Representations, ICLR, 2020.
  12. Graph neural networks for social recommendation. In The Web Conference 2019, WWW, pages 417–426, 2019.
  13. Generalization and representational limits of graph neural networks. In Proceedings of the 37th International Conference on Machine Learning, ICML, pages 3419–3430, 2020.
  14. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, ICML, pages 1263–1272, 2017.
  15. A new model for learning in graph domains. In Proceedings of the International Joint Conference on Neural Networks, IJCNN, volume 2, pages 729–734, 2005.
  16. William L Hamilton. Graph representation learning. Synthesis Lectures on Artifical Intelligence and Machine Learning, 14(3):1–159, 2020.
  17. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, NeurIPS, pages 1024–1034, 2017.
  18. Vision GNN: an image is worth graph of nodes. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS, 2022.
  19. GCN-MF: disease-gene association identification by graph convolutional networks and matrix factorization. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD, pages 705–713, 2019.
  20. Metric recovery from directed unweighted graphs. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics, AISTATS, 2015.
  21. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR, pages 639–648, 2020.
  22. GPT-GNN: generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD, pages 1857–1867, 2020.
  23. The procrustes program: Producing direct rotation to test a hypothesized factor structure. Behavioral science, 7(2):258, 1962.
  24. Stefanie Jegelka. Theory of graph neural networks: Representation and learning. arXiv, abs/2204.07697, 2022.
  25. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, ICLR, 2015.
  26. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations, ICLR, 2014.
  27. Semi-supervised classification with graph convolutional networks. In Proceedings of the 5th International Conference on Learning Representations, ICLR, 2017.
  28. Predict then propagate: Graph neural networks meet personalized pagerank. In Proceedings of the 7th International Conference on Learning Representations, ICLR, 2019.
  29. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, AAAI, pages 3538–3545, 2018.
  30. Structure-aware interactive graph neural networks for the prediction of protein-ligand binding affinity. In Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD, pages 975–985, 2021.
  31. Andreas Loukas. What graph neural networks cannot learn: depth vs width. In Proceedings of the 8th International Conference on Learning Representations, ICLR, 2020.
  32. Provably powerful graph networks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS, pages 2153–2164, 2019.
  33. Invariant and equivariant graph networks. In Proceedings of the 7th International Conference on Learning Representations, ICLR, 2019.
  34. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, AAAI, pages 4602–4609, 2019.
  35. Relational pooling for graph representations. In Proceedings of the 36th International Conference on Machine Learning, ICML, pages 4663–4673, 2019.
  36. Revisiting graph neural networks: All we have is low-pass filters. arXiv, abs/1905.09550, 2019.
  37. Graph neural networks exponentially lose expressive power for node classification. In Proceedings of the 8th International Conference on Learning Representations, ICLR, 2020.
  38. Learning mesh-based simulation with graph networks. In Proceedings of the 9th International Conference on Learning Representations, ICLR, 2021.
  39. GCC: graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD, pages 1150–1160, 2020.
  40. Ryoma Sato. A survey on the expressive power of graph neural networks. arXiv, abs/2003.04078, 2020.
  41. Ryoma Sato. Towards principled user-side recommender systems. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM, pages 1757–1766, 2022.
  42. Approximation ratios of graph neural networks for combinatorial problems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS, pages 4083–4092, 2019.
  43. Random features strengthen graph neural networks. In Proceedings of the 2021 SIAM International Conference on Data Mining, SDM, pages 333–341, 2021.
  44. Constant time graph neural networks. ACM Trans. Knowl. Discov. Data, 16(5):92:1–92:31, 2022.
  45. The graph neural network model. IEEE Trans. Neural Networks, 20(1):61–80, 2009.
  46. The vapnik-chervonenkis dimension of graph and recursive neural networks. Neural Networks, 108:248–259, 2018.
  47. Peter H Schönemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1–10, 1966.
  48. Pitfalls of graph neural network evaluation. arXiv, 2018.
  49. Robin Sibson. Studies in the robustness of multidimensional scaling: Procrustes statistics. Journal of the Royal Statistical Society: Series B (Methodological), 40(2):234–238, 1978.
  50. Robin Sibson. Studies in the robustness of multidimensional scaling: Perturbational analysis of classical scaling. Journal of the Royal Statistical Society: Series B (Methodological), 41(2):217–229, 1979.
  51. Diffusion processes with boundary conditions. Communications on Pure and Applied Mathematics, 24(2):147–225, 1971.
  52. Consistent latent position estimation and vertex classification for random dot product graphs. IEEE Trans. Pattern Anal. Mach. Intell., 36(1):48–57, 2014.
  53. A global geometric framework for nonlinear dimensionality reduction. science, 290(5500):2319–2323, 2000.
  54. Yoshikazu Terada and Ulrike von Luxburg. Local ordinal embedding. In Proceedings of the 31st International Conference on Machine Learning, ICML, pages 847–855, 2014.
  55. Graph attention networks. In Proceedings of the 6th International Conference on Learning Representations, ICLR, 2018.
  56. Deep graph infomax. In Proceedings of the 7th International Conference on Learning Representations, ICLR, 2019.
  57. Ulrike von Luxburg and Morteza Alamgir. Density estimation from unweighted k-nearest neighbor graphs: a roadmap. In Advances in Neural Information Processing Systems 26: Annual Conference on Neural Information Processing Systems 2013, NeurIPS, pages 225–233, 2013.
  58. AM-GCN: adaptive multi-channel graph convolutional networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD, pages 1243–1253, 2020.
  59. Traffic flow prediction via spatial temporal graph neural network. In The Web Conference 2020, WWW, pages 1082–1092, 2020.
  60. Simplifying graph convolutional networks. In Proceedings of the 36th International Conference on Machine Learning, ICML, pages 6861–6871, 2019.
  61. How powerful are graph neural networks? In Proceedings of the 7th International Conference on Learning Representations, ICLR, 2019.
  62. What can neural networks reason about? In Proceedings of the 8th International Conference on Learning Representations, ICLR, 2020.
  63. Revisiting semi-supervised learning with graph embeddings. In Proceedings of the 33rd International Conference on Machine Learning, ICML, pages 40–48, 2016.
  64. When does self-supervision help graph convolutional networks? In Proceedings of the 37th International Conference on Machine Learning, ICML, pages 10871–10880, 2020.
  65. Link prediction based on graph neural networks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS, pages 5171–5181, 2018.
  66. Layer-dependent importance sampling for training deep and large graph convolutional networks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS, pages 11247–11256, 2019.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now