Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Inductive Graph Alignment Prompt: Bridging the Gap between Graph Pre-training and Inductive Fine-tuning From Spectral Perspective (2402.13556v1)

Published 21 Feb 2024 in cs.LG and cs.AI

Abstract: The "Graph pre-training and fine-tuning" paradigm has significantly improved Graph Neural Networks(GNNs) by capturing general knowledge without manual annotations for downstream tasks. However, due to the immense gap of data and tasks between the pre-training and fine-tuning stages, the model performance is still limited. Inspired by prompt fine-tuning in Natural Language Processing(NLP), many endeavors have been made to bridge the gap in graph domain. But existing methods simply reformulate the form of fine-tuning tasks to the pre-training ones. With the premise that the pre-training graphs are compatible with the fine-tuning ones, these methods typically operate in transductive setting. In order to generalize graph pre-training to inductive scenario where the fine-tuning graphs might significantly differ from pre-training ones, we propose a novel graph prompt based method called Inductive Graph Alignment Prompt(IGAP). Firstly, we unify the mainstream graph pre-training frameworks and analyze the essence of graph pre-training from graph spectral theory. Then we identify the two sources of the data gap in inductive setting: (i) graph signal gap and (ii) graph structure gap. Based on the insight of graph pre-training, we propose to bridge the graph signal gap and the graph structure gap with learnable prompts in the spectral space. A theoretical analysis ensures the effectiveness of our method. At last, we conduct extensive experiments among nodes classification and graph classification tasks under the transductive, semi-inductive and inductive settings. The results demonstrate that our proposed method can successfully bridge the data gap under different settings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. The pushshift reddit dataset. In Proceedings of the international AAAI conference on web and social media, Vol. 14. 830–839.
  2. Aleksandar Bojchevski and Stephan Günnemann. 2017. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. arXiv preprint arXiv:1707.03815 (2017).
  3. Flex: Unifying evaluation for few-shot nlp. Advances in Neural Information Processing Systems 34 (2021), 15787–15800.
  4. Spectral networks and locally connected networks on graphs. In Proceedings of the 2nd International Conference on Learning Representations.
  5. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 1597–1607.
  6. Xinlei Chen and Kaiming He. 2021. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 15750–15758.
  7. Masked spectrogram prediction for self-supervised audio pre-training. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5.
  8. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
  9. Unified language model pre-training for natural language understanding and generation. Advances in neural information processing systems 32 (2019).
  10. Universal Prompt Tuning for Graph Neural Networks. arXiv preprint arXiv:2209.15240 (2022).
  11. Spatial-temporal graph ode networks for traffic flow forecasting. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 364–373.
  12. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems 33 (2020), 21271–21284.
  13. Inductive Representation Learning on Large Graphs. In Proceedings of the 31st Conference on Neural Information Processing Systems.
  14. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9729–9738.
  15. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1857–1867.
  16. Visual prompt tuning. In European Conference on Computer Vision. Springer, 709–727.
  17. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020).
  18. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
  19. Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In Proceedings of the 5th International Conference on Learning Representations.
  20. Greg Landrum. 2006. RDKit: Open-source cheminformatics. 2006. Google Scholar (2006).
  21. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021).
  22. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. Comput. Surveys 55, 9 (2023), 1–35.
  23. Vivian Liu and Lydia B Chilton. 2022. Design guidelines for prompt engineering text-to-image generative models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–23.
  24. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019).
  25. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In Proceedings of the ACM Web Conference 2023. 417–428.
  26. Graph structural-topic neural network. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1065–1073.
  27. Theoretically improving graph neural networks via anonymous walk graph kernels. In Proceedings of the Web Conference 2021. 1204–1214.
  28. HGK-GNN: Heterogeneous Graph Kernel based Graph Neural Networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 1129–1138.
  29. Gcan: Graph convolutional adversarial network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8266–8276.
  30. Spatio-temporal deep graph infomax. arXiv preprint arXiv:1904.06316 (2019).
  31. Graph Representation Learning via Graphical Mutual Information Maximization. In Proceedings of The Web Conference 2020. 259–270.
  32. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748–8763.
  33. Improving language understanding by generative pre-training. (2018).
  34. Curriculum graph co-teaching for multi-target domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5351–5360.
  35. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868 (2018).
  36. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1717–1727.
  37. All in One: Multi-Task Prompting for Graph Neural Networks. (2023).
  38. Graph attention networks. In Proceedings of the 6th International Conference on Learning Representations.
  39. Deep Graph Infomax.. In ICLR (Poster).
  40. Microsoft academic graph: When experts are not enough. Quantitative Science Studies 1, 1 (2020), 396–413.
  41. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021).
  42. Unsupervised domain adaptive graph convolutional networks. In Proceedings of The Web Conference 2020. 1457–1467.
  43. MoleculeNet: a benchmark for molecular machine learning. Chemical science 9, 2 (2018), 513–530.
  44. How Powerful are Graph Neural Networks?. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
  45. Jiaqi Zeng and Pengtao Xie. 2021. Contrastive self-supervised learning for graph classification. In Proceedings of the AAAI conference on Artificial Intelligence, Vol. 35. 10824–10832.
  46. Continual Learning on Dynamic Graphs via Parameter Isolation. arXiv preprint arXiv:2305.13825 (2023).
  47. Dane: Domain adaptive network embedding. arXiv preprint arXiv:1906.00684 (2019).
  48. Exploring recommendation capabilities of gpt-4v (ision): A preliminary case study. arXiv preprint arXiv:2311.04199 (2023).
  49. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131 (2020).
  50. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021. 2069–2080.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Yuchen Yan (44 papers)
  2. Peiyan Zhang (21 papers)
  3. Zheng Fang (103 papers)
  4. Qingqing Long (25 papers)
Citations (9)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets