Papers
Topics
Authors
Recent
Search
2000 character limit reached

Reliable and Compact Graph Fine-tuning via GraphSparse Prompting

Published 29 Oct 2024 in cs.LG | (2410.21749v1)

Abstract: Recently, graph prompt learning has garnered increasing attention in adapting pre-trained GNN models for downstream graph learning tasks. However, existing works generally conduct prompting over all graph elements (e.g., nodes, edges, node attributes, etc.), which is suboptimal and obviously redundant. To address this issue, we propose exploiting sparse representation theory for graph prompting and present Graph Sparse Prompting (GSP). GSP aims to adaptively and sparsely select the optimal elements (e.g., certain node attributes) to achieve compact prompting for downstream tasks. Specifically, we propose two kinds of GSP models, termed Graph Sparse Feature Prompting (GSFP) and Graph Sparse multi-Feature Prompting (GSmFP). Both GSFP and GSmFP provide a general scheme for tuning any specific pre-trained GNNs that can achieve attribute selection and compact prompt learning simultaneously. A simple yet effective algorithm has been designed for solving GSFP and GSmFP models. Experiments on 16 widely-used benchmark datasets validate the effectiveness and advantages of the proposed GSFPs.

Authors (5)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. X. Li, L. Sun, M. Ling, and Y. Peng, “A survey of graph neural network based recommendation in social networks,” Neurocomputing, vol. 549, p. 126441, 2023.
  2. J. Xiong, Z. Xiong, K. Chen, H. Jiang, and M. Zheng, “Graph neural networks for automated de novo drug design,” Drug Discovery Today, vol. 26, no. 6, pp. 1382–1393, 2021.
  3. Y. Liu, X. Ao, Z. Qin, J. Chi, J. Feng, H. Yang, and Q. He, “Pick and choose: A gnn-based imbalanced learning approach for fraud detection,” in Proceedings of the Web Conference 2021, p. 3168–3177, 2021.
  4. L. Kong, H. Yang, W. Li, J. Guan, and S. Zhou, “Traffexplainer: A framework towards gnn-based interpretable traffic prediction,” IEEE Transactions on Artificial Intelligence, pp. 1–15, 2024.
  5. Y. Xie, Z. Xu, J. Zhang, Z. Wang, and S. Ji, “Self-supervised learning of graph neural networks: A unified review,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 2, pp. 2412–2429, 2023.
  6. L. Wang, M. Zhang, Z. Jia, Q. Li, C. Bao, K. Ma, J. Zhu, and Y. Zhong, “Afec: Active forgetting of negative transfer in continual learning,” Advances in Neural Information Processing Systems, vol. 34, pp. 22379–22391, 2021.
  7. X. Sun, J. Zhang, X. Wu, H. Cheng, Y. Xiong, and J. Li, “Graph prompt learning: A comprehensive survey and beyond,” arXiv:2311.16534, 2023.
  8. M. Sun, K. Zhou, X. He, Y. Wang, and X. Wang, “Gppt: Graph pre-training and prompt tuning to generalize graph neural networks,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 1717–1727, 2022.
  9. Z. Liu, X. Yu, Y. Fang, and X. Zhang, “Graphprompt: Unifying pre-training and downstream tasks for graph neural networks,” in Proceedings of the ACM Web Conference 2023, p. 417–428, 2023.
  10. X. Sun, H. Cheng, J. Li, B. Liu, and J. Guan, “All in one: Multi-task prompting for graph neural networks,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 2120–2131, 2023.
  11. Y. Guo, C. Yang, Y. Chen, J. Liu, C. Shi, and J. Du, “A data-centric framework to endow graph neural networks with out-of-distribution detection ability,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 638–648, 2023.
  12. Q. Ge, Z. Zhao, Y. Liu, A. Cheng, X. Li, S. Wang, and D. Yin, “Psp: Pre-training and structure prompt tuning for graph neural networks,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases, p. 423–439, 2024.
  13. T. Fang, Y. Zhang, Y. YANG, C. Wang, and L. Chen, “Universal prompt tuning for graph neural networks,” in Advances in Neural Information Processing Systems, vol. 36, pp. 52464–52489, 2023.
  14. J. Lee, W. Yang, and J. Kang, “Subgraph-level universal prompt tuning,” arXiv preprint arXiv:2402.10380, 2024.
  15. B. Jiang, H. Wu, Z. Zhang, B. Wang, and J. Tang, “A unified graph selective prompt learning for graph neural networks,” arXiv preprint arXiv:2406.10498, 2024.
  16. Z. Tan, R. Guo, K. Ding, and H. Liu, “Virtual node tuning for few-shot node classification,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 2177–2188, 2023.
  17. Q. Wang, X. Sun, and H. Cheng, “Does graph prompt work? a data operation perspective with theoretical analysis,” arXiv preprint arXiv:2410.01635, 2024.
  18. F. Nie, H. Huang, X. Cai, and C. Ding, “Efficient and robust feature selection via joint ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT-norms minimization,” in Advances in Neural Information Processing Systems, vol. 23, pp. 1813–1821, 2010.
  19. J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma, “Robust face recognition via sparse representation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 2, pp. 210–227, 2009.
  20. P. Veličković, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengio, and R. D. Hjelm, “Deep graph infomax,” in International Conference on Learning Representations, 2019.
  21. Z. Hou, X. Liu, Y. Cen, Y. Dong, H. Yang, C. Wang, and J. Tang, “Graphmae: Self-supervised masked graph autoencoders,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 594–604, 2022.
  22. T. N. Kipf and M. Welling, “Variational graph auto-encoders,” NeurIPS Workshop on Bayesian Deep Learning, 2016.
  23. Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, “Graph contrastive learning with augmentations,” in Advances in Neural Information Processing Systems, vol. 33, pp. 5812–5823, 2020.
  24. J. Xia, L. Wu, J. Chen, B. Hu, and S. Z. Li, “Simgrace: A simple framework for graph contrastive learning without data augmentation,” in Proceedings of the ACM Web Conference 2022, p. 1070–1079, 2022.
  25. W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang, “Self-supervised learning on graphs: Deep insights and new direction,” arXiv preprint arXiv:2006.10141, 2020.
  26. Q. Huang, H. Ren, P. Chen, G. Kržmanc, D. Zeng, P. S. Liang, and J. Leskovec, “Prodigy: Enabling in-context learning over graphs,” in Advances in Neural Information Processing Systems, vol. 36, pp. 16302–16317, 2023.
  27. H. Liu, J. Feng, L. Kong, N. Liang, D. Tao, Y. Chen, and M. Zhang, “One for all: Towards training one graph model for all classification tasks,” in The Twelfth International Conference on Learning Representations, 2024.
  28. Y. Ma, N. Yan, J. Li, M. Mortazavi, and N. V. Chawla, “Hetgpt: Harnessing the power of prompt tuning in pre-trained heterogeneous graph neural networks,” in Proceedings of the ACM Web Conference 2024, p. 1015–1023, 2024.
  29. X. Yu, Y. Fang, Z. Liu, and X. Zhang, “Hgprompt: Bridging homogeneous and heterogeneous graphs for few-shot prompt learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 16578–16586, 2024.
  30. X. Yu, C. Zhou, Y. Fang, and X. Zhang, “Text-free multi-domain graph pre-training: Toward graph foundation models,” arXiv preprint arXiv:2405.13934, 2024.
  31. R. Tibshirani, “Regression shrinkage and selection via the lasso,” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 58, no. 1, pp. 267–288, 1996.
  32. P. L. Combettes and J.-C. Pesquet, “Proximal splitting methods in signal processing,” Fixed-Point Algorithms for Inverse Problems in Science and Engineering, pp. 185–212, 2011.
  33. P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Galligher, and T. Eliassi-Rad, “Collective classification in network data,” AI magazine, vol. 29, no. 3, pp. 93–93, 2008.
  34. O. Shchur, M. Mumme, A. Bojchevski, and S. Günnemann, “Pitfalls of graph neural network evaluation,” NeurIPS Workshop on Relational Representation Learning, 2018.
  35. A. Sinha, Z. Shen, Y. Song, H. Ma, D. Eide, B.-J. P. Hsu, and K. Wang, “An overview of microsoft academic service (mas) and applications,” in Proceedings of the 24th International Conference on World Wide Web, p. 243–246, 2015.
  36. H. Pei, B. Wei, K. C.-C. Chang, Y. Lei, and B. Yang, “Geom-gcn: Geometric graph convolutional networks,” in International Conference on Learning Representations, 2020.
  37. W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec, “Open graph benchmark: Datasets for machine learning on graphs,” in Advances in Neural Information Processing Systems, vol. 33, pp. 22118–22133, 2020.
  38. R. A. Rossi and N. K. Ahmed, “The network data repository with interactive graph analytics and visualization,” in Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, p. 4292–4293, 2015.
  39. N. Kriege and P. Mutzel, “Subgraph matching kernels for attributed graphs,” in Proceedings of the 29th International Coference on International Conference on Machine Learning, p. 291–298, 2012.
  40. P. Yanardag and S. Vishwanathan, “Deep graph kernels,” in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, p. 1365–1374, 2015.
  41. C. Zi, H. Zhao, X. Sun, Y. Lin, H. Cheng, and J. Li, “Prog: A graph prompt learning benchmark,” Advances in Neural Information Processing Systems, 2024.
  42. H. Zhao, A. Chen, X. Sun, H. Cheng, and J. Li, “All in one and one for all: A simple yet effective method towards cross-domain graph pretraining,” p. 4443–4454, 2024.
  43. T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in International Conference on Learning Representations, 2017.
  44. H. Zeng, H. Zhou, A. Srivastava, R. Kannan, and V. Prasanna, “Graphsaint: Graph sampling based inductive learning method,” in International Conference on Learning Representations, 2020.
  45. P. D. Dobson and A. J. Doig, “Distinguishing enzyme structures from non-enzymes without alignments,” Journal of Molecular Biology, vol. 330, no. 4, pp. 771–783, 2003.
  46. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in International Conference on Learning Representations, 2018.
  47. W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in Advances in Neural Information Processing Systems, vol. 30, pp. 1024–1034, 2017.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free