Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Exploring Task Unification in Graph Representation Learning via Generative Approach (2403.14340v1)

Published 21 Mar 2024 in cs.LG and cs.AI

Abstract: Graphs are ubiquitous in real-world scenarios and encompass a diverse range of tasks, from node-, edge-, and graph-level tasks to transfer learning. However, designing specific tasks for each type of graph data is often costly and lacks generalizability. Recent endeavors under the "Pre-training + Fine-tuning" or "Pre-training + Prompt" paradigms aim to design a unified framework capable of generalizing across multiple graph tasks. Among these, graph autoencoders (GAEs), generative self-supervised models, have demonstrated their potential in effectively addressing various graph tasks. Nevertheless, these methods typically employ multi-stage training and require adaptive designs, which on one hand make it difficult to be seamlessly applied to diverse graph tasks and on the other hand overlook the negative impact caused by discrepancies in task objectives between the different stages. To address these challenges, we propose GA2E, a unified adversarially masked autoencoder capable of addressing the above challenges seamlessly. Specifically, GA2E proposes to use the subgraph as the meta-structure, which remains consistent across all graph tasks (ranging from node-, edge-, and graph-level to transfer learning) and all stages (both during training and inference). Further, GA2E operates in a \textbf{"Generate then Discriminate"} manner. It leverages the masked GAE to reconstruct the input subgraph whilst treating it as a generator to compel the reconstructed graphs resemble the input subgraph. Furthermore, GA2E introduces an auxiliary discriminator to discern the authenticity between the reconstructed (generated) subgraph and the input subgraph, thus ensuring the robustness of the graph representation through adversarial training mechanisms. We validate GA2E's capabilities through extensive experiments on 21 datasets across four types of graph tasks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. Subgraph neural networks. Advances in Neural Information Processing Systems 33 (2020), 8017–8029.
  2. NetGAN: Generating Graphs via Random Walks. arXiv:1803.00816 [stat.ML]
  3. Mining hidden community in heterogeneous social networks. In Proceedings of the 3rd international workshop on Link discovery. 58–65.
  4. When to Pre-Train Graph Neural Networks? An Answer from Data Generation Perspective! arXiv preprint arXiv:2303.16458 (2023).
  5. Adversarial Network Embedding. arXiv:1711.07838 [cs.LG]
  6. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
  7. Understanding and extending subgraph gnns by rethinking their symmetries. Advances in Neural Information Processing Systems 35 (2022), 31376–31390.
  8. Generative adversarial nets. Advances in neural information processing systems 27 (2014).
  9. Semi-Implicit Graph Variational Auto-Encoders. arXiv:1908.07078 [cs.LG]
  10. Semi-implicit graph variational auto-encoders. Advances in neural information processing systems 32 (2019).
  11. Kaveh Hassani and Amir Hosein Khasahmadi. 2020a. Contrastive multi-view representation learning on graphs. In International conference on machine learning. PMLR, 4116–4126.
  12. Kaveh Hassani and Amir Hosein Khasahmadi. 2020b. Contrastive multi-view representation learning on graphs. In International conference on machine learning. PMLR, 4116–4126.
  13. Masked autoencoders are scalable vision learners. 2022 IEEE. In CVF Conference on Computer Vision and Pattern Recognition (CVPR). 15979–15988.
  14. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 (2018).
  15. GraphMAE: Self-Supervised Masked Graph Autoencoders. arXiv:2205.10803 [cs.LG]
  16. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 594–604.
  17. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1857–1867.
  18. Kexin Huang and Marinka Zitnik. 2020. Graph meta learning via local subgraphs. Advances in neural information processing systems 33 (2020), 5862–5874.
  19. A survey on generative adversarial networks: Variants, applications, and training. ACM Computing Surveys (CSUR) 54, 8 (2021), 1–49.
  20. Application of deep learning methods in biological networks. Briefings in bioinformatics 22, 2 (2021), 1902–1917.
  21. Thomas N Kipf and Max Welling. 2016a. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).
  22. Thomas N Kipf and Max Welling. 2016b. Semi-Supervised Classification with Graph Convolutional Networks. In International Conference on Learning Representations.
  23. Thomas N. Kipf and Max Welling. 2016c. Variational Graph Auto-Encoders. arXiv:1611.07308 [stat.ML]
  24. A Robust and Generalized Framework for Adversarial Graph Embedding. IEEE Transactions on Knowledge and Data Engineering 35, 11 (2023), 11004–11018. https://doi.org/10.1109/TKDE.2023.3235944
  25. What’s Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoders. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1268–1279.
  26. What’s Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoders. arXiv:2205.10053 [cs.LG]
  27. SeeGera: Self-supervised Semi-implicit Graph Variational Auto-encoders with Masking. arXiv:2301.12458 [cs.LG]
  28. SeeGera: Self-supervised Semi-implicit Graph Variational Auto-encoders with Masking. In Proceedings of the ACM Web Conference 2023. 143–153.
  29. One for All: Towards Training One Graph Model for All Classification Tasks. arXiv preprint arXiv:2310.00149 (2023).
  30. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. Comput. Surveys 55, 9 (2023), 1–35.
  31. Graph self-supervised learning: A survey. IEEE Transactions on Knowledge and Data Engineering 35, 6 (2022), 5879–5900.
  32. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In Proceedings of the ACM Web Conference 2023. 417–428.
  33. GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks. In Proceedings of the ACM Web Conference 2023 (Austin, TX, USA) (WWW ’23). Association for Computing Machinery, New York, NY, USA, 417–428. https://doi.org/10.1145/3543507.3583386
  34. László Lovász. 1993. Random walks on graphs. Combinatorics, Paul erdos is eighty 2, 1-46 (1993), 4.
  35. Co-embedding attributed networks. In Proceedings of the twelfth ACM international conference on web search and data mining. 393–401.
  36. Adversarially Regularized Graph Autoencoder for Graph Embedding. arXiv:1802.04407 [cs.LG]
  37. Symmetric graph convolutional autoencoder for unsupervised graph representation learning. In Proceedings of the IEEE/CVF international conference on computer vision. 6519–6528.
  38. Gcc: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 1150–1160.
  39. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research 12, 9 (2011).
  40. A survey of heterogeneous information network analysis. IEEE Transactions on Knowledge and Data Engineering 29, 1 (2016), 17–37.
  41. Teague Sterling and John J Irwin. 2015. ZINC 15–ligand discovery for everyone. Journal of chemical information and modeling 55, 11 (2015), 2324–2337.
  42. InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization. In International Conference on Learning Representations.
  43. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1717–1727.
  44. All in One: Multi-Task Prompting for Graph Neural Networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (¡conf-loc¿, ¡city¿Long Beach¡/city¿, ¡state¿CA¡/state¿, ¡country¿USA¡/country¿, ¡/conf-loc¿) (KDD ’23). Association for Computing Machinery, New York, NY, USA, 2120–2131. https://doi.org/10.1145/3580305.3599256
  45. S2GAE: Self-Supervised Graph Autoencoders Are Generalizable Learners with Graph Masking (WSDM ’23). 787–795.
  46. Heterogeneous Graph Masked Autoencoders. arXiv:2208.09957 [cs.LG]
  47. Graph attention networks. stat 1050, 20 (2017), 10–48550.
  48. Deep graph infomax. arXiv preprint arXiv:1809.10341 (2018).
  49. Deep Graph Infomax. In International Conference on Learning Representations. https://openreview.net/forum?id=rklz9iAcKQ
  50. GraphGAN: Graph Representation Learning with Generative Adversarial Nets. arXiv:1711.08267 [cs.LG]
  51. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837.
  52. Graph neural networks in recommender systems: a survey. Comput. Surveys 55, 5 (2022), 1–37.
  53. A survey of graph prompting methods: techniques, applications, and challenges. arXiv preprint arXiv:2303.07275 (2023).
  54. Graph convolutional networks with markov random field reasoning for social spammer detection. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 1054–1061.
  55. MoleculeNet: a benchmark for molecular machine learning. Chemical science 9, 2 (2018), 513–530.
  56. Towards effective and generalizable fine-tuning for pre-trained molecular graph models. bioRxiv (2022), 2022–02.
  57. A survey of pretraining on graphs: Taxonomy, methods, and applications. arXiv preprint arXiv:2202.07893 (2022).
  58. Infogcl: Information-aware graph contrastive learning. Advances in Neural Information Processing Systems 34 (2021), 30414–30425.
  59. Self-supervised graph-level representation learning with local and global structure. In International Conference on Machine Learning. PMLR, 11548–11558.
  60. Pinar Yanardag and S.V.N. Vishwanathan. 2015a. Deep Graph Kernels. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Sydney, NSW, Australia) (KDD ’15). Association for Computing Machinery, New York, NY, USA, 1365–1374. https://doi.org/10.1145/2783258.2783417
  61. Pinar Yanardag and SVN Vishwanathan. 2015b. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 1365–1374.
  62. Graph contrastive learning automated. In International Conference on Machine Learning. PMLR, 12121–12132.
  63. Graph contrastive learning with augmentations. Advances in neural information processing systems 33 (2020), 5812–5823.
  64. Generalized Graph Prompt: Toward a Unification of Pre-Training and Downstream Tasks on Graphs. arXiv:2311.15317 [cs.LG]
  65. From canonical correlation analysis to self-supervised graph neural networks. Advances in Neural Information Processing Systems 34 (2021), 76–89.
  66. GCGAN: Generative Adversarial Nets with Graph CNN for Network-Scale Traffic Prediction. 2019 International Joint Conference on Neural Networks (IJCNN) (2019), 1–8. https://api.semanticscholar.org/CorpusID:203605505
  67. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131 (2020).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Yulan Hu (13 papers)
  2. Sheng Ouyang (13 papers)
  3. Zhirui Yang (6 papers)
  4. Ge Chen (64 papers)
  5. Junchen Wan (7 papers)
  6. Xiao Wang (507 papers)
  7. Yong Liu (721 papers)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now