Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

UniGraph: Learning a Unified Cross-Domain Foundation Model for Text-Attributed Graphs (2402.13630v2)

Published 21 Feb 2024 in cs.LG

Abstract: Foundation models like ChatGPT and GPT-4 have revolutionized artificial intelligence, exhibiting remarkable abilities to generalize across a wide array of tasks and applications beyond their initial training objectives. However, graph learning has predominantly focused on single-graph models, tailored to specific tasks or datasets, lacking the ability to transfer learned knowledge to different domains. This limitation stems from the inherent complexity and diversity of graph structures, along with the different feature and label spaces specific to graph data. In this paper, we recognize text as an effective unifying medium and employ Text-Attributed Graphs (TAGs) to leverage this potential. We present our UniGraph framework, designed to learn a foundation model for TAGs, which is capable of generalizing to unseen graphs and tasks across diverse domains. Unlike single-graph models that use pre-computed node features of varying dimensions as input, our approach leverages textual features for unifying node representations, even for graphs such as molecular graphs that do not naturally have textual features. We propose a novel cascaded architecture of LLMs (LMs) and Graph Neural Networks (GNNs) as backbone networks. Additionally, we propose the first pre-training algorithm specifically designed for large-scale self-supervised learning on TAGs, based on Masked Graph Modeling. We introduce graph instruction tuning using LLMs to enable zero-shot prediction ability. Our comprehensive experiments across various graph learning tasks and domains demonstrate the model's effectiveness in self-supervised representation learning on unseen graphs, few-shot in-context transfer, and zero-shot transfer, even surpassing or matching the performance of GNNs that have undergone supervised training on target datasets.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Inside pagerank. ACM Transactions on Internet Technology (TOIT), 5(1):92–128, 2005.
  2. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  3. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  4. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  9650–9660, 2021.
  5. Context autoencoder for self-supervised representation learning. International Journal of Computer Vision, pp.  1–16, 2023.
  6. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.
  7. Graph transfer learning via adversarial domain adaptation with graph convolution. IEEE Transactions on Knowledge and Data Engineering, 35(5):4908–4922, 2022.
  8. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.
  9. Bootstrapped masked autoencoders for vision bert pretraining. In European Conference on Computer Vision, pp.  247–264. Springer, 2022.
  10. Predict then propagate: Graph neural networks meet personalized pagerank. International Conference on Learning Representations, 2018.
  11. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271–21284, 2020.
  12. Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint arXiv:2305.15066, 2023.
  13. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  16000–16009, 2022.
  14. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654, 2020.
  15. Harnessing explanations: Llm-to-lm interpreter for enhanced text-attributed graph representation learning. International Conference on Learning Representations, 2024.
  16. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp.  594–604, 2022.
  17. Graphmae2: A decoding-enhanced masked self-supervised graph learner. In Proceedings of the ACM Web Conference 2023, pp.  737–746, 2023.
  18. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021a.
  19. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118–22133, 2020.
  20. Ogb-lsc: A large-scale challenge for machine learning on graphs. NeurIPS, 34, 2021b.
  21. Prodigy: Enabling in-context learning over graphs. NeurIPS, 2023.
  22. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pp.  4171–4186, 2019.
  23. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2016a.
  24. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308, 2016b.
  25. Evaluating large language models on graphs: Performance insights and comparative analysis. arXiv preprint arXiv:2308.11224, 2023.
  26. One for all: Towards training one graph model for all classification tasks. arXiv preprint arXiv:2310.00149, 2023a.
  27. Pre-training molecular graph representation with 3d geometry. In International Conference on Learning Representations, 2021.
  28. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In Proceedings of the ACM Web Conference 2023, pp.  417–428, 2023b.
  29. Personalized pagerank estimation and search: A bidirectional approach. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, pp.  163–172, 2016.
  30. Wiki-cs: A wikipedia-based benchmark for graph neural networks. arXiv preprint arXiv:2007.02901, 2020.
  31. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
  32. Newman, M. Networks. Oxford university press, 2018.
  33. Gcc: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp.  1150–1160, 2020.
  34. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.  8748–8763. PMLR, 2021.
  35. All in one: Multi-task prompting for graph neural networks. 2023.
  36. Graphgpt: Graph instruction tuning for large language models. arXiv preprint arXiv:2310.13023, 2023.
  37. Bootstrapped representation learning on graphs. In ICLR 2021 Workshop on Geometrical and Topological Representation Learning, 2021.
  38. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.
  39. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  40. Graph attention networks. In International Conference on Learning Representations, 2018a.
  41. Deep graph infomax. In International Conference on Learning Representations, 2018b.
  42. Can language models solve graph problems in natural language? arXiv preprint arXiv:2305.10037, 2023.
  43. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  3733–3742, 2018.
  44. Natural language is all a graph needs. arXiv preprint arXiv:2308.07134, 2023.
  45. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Yufei He (20 papers)
  2. Bryan Hooi (158 papers)
  3. Yuan Sui (14 papers)
  4. Xiaoxin He (14 papers)
Citations (6)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets