Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Disentangled Representation Learning with Large Language Models for Text-Attributed Graphs (2310.18152v4)

Published 27 Oct 2023 in cs.CL and cs.LG

Abstract: Text-attributed graphs (TAGs) are prevalent on the web and research over TAGs such as citation networks, e-commerce networks and social networks has attracted considerable attention in the web community. Recently, LLMs have demonstrated exceptional capabilities across a wide range of tasks. However, the existing works focus on harnessing the potential of LLMs solely relying on prompts to convey graph structure information to LLMs, thus suffering from insufficient understanding of the complex structural relationships within TAGs. To address this problem, in this paper we present the Disentangled Graph-Text Learner (DGTL) model, which is able to enhance the reasoning and predicting capabilities of LLMs for TAGs. Our proposed DGTL model incorporates graph structure information through tailored disentangled graph neural network (GNN) layers, enabling LLMs to capture the intricate relationships hidden in text-attributed graphs from multiple structural factors. Furthermore, DGTL operates with frozen pre-trained LLMs, reducing computational costs and allowing much more flexibility in combining with different LLM models. Experimental evaluations demonstrate the effectiveness of the proposed DGTL model on achieving superior or comparable performance over state-of-the-art baselines. Additionally, we also demonstrate that our DGTL model can offer natural language explanations for predictions, thereby significantly enhancing model interpretability.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  2. Exploring the potential of large language models (llms) in learning on graphs. arXiv preprint arXiv:2307.03393, 2023.
  3. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220–235, 2023.
  4. Simteg: A frustratingly simple approach improves textual graph learning. arXiv preprint arXiv:2308.02565, 2023.
  5. S3: Social-network simulation system with large language model-empowered agents. arXiv preprint arXiv:2307.14984, 2023.
  6. Neural message passing for quantum chemistry. In International conference on machine learning, pp.  1263–1272, 2017.
  7. Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint arXiv:2305.15066, 2023.
  8. Inductive representation learning on large graphs. In Advances in neural information processing systems, 2017.
  9. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654, 2020.
  10. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. arXiv preprint arXiv:2402.07630, 2024.
  11. Open graph benchmark: Datasets for machine learning on graphs. In Advances in neural information processing systems, 2020.
  12. Prompt-based node feature extractor for few-shot learning on text-attributed graphs. arXiv preprint arXiv:2309.02848, 2023.
  13. Semi-supervised classification with graph convolutional networks. In Proceedings of the 5th International Conference on Learning Representations, 2017.
  14. Evaluating large language models on graphs: Performance insights and comparative analysis. arXiv preprint arXiv:2308.11224, 2023.
  15. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
  16. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp.  2383–2392, 2016.
  17. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp.  3982–3992, 2019.
  18. Collective classification in network data. AI magazine, 29(3):93–93, 2008.
  19. sktsherlock. Tag-benchmark. Online, Oct 2023. URL https://github.com/sktsherlock/TAG-Benchmark.
  20. Large language models in medicine. Nature medicine, 29(8):1930–1940, 2023.
  21. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.
  22. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  23. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  24. Graph attention networks. In Proceedings of the 6th International Conference on Learning Representations, 2018.
  25. Can language models solve graph problems in natural language? arXiv preprint arXiv:2305.10037, 2023.
  26. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022.
  27. Community preserving network embedding. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
  28. Heterogeneous graph attention network. In The world wide web conference, pp.  2022–2032, 2019.
  29. A survey on large language models for recommendation. arXiv preprint arXiv:2305.19860, 2023.
  30. How powerful are graph neural networks? In 7th International Conference on Learning Representations, 2019.
  31. Bert-enhanced text graph neural network for classification. Entropy, 23(11):1536, 2021.
  32. Natural language is all a graph needs. arXiv preprint arXiv:2308.07134, 2023.
  33. Graph-bert: Only attention is needed for learning graph representations. arXiv preprint arXiv:2001.05140, 2020.
  34. Understanding bag-of-words model: a statistical framework. International journal of machine learning and cybernetics, 1:43–52, 2010.
  35. Large graph models: A perspective. arXiv preprint arXiv:2308.14522, 2023.
  36. Learning on large-scale text-attributed graphs via variational inference. In The Eleventh International Conference on Learning Representations, 2022.
  37. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Yijian Qin (9 papers)
  2. Xin Wang (1306 papers)
  3. Ziwei Zhang (40 papers)
  4. Wenwu Zhu (104 papers)
Citations (25)
View on Semantic Scholar