2000 character limit reached
Graph Linearization Methods for Reasoning on Graphs with Large Language Models (2410.19494v3)
Published 25 Oct 2024 in cs.CL and cs.LG
Abstract: LLMs have evolved to process multiple modalities beyond text, such as images and audio, which motivates us to explore how to effectively leverage them for graph reasoning tasks. The key question, therefore, is how to transform graphs into linear sequences of tokens, a process we term "graph linearization", so that LLMs can handle graphs naturally. We consider that graphs should be linearized meaningfully to reflect certain properties of natural language text, such as local dependency and global alignment, in order to ease contemporary LLMs, trained on trillions of textual tokens, better understand graphs. To achieve this, we developed several graph linearization methods based on graph centrality and degeneracy. These methods are further enhanced using node relabeling techniques. The experimental results demonstrate the effectiveness of our methods compared to the random linearization baseline. Our work introduces novel graph representations suitable for LLMs, contributing to the potential integration of graph machine learning with the trend of multimodal processing using a unified transformer model.
- Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186, Sofia, Bulgaria. Association for Computational Linguistics.
- Graph markup language (graphml). In Roberto Tamassia, editor, Handbook of graph drawing visualization, Discrete mathematics and its applications, pages 517–541. CRC Press, Boca Raton [u.a.].
- Sergey Brin and Lawrence Page. 1998. The anatomy of a large-scale hypertextual web search engine. In Computer Networks and ISDN Systems, volume 30, pages 107–117. Elsevier.
- Exploring the potential of large language models (llms) in learning on graph. In NeurIPS 2023 Workshop: New Frontiers in Graph Learning.
- GroundHog: Dialogue generation using multi-grained linguistic input. In Proceedings of the 5th Workshop on Computational Approaches to Discourse (CODI 2024), pages 149–160, St. Julians, Malta. Association for Computational Linguistics.
- Which modality should i use–text, motif, or image?: Understanding graphs with large language models. arXiv preprint arXiv:2311.09862.
- Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
- BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
- Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5884–5888. IEEE.
- Learning structural node embeddings via diffusion wavelets. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1320–1329.
- An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
- The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
- Vijay Prakash Dwivedi and Xavier Bresson. 2020. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699.
- Talk like a graph: Encoding graphs for large language models. arXiv preprint arXiv:2310.04560.
- John R Firth. 1957. A synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis.
- Linton C Freeman. 1978. Centrality in social networks conceptual clarification. Social networks, 1(3):215–239.
- Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint arXiv:2305.15066.
- Inductive representation learning on large graphs. Advances in neural information processing systems, 30.
- Zellig S Harris. 1954. Distributional structure. Word, 10(2-3):146–162.
- Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
- Michael Himsolt. 1997. Gml: Graph modelling language. University of Passau.
- Promoting graph awareness in linearized graph-to-text generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 944–956, Online. Association for Computational Linguistics.
- Beyond text: A deep dive into large language models’ ability on understanding graph data. arXiv preprint arXiv:2310.04944.
- Can llms effectively leverage graph structural information through prompts, and why? Transactions on Machine Learning Research.
- Martin Joos. 1950. Description of language design. The Journal of the Acoustical Society of America, 22(6):701–707.
- Pure transformers are powerful graph learners. Advances in Neural Information Processing Systems, 35:14582–14595.
- BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.
- Evaluating large language models on graphs: Performance insights and comparative analysis. arXiv preprint arXiv:2308.11224.
- Graphinstruct: Empowering large language models with graph understanding and reasoning capability. arXiv preprint arXiv:2403.04483.
- The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab.
- Terry Patten. 1993. Book reviews: Text generation and systemic-functional linguistics: Experiences from English and Japanese. Computational Linguistics, 19(1).
- Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Leveraging discourse structure for extractive meeting summarization. Preprint, arXiv:2405.11055.
- Investigating pretrained language models for graph-to-text generation. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 211–227, Online. Association for Computational Linguistics.
- Stephen B Seidman. 1983. Network structure and minimum degree. Social networks, 5(3):269–287.
- Graph reasoning with large language models via pseudo-code prompting. arXiv preprint arXiv:2409.17906.
- Attention is all you need. Advances in neural information processing systems, 30.
- Can language models solve graph problems in natural language? Advances in Neural Information Processing Systems, 36.
- Grapheval2000: Benchmarking and improving large language models on graph datasets. arXiv preprint arXiv:2406.16176.
- Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence.
- Exploring the potential of large language models in graph generation. arXiv preprint arXiv:2403.14358.
- Natural language is all a graph needs. arXiv preprint arXiv:2308.07134.
- A survey on multimodal large language models. arXiv preprint arXiv:2306.13549.
- Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34:28877–28888.
- Gracore: Benchmarking graph comprehension and complex reasoning in large language models. arXiv preprint arXiv:2407.02936.
- Llm4dyg: Can large language models solve problems on dynamic graphs? arXiv preprint arXiv:2310.17110.
- Graph meets llms: Towards large graph models. In NeurIPS 2023 Workshop: New Frontiers in Graph Learning.
- Graphtext: Graph reasoning in text space. arXiv preprint arXiv:2310.01089.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
