Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Graph of Thoughts: Solving Elaborate Problems with Large Language Models (2308.09687v4)

Published 18 Aug 2023 in cs.CL, cs.AI, and cs.LG

Abstract: We introduce Graph of Thoughts (GoT): a framework that advances prompting capabilities in LLMs beyond those offered by paradigms such as Chain-of-Thought or Tree of Thoughts (ToT). The key idea and primary advantage of GoT is the ability to model the information generated by an LLM as an arbitrary graph, where units of information ("LLM thoughts") are vertices, and edges correspond to dependencies between these vertices. This approach enables combining arbitrary LLM thoughts into synergistic outcomes, distilling the essence of whole networks of thoughts, or enhancing thoughts using feedback loops. We illustrate that GoT offers advantages over state of the art on different tasks, for example increasing the quality of sorting by 62% over ToT, while simultaneously reducing costs by >31%. We ensure that GoT is extensible with new thought transformations and thus can be used to spearhead new prompting schemes. This work brings the LLM reasoning closer to human thinking or brain mechanisms such as recurrence, both of which form complex networks.

PDF HTML Abstract

The paper "Graph of Thoughts: Solving Elaborate Problems with LLMs" introduces the Graph of Thoughts (GoT) framework, a novel approach aimed at enhancing the problem-solving capabilities of LLMs through advanced prompting techniques that surpass contemporary paradigms such as Chain-of-Thought (CoT) and Tree of Thoughts (ToT).

Introduction to GoT

Prompt engineering is critical in leveraging the potential of LLMs without modifying their internal architectures. This method involves designing prompts that effectively convey the desired task to the LLM, facilitating useful outputs. However, creating effective prompts is challenging. GoT aims to address this issue by transforming the reasoning process of LLMs into a graph structure, enhancing their ability to solve complex problems.

Framework and Mechanisms

  • Graph Structure Representation: In GoT, the reasoning process is modeled as a graph where vertices represent individual thoughts (intermediate steps towards a solution), and edges denote dependencies or logical flows between these thoughts. This model allows for intricate interactions beyond linear or tree-based structures.
  • Thought Transformations: GoT introduces operations for thought transformations — aggregation, refinement, and generation:
    • Aggregation: Combines multiple thoughts to create a unified, synergistic outcome.
    • Refinement: Enhances thoughts by refining based on interconnected information.
    • Generation: Produces new thoughts from existing ones, enriching the reasoning graph.
  • Scoring and Ranking Mechanisms: Part of GoT’s framework includes evaluating thoughts by scoring and ranking them to select the most promising solutions from a pool of generated thoughts.

Practical Implications and Performance

  • Mimicking Human Cognitive Processes: The graph-based reasoning approach of GoT aligns more closely with human cognitive mechanisms and complex networks found in brain structures.
  • Performance Evaluation: The framework has been tested across various tasks, including sorting and set operations, showing significant improvements. For instance, GoT increased sorting quality by 62% over ToT and reduced costs by over 31%.
  • New Evaluation Metric: The paper introduces the concept of "volume of a thought" as a metric to quantify the breadth of information encapsulated by a thought, offering a novel perspective on evaluating prompting strategies. GoT achieves a balance of maintaining low latency while ensuring a high volume of contributing thoughts.

Theoretical Insights and Future Directions

  • Alignment with Cognitive Structures: GoT's design promotes a deeper alignment with human-like reasoning processes, which could lead to advancements in making LLMs think more humanly.
  • Graph Theory and AI: The successful integration of graph structures in GoT suggests potential for further exploration at the intersection of graph theory and artificial intelligence, potentially driving future breakthroughs in enhancing machine reasoning capabilities.

Conclusion

The Graph of Thoughts (GoT) framework marks a significant leap in the field of prompt engineering for LLMs. By adopting a graph-based approach to model reasoning processes, it encapsulates complex thought patterns similar to human cognition, offering significant improvements in both the efficiency and efficacy of problem-solving. This work sets a new standard in the domain, paving the way for the development of more sophisticated prompting techniques and further research into the symbiosis between artificial intelligence and graph theory.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (84)
  1. Practice of Streaming Processing of Dynamic Graphs: Concepts, Models, and Systems. IEEE Transactions on Parallel and Distributed Systems, 34(6): 1860–1876.
  2. GDI: A Graph Database Interface Standard. https://github.com/spcl/GDI-RMA. Accessed: 2023-09-05.
  3. The Graph Database Interface: Scaling Online Transactional and Analytical Graph Workloads to Hundreds of Thousands of Cores. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’23. ACM.
  4. Demystifying Graph Databases: Analysis and Taxonomy of Data Organization, System Designs, and Graph Queries. ACM Comput. Surv., 56(2).
  5. Motif Prediction with Graph Neural Networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, 35–45.
  6. Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis. arXiv:2205.09702.
  7. Neural Graph Databases. In Proceedings of the First Learning on Graphs Conference, volume 198 of Proceedings of Machine Learning Research, 31:1–31:38. PMLR.
  8. SISA: Set-Centric Instruction Set Architecture for Graph Mining on Processing-in-Memory Systems. In Proceedings of the 54th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO ’21, 282–297.
  9. Communication-Efficient Jaccard Similarity for High-Performance Distributed Genome Comparisons. In Proceedings of the IEEE International Parallel and Distributed Processing Symposium, IPDPS ’20, 1122–1132.
  10. ProbGraph: High-Performance and High-Accuracy Graph Mining with Probabilistic Set Representations. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC ’22. IEEE.
  11. GraphMineSuite: Enabling High-Performance and Programmable Graph Mining Algorithms with Set Algebra. Proc. VLDB Endow., 14(11): 1922–1935.
  12. Geometric Deep Learning: Going beyond Euclidean data. IEEE Signal Processing Magazine, 34(4): 18–42.
  13. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems (NeurIPS ’20), volume 33, 1877–1901. Curran Associates.
  14. Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv:2303.12712.
  15. Graph Mining: Laws, Generators, and Algorithms. ACM Comput. Surv., 38(1).
  16. Machine Learning on Graphs: A Model and Comprehensive Taxonomy. arXiv:2005.03675.
  17. Teaching Large Language Models to Self-Debug. arXiv:2304.05128.
  18. Fast Graph Pattern Matching. In Proceedings of the IEEE 24th International Conference on Data Engineering, ICDE ’08, 913–922.
  19. PaLM: Scaling Language Modeling with Pathways. arXiv:2204.02311.
  20. Mining Graph Data. John Wiley & Sons.
  21. Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning. arXiv:2205.09712.
  22. Low-Latency Graph Streaming Using Compressed Purely-Functional Trees. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’19, 918–934.
  23. Language Model Cascades. In Beyond Bayes: Paths Towards Universal Reasoning Systems, Workshop at ICML ’22.
  24. A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. Proceedings of the National Academy of Sciences, 119(32): e2123433119.
  25. Graph Pattern Matching: From Intractable to Polynomial Time. Proc. VLDB Endow., 3(1–2): 264–275.
  26. DISTINGER: A distributed graph data structure for massive dynamic graph processing. In Proccedings of the IEEE International Conference on Big Data, Big Data ’15, 1814–1822.
  27. Triangles to Capture Social Cohesion. In Proceedings of the IEEE Third International Conference on Privacy, Security, Risk and Trust and IEEE Third International Conference on Social Computing, PASSAT/SocialCom ’11, 258–265.
  28. Friston, K. 2008. Hierarchical Models in the Brain. PLOS Computational Biology, 4(11): 1–24.
  29. Complexity-Based Prompting for Multi-Step Reasoning. arXiv:2210.00720.
  30. Learning Combinatorial Node Labeling Algorithms. arXiv:2106.03594.
  31. Lifting Sequential Graph Algorithms for Distributed-Memory Parallel Computation. SIGPLAN Not., 40(10): 423–437.
  32. The Parallel BGL: A generic library for distributed graph computations. Parallel Object-Oriented Scientific Computing (POOSC).
  33. Representation Learning on Graphs: Methods and Applications. Bulletin of the Technical Committee on Data Engineering, 40(3): 52–74.
  34. A survey on improving NLP models with human explanations. In Proceedings of the First Workshop on Learning with Natural Language Supervision, 40–47. Association for Computational Linguistics.
  35. Cyclic Pattern Kernels for Predictive Graph Mining. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’04, 158–167.
  36. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, 9118–9147. PMLR.
  37. Inner Monologue: Embodied Reasoning through Planning with Language Models. arXiv:2207.05608.
  38. A survey of frequent subgraph mining algorithms. The Knowledge Engineering Review, 28(1): 75–105.
  39. Language Models can Solve Computer Tasks. arXiv:2303.17491.
  40. Explanation-Based Human Debugging of NLP Models: A Survey. Transactions of the Association for Computational Linguistics, 9: 1508–1528.
  41. The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’21, 3045–3059. Association for Computational Linguistics.
  42. Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv:2101.00190.
  43. Long, J. 2023. Large Language Model Guided Tree-of-Thought. arXiv:2305.08291.
  44. Challenges in Parallel Graph Processing. Parallel Processing Letters, 17(1): 5–20.
  45. Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651.
  46. Pregel: A System for Large-Scale Graph Processing. In Proceedings of the International Conference on Management of Data, SIGMOD ’10, 135–146. ACM.
  47. Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding. arXiv:2307.15337.
  48. Show Your Work: Scratchpads for Intermediate Computation with Language Models. arXiv:2112.00114.
  49. REFINER: Reasoning Feedback on Intermediate Representations. arXiv:2304.01904.
  50. Shaping Communities out of Triangles. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management, CIKM ’12, 1677–1681.
  51. Reasoning with Language Model Prompting: A Survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, ACL ’23, 5368–5393. Association for Computational Linguistics.
  52. qrdlgit. 2023. graph-of-thoughts Repository. https://github.com/qrdlgit/graph-of-thoughts. Accessed: 2023-10-11.
  53. Improving Language Understanding by Generative Pre-Training. https://openai.com/research/language-unsupervised. Accessed: 2023-09-06.
  54. Language Models are Unsupervised Multitask Learners. https://openai.com/research/better-language-models. Accessed: 2023-09-06.
  55. Graph Databases: New Opportunities for Connected Data. O’Reilly Media, 2nd edition.
  56. The Future is Big Graphs: A Community View on Graph Processing Systems. Commun. ACM, 64(9): 62–71.
  57. The Graph Neural Network Model. IEEE Transactions on Neural Networks, 20(1): 61–80.
  58. Schaeffer, S. E. 2007. Graph clustering. Computer Science Review, 1(1): 27–64.
  59. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. arXiv:2010.15980.
  60. Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366.
  61. Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data. arXiv:2302.12822.
  62. Arabesque: A System for Distributed Graph Mining. In Proceedings of the 25th Symposium on Operating Systems Principles, SOSP ’15, 425–440. ACM.
  63. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.
  64. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288.
  65. Attention is All you Need. In Advances in Neural Information Processing Systems (NIPS ’17), volume 30. Curran Associates.
  66. Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, ACL ’23, 2609–2634. Association for Computational Linguistics.
  67. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In Proceedings of the Eleventh International Conference on Learning Representations, ICLR ’23.
  68. Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents. arXiv:2302.01560.
  69. Interactive Natural Language Processing. arXiv:2305.13246.
  70. Putting Humans in the Natural Language Processing Loop: A Survey. In Proceedings of the First Workshop on Bridging Human-Computer Interaction and Natural Language Processing, 47–52. Association for Computational Linguistics.
  71. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903.
  72. PromptChainer: Chaining Large Language Model Prompts through Visual Programming. In Extended Abstracts of the Conference on Human Factors in Computing Systems, CHI EA ’22. ACM.
  73. AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In Proceedings of the Conference on Human Factors in Computing Systems, CHI ’22. ACM.
  74. A Comprehensive Survey on Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1): 4–24.
  75. Decomposition Enhances Reasoning via Self-Evaluation Guided Decoding. arXiv:2305.00633.
  76. Foundation Models for Decision Making: Problems, Methods, and Opportunities. arXiv:2303.04129.
  77. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv:2305.10601.
  78. ReAct: Synergizing Reasoning and Acting in Language Models. In Proceedings of the Eleventh International Conference on Learning Representations, ICLR ’23.
  79. Beyond Chain-of-Thought, Effective Graph-of-Thought Reasoning in Large Language Models. arXiv:2305.16582.
  80. STaR: Bootstrapping Reasoning With Reasoning. In Advances in Neural Information Processing Systems (NeurIPS ’22), volume 35, 15476–15488. Curran Associates.
  81. Planning with Large Language Models for Code Generation. In Proceedings of the Eleventh International Conference on Learning Representations, ICLR ’23.
  82. Deep Learning on Graphs: A Survey. IEEE Transactions on Knowledge and Data Engineering, 34(1): 249–270.
  83. Graph neural networks: A review of methods and applications. AI Open, 1: 57–81.
  84. Large Language Models Are Human-Level Prompt Engineers. arXiv:2211.01910.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (11)
  1. Maciej Besta (66 papers)
  2. Nils Blach (10 papers)
  3. Ales Kubicek (9 papers)
  4. Robert Gerstenberger (12 papers)
  5. Lukas Gianinazzi (23 papers)
  6. Joanna Gajda (2 papers)
  7. Tomasz Lehmann (2 papers)
  8. Michal Podstawski (10 papers)
  9. Hubert Niewiadomski (9 papers)
  10. Piotr Nyczyk (7 papers)
  11. Torsten Hoefler (203 papers)
Citations (399)
View on Semantic Scholar
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube