Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Thought Propagation: An Analogical Approach to Complex Reasoning with Large Language Models (2310.03965v3)

Published 6 Oct 2023 in cs.AI and cs.CL
Thought Propagation: An Analogical Approach to Complex Reasoning with Large Language Models

Abstract: LLMs have achieved remarkable success in reasoning tasks with the development of prompting methods. However, existing prompting approaches cannot reuse insights of solving similar problems and suffer from accumulated errors in multi-step reasoning, since they prompt LLMs to reason \textit{from scratch}. To address these issues, we propose \textbf{\textit{Thought Propagation} (TP)}, which explores the analogous problems and leverages their solutions to enhance the complex reasoning ability of LLMs. These analogous problems are related to the input one, with reusable solutions and problem-solving strategies. Thus, it is promising to propagate insights of solving previous analogous problems to inspire new problem-solving. To achieve this, TP first prompts LLMs to propose and solve a set of analogous problems that are related to the input one. Then, TP reuses the results of analogous problems to directly yield a new solution or derive a knowledge-intensive plan for execution to amend the initial solution obtained from scratch. TP is compatible with existing prompting approaches, allowing plug-and-play generalization and enhancement in a wide range of tasks without much labor in task-specific prompt engineering. Experiments across three challenging tasks demonstrate TP enjoys a substantial improvement over the baselines by an average of 12\% absolute increase in finding the optimal solutions in Shortest-path Reasoning, 13\% improvement of human preference in Creative Writing, and 15\% enhancement in the task completion rate of LLM-Agent Planning.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  2. Retrieval-based language models and applications. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts), pp.  41–46, 2023.
  3. Paul Bartha. Analogy and analogical reasoning. 2013.
  4. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687, 2023.
  5. Analogy generation by prompting large language models: A case study of instructgpt. arXiv preprint arXiv:2210.04186, 2022.
  6. Cam: A large language model-based creative analogy mining framework. In Proceedings of the ACM Web Conference 2023, pp.  3903–3914, 2023.
  7. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp.  2206–2240. PMLR, 2022.
  8. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  9. Large language models as tool makers. arXiv preprint arXiv:2305.17126, 2023.
  10. Jaime Guillermo Carbonell. Derivational analogy: A theory of reconstructive problem solving and expertise acquisition. Carnegie-Mellon University, Department of Computer Science, 1985.
  11. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201, 2023.
  12. E-kar: A benchmark for rationalizing natural language analogical reasoning. arXiv preprint arXiv:2203.08480, 2022.
  13. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023a.
  14. Exploring the potential of large language models (llms) in learning on graphs. arXiv preprint arXiv:2307.03393, 2023b.
  15. Textworld: A learning environment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Conjunction with the 27th International Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers 7, pp.  41–75. Springer, 2019.
  16. Faith and fate: Limits of transformers on compositionality. arXiv preprint arXiv:2305.18654, 2023.
  17. Neural message passing for quantum chemistry. In International conference on machine learning, pp.  1263–1272. PMLR, 2017.
  18. Rogers P Hall. Computational approaches to analogical reasoning: A comparative analysis. Artificial intelligence, 39(1):39–120, 1989.
  19. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
  20. Explanations as features: Llm-based features for text-attributed graphs. arXiv preprint arXiv:2305.19523, 2023.
  21. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299, 2022.
  22. Structgpt: A general framework for large language model to reason over structured data. arXiv preprint arXiv:2305.09645, 2023.
  23. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022.
  24. Semi-supervised classification with graph convolutional networks. In The International Conference on Representation Learning, 2017.
  25. Copy is all you need. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=CROlOA9Nd8C.
  26. Camel: Communicative agents for” mind” exploration of large scale language model society. arXiv preprint arXiv:2303.17760, 2023.
  27. Chain of hindsight aligns language models with feedback. arXiv preprint arXiv:2302.02676, 3, 2023a.
  28. Generated knowledge prompting for commonsense reasoning. arXiv preprint arXiv:2110.08387, 2021.
  29. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35, 2023b.
  30. Mind’s eye: Grounded language model reasoning through simulation. arXiv preprint arXiv:2210.05359, 2022.
  31. Deep learning methods for abstract visual reasoning: A survey on raven’s progressive matrices. arXiv preprint arXiv:2201.12382, 2022.
  32. Augmented language models: a survey. arXiv preprint arXiv:2302.07842, 2023.
  33. OpenAI. https://platform.openai.com/docs/models/gpt-3-5. 2022.
  34. OpenAI. Gpt-4 technical report, 2023.
  35. Joseph Priestley. “The” History and Present State of Electricity: With Original Experiments. C. Bathurst and T. Lowndes, in Fleet-Street, J. Rivington and J. Johnson, in …, 1775.
  36. Can large language models empower molecular property prediction? arXiv preprint arXiv:2307.07443, 2023.
  37. Reasoning with language model prompting: A survey. arXiv preprint arXiv:2212.09597, 2022.
  38. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
  39. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652, 2023.
  40. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.
  41. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10740–10749, 2020.
  42. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021.
  43. Life is a circus and we are the clowns: Automatically finding analogies between situations and processes. arXiv preprint arXiv:2210.12197, 2022.
  44. Can language models solve graph problems in natural language? arXiv preprint arXiv:2305.10037, 2023a.
  45. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432, 2023b.
  46. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022a.
  47. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966, 2023c.
  48. Retrieval-based controllable molecule generation. arXiv preprint arXiv:2208.11126, 2022b.
  49. Emergent analogical reasoning in large language models. Nature Human Behaviour, pp.  1–16, 2023.
  50. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  51. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 2023.
  52. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
  53. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023.
  54. Retrieval-augmented multimodal language modeling. 2023.
  55. Analogykb: Unlocking analogical reasoning of language models with a million-scale knowledge base. arXiv preprint arXiv:2305.05994, 2023.
  56. Multimodal analogical reasoning over knowledge graphs. arXiv preprint arXiv:2210.00312, 2022a.
  57. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022b.
  58. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
  59. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022.
  60. Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory. arXiv preprint arXiv:2305.17144, 2023a.
  61. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107, 2023b.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Junchi Yu (12 papers)
  2. Ran He (172 papers)
  3. Rex Ying (90 papers)
Citations (19)
View on Semantic Scholar