Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

UPAR: A Kantian-Inspired Prompting Framework for Enhancing Large Language Model Capabilities (2310.01441v2)

Published 30 Sep 2023 in cs.CL and cs.AI

Abstract: LLMs have demonstrated impressive inferential capabilities, with numerous research endeavors devoted to enhancing this capacity through prompting. Despite these efforts, a unified epistemological foundation is still conspicuously absent. Drawing inspiration from Kant's a priori philosophy, we propose the UPAR prompting framework, designed to emulate the structure of human cognition within LLMs. The UPAR framework is delineated into four phases: "Understand", "Plan", "Act", and "Reflect", enabling the extraction of structured information from complex contexts, prior planning of solutions, execution according to plan, and self-reflection. This structure significantly augments the explainability and accuracy of LLM inference, producing a human-understandable and inspectable inferential trajectory. Furthermore, our work offers an epistemological foundation for existing prompting techniques, allowing for a possible systematic integration of these methods. With GPT-4, our approach elevates the accuracy from COT baseline of 22.92% to 58.33% in a challenging subset of GSM8K, and from 67.91% to 75.40% in the causal judgment task. Without using few-shot examples or external tools, UPAR significantly outperforms existing prompting methods on SCIBENCH, a challenging dataset containing collegiate-level mathematics, chemistry, and physics scientific problems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (81)
  1. Henry E Allison. Kant’s transcendental idealism. Yale University Press, 2004.
  2. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023, 2023.
  3. Graph of thoughts: Solving elaborate problems with large language models, 2023.
  4. Graham Bird. The revolutionary Kant: A commentary on the critique of pure reason. Open Court, 2013.
  5. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  6. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
  7. Large language models as tool makers, 2023.
  8. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, 2022.
  9. Factool: Factuality detection in generative ai – a tool augmented framework for multi-task and multi-domain scenarios, 2023.
  10. Dola: Decoding by contrasting layers improves factuality in large language models, 2023.
  11. Training verifiers to solve math word problems, 2021.
  12. Selection-inference: Exploiting large language models for interpretable logical reasoning, 2022.
  13. Language model cascades, 2022.
  14. A survey for in-context learning. arXiv preprint arXiv:2301.00234, 2022.
  15. A survey on in-context learning, 2023.
  16. Faith and fate: Limits of transformers on compositionality, 2023.
  17. Complexity-based prompting for multi-step reasoning, 2023.
  18. Pal: Program-aided language models, 2023a.
  19. Enabling large language models to generate text with citations, 2023b.
  20. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361, 2021. doi: 10.1162/tacl˙a˙00370. URL https://aclanthology.org/2021.tacl-1.21.
  21. History of western philosophy: Collectors edition. Routledge, 2013.
  22. Georg Wilhelm Fredrich Hegel. Georg Wilhelm Friedrich Hegel: the science of logic. Cambridge University Press, 2010.
  23. Investigating causal understanding in llms. In NeurIPS ML Safety Workshop, 2022.
  24. David Hume. The Philosophical Works of David Hume: In Four Volumes. Essays moral, political, and literary; 1, volume 3. Longmans, Green, 1875.
  25. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, mar 2023. doi: 10.1145/3571730. URL https://doi.org/10.1145%2F3571730.
  26. Maieutic prompting: Logically consistent reasoning with recursive explanations, 2022.
  27. Language models (mostly) know what they know, 2022.
  28. Daniel Kahneman. Thinking, fast and slow. macmillan, 2011.
  29. Critique of pure reason. JM Dent London, 1934.
  30. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pp.  22199–22213, 2022.
  31. Large language models are zero-shot reasoners, 2023.
  32. Causal reasoning and large language models: Opening a new frontier for causality, 2023.
  33. Deep learning. nature, 521(7553):436–444, 2015.
  34. Halueval: A large-scale hallucination evaluation benchmark for large language models. arXiv e-prints, pp.  arXiv–2305, 2023.
  35. Holistic evaluation of language models, 2022.
  36. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  158–167, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1015. URL https://aclanthology.org/P17-1015.
  37. Generated knowledge prompting for commonsense reasoning, 2022.
  38. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, 2021.
  39. Chameleon: Plug-and-play compositional reasoning with large language models, 2023a.
  40. A survey of deep learning for mathematical reasoning, 2023b.
  41. George F Luger. Artificial intelligence: structures and strategies for complex problem solving. Pearson education, 2005.
  42. Faithful chain-of-thought reasoning. arXiv preprint arXiv:2301.13379, 2023.
  43. Self-refine: Iterative refinement with self-feedback, 2023.
  44. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
  45. Rationalism vs. empiricism. 2004.
  46. Sources of hallucination by large language models on inference tasks, 2023.
  47. Augmented language models: a survey, 2023.
  48. Nils J Nilsson. Principles of artificial intelligence. Springer Science & Business Media, 1982.
  49. OpenAI. Gpt-4 technical report, 2023.
  50. Prompting contrastive explanations for commonsense reasoning tasks, 2021.
  51. Art: Automatic multi-step reasoning and tool-use for large language models, 2023.
  52. Reasoning with language model prompting: A survey, 2023.
  53. A survey of hallucination in large foundation models, 2023.
  54. Investigating the factual knowledge boundary of large language models with retrieval augmentation, 2023.
  55. Toolformer: Language models can teach themselves to use tools, 2023.
  56. Algorithm of thoughts: Enhancing exploration of ideas in large language models, 2023.
  57. Wilfrid Sellars et al. Empiricism and the philosophy of mind. Minnesota studies in the philosophy of science, 1(19):253–329, 1956.
  58. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023.
  59. Large language models can be easily distracted by irrelevant context, 2023.
  60. Unsupervised commonsense question answering with self-talk, 2020.
  61. The bounds of sense: An essay on Kant’s critique of pure reason. Routledge, 2018.
  62. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022.
  63. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp.  4149–4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421. URL https://aclanthology.org/N19-1421.
  64. Scibench: Evaluating college-level scientific problem-solving abilities of large language models, 2023a.
  65. Self-consistency improves chain of thought reasoning in language models, 2023b.
  66. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022.
  67. Chain-of-thought prompting elicits reasoning in large language models, 2023.
  68. Can foundation models talk causality?, 2022.
  69. From word models to world models: Translating from natural language to the probabilistic language of thought, 2023.
  70. Decomposition enhances reasoning via self-evaluation guided decoding, 2023.
  71. Large language models as optimizers, 2023a.
  72. Seqzero: Few-shot compositional semantic parsing with sequential prompts and zero-shot models, 2022.
  73. Mm-react: Prompting chatgpt for multimodal reasoning and action, 2023b.
  74. Tree of thoughts: Deliberate problem solving with large language models, 2023a.
  75. React: Synergizing reasoning and acting in language models, 2023b.
  76. Causal parrots: Large language models may talk causality but are not causal, 2023.
  77. Automatic chain of thought prompting in large language models, 2022.
  78. Explainability for large language models: A survey, 2023a.
  79. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023b.
  80. Least-to-most prompting enables complex reasoning in large language models, 2023a.
  81. Large language models are human-level prompt engineers, 2023b.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now