Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Eliciting Problem Specifications via Large Language Models (2405.12147v2)

Published 20 May 2024 in cs.AI and cs.CL

Abstract: Cognitive systems generally require a human to translate a problem definition into some specification that the cognitive system can use to attempt to solve the problem or perform the task. In this paper, we illustrate that LLMs can be utilized to map a problem class, defined in natural language, into a semi-formal specification that can then be utilized by an existing reasoning and learning system to solve instances from the problem class. We present the design of LLM-enabled cognitive task analyst agent(s). Implemented with LLM agents, this system produces a definition of problem spaces for tasks specified in natural language. LLM prompts are derived from the definition of problem spaces in the AI literature and general problem-solving strategies (Polya's How to Solve It). A cognitive system can then use the problem-space specification, applying domain-general problem solving strategies ("weak methods" such as search), to solve multiple instances of problems from the problem class. This result, while preliminary, suggests the potential for speeding cognitive systems research via disintermediation of problem formulation while also retaining core capabilities of cognitive systems, such as robust inference and online learning.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Building Ontologies with Basic Formal Ontology. MIT Press.
  2. Assanie, M. (2022). Learning General and Correct Procedural Knowledge in a Cognitive Architecture. Thesis, University of Michigan, Ann Arbor. From http://deepblue.lib.umich.edu/handle/2027.42/174544. Accepted: 2022-09-06T16:20:06Z.
  3. Graph of Thoughts: Solving Elaborate Problems with Large Language Models. Proc. of 38th Annual AAAI Conference on Artificial Intelligence. Vancouver: arXiv. ArXiv:2308.09687 [cs].
  4. Feature selection in machine learning: A new perspective. Neurocomputing, 300, 70–79.
  5. Working Minds. Cambridge: MIT Press.
  6. Towards Next-Generation Intelligent Assistants Leveraging LLM Techniques. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 5792–5793). Long Beach CA USA: ACM.
  7. STRIPS: A new approach in the application of theorem proving to problem solving. Artificial Intelligence, 2, 189–208.
  8. PDDL2.1: An Extension to PDDL for Expressing Temporal Planning Domains. Journal of Artificial Intelligence Research, 20, 61–124.
  9. ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120, e2305016120.
  10. Green, A. (2024). What is PDDL?
  11. Large Language Models for Software Engineering: A Systematic Literature Review. ArXiv:2308.10620 [cs].
  12. The GOMS family of user interface analysis techniques: comparison and contrast. ACM Trans. Comput.-Hum. Interact., 3, 320–351.
  13. Language models can solve computer tasks. ArXiv:2303.17491 [cs].
  14. Learning General and Efficient Representations of Novel Games Through Interactive Instruction. Proc. of the 4th Conference on Advances in Cognitive Systems.
  15. Improving Knowledge Extraction from LLMs for Task Learning through Agent Analysis. Proceedings of the 38th Annual AAAI Conference on Artificial Intelligence. Vancouver: arXiv. ArXiv:2306.06770 [cs].
  16. Laird, J. E. (2012). The Soar Cognitive Architecture. Cambridge, MA: MIT Press.
  17. A universal weak method. In P. S. Rosenbloom, J. E. Laird, & A. Newell (Eds.), The Soar Papers: Research on Integrated Intelligence, volume 1, 245–292. Cambridge, MA: MIT Press.
  18. Chunking in Soar: The anatomy of a general learning mechanism. Machine Learning, 1, 11–46.
  19. Langley, P. (1994). Selection of relevant features in machine learning. Proceedings of the AAAI Fall Symposium on Relevance (pp. 140–144). New Orleans: AAAI Press.
  20. NLtoPDDL. Working Notes of the ICAPS’20 Workshop on Knowledge Engineering for Planning and Scheduling (KEPS’20). Publisher: ICAPS.
  21. Mininger, A. (2021). Expanding Task Diversity in Explanation-Based Interactive Task Learning. Ph.D. Thesis, University of Michigan, Ann Arbor.
  22. Newell, A. (1969). Heuristic Programming: Ill-structured Problems. In J. Aronofsky (Ed.), Progress in Operations Research III, 360–414. New York: Wiley.
  23. Newell, A. (1980). Reasoning, problem solving and decision processes: The problem space as a fundamental category. In R. Nickerson (Ed.), Attention and Performance VIII. Hillsdale, NJ: Erlbaum.
  24. GPS: A program that simulates human thought. In E. A. Feigenbaum & J. Feldman (Eds.), Computers and Thought. New York: McGraw-Hill.
  25. Human Problem Solving. Englewood Cliffs, NJ: Prentice-Hall.
  26. Computer Science as Empirical enquiry: Symbols and Search. Communications of the ACM, 19, 113–126.
  27. Nilsson, N. (1971). Problem solving methods in artificial intelligence. New York: McGraw-Hill.
  28. Ozkaya, I. (2023). Application of Large Language Models to Software Engineering Tasks: Opportunities, Risks, and Implications. IEEE Software, 40, 4–8. Conference Name: IEEE Software.
  29. Polya, G. (2015). How to Solve It: A New Aspect of Mathematical Method. Princeton N.J.: Princeton University Press.
  30. Tool Learning with Foundation Models. Publisher: [object Object] Version Number: 2.
  31. Learning representations by back-propagating errors. Nature, 323, 533–536. Publisher: Nature Publishing Group.
  32. Cognitive Task Analysis. New York: Psychology Press.
  33. Knowledge Engineering and Management: The CommonKADS Methodology. Cambridge: MIT Press.
  34. Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models.
  35. Reflexion: Language Agents with Verbal Reinforcement Learning. 37th Conference on Neural Information Processing Systems.
  36. Reinforcement Learning. Cambridge, MA: MIT Press.
  37. Törnberg, P. (2023). How to use LLMs for Text Analysis. ArXiv:2307.13106 [cs].
  38. On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark). ArXiv:2302.06706 [cs].
  39. A Survey on Large Language Model based Autonomous Agents. Frontiers of Computer Science, (to appear). ArXiv:2308.11432 [cs].
  40. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems, 35, 24824–24837.
  41. ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models. ArXiv:2305.18323 [cs].
  42. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. ArXiv:2305.10601 [cs].
  43. ReAct: Synergizing Reasoning and Acting in Language Models. The Eleventh International Conference on Learning Representations, ICLR 2023. Kigali, Rwanda: OpenReview.net.
  44. Generating Data for Symbolic Language with Large Language Models. 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023). Singapore.
  45. SatLM: Satisfiability-Aided Language Models Using Declarative Prompting. NeurIPS 2023. New Orleans: arXiv. ArXiv:2305.09656 [cs].
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Robert E. Wray (9 papers)
  2. James R. Kirk (8 papers)
  3. John E. Laird (15 papers)