Papers
Topics
Authors
Recent
2000 character limit reached

Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspective (2305.17760v6)

Published 28 May 2023 in cs.CL and cs.LG

Abstract: How do LLMs "think"? This paper formulates a probabilistic cognitive model called the bounded pragmatic speaker, which can characterize the operation of different variations of LLMs. Specifically, we demonstrate that LLMs fine-tuned with reinforcement learning from human feedback (Ouyang et al., 2022) embody a model of thought that conceptually resembles a fast-and-slow model (Kahneman, 2011), which psychologists have attributed to humans. We discuss the limitations of reinforcement learning from human feedback as a fast-and-slow model of thought and propose avenues for expanding this framework. In essence, our research highlights the value of adopting a cognitive probabilistic modeling approach to gain insights into the comprehension, evaluation, and advancement of LLMs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
  2. Andreas, J. Language models as agent models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp.  5769–5779, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
  3. Reasoning about pragmatics with neural listeners and speakers. arXiv preprint arXiv:1604.00562, 2016.
  4. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
  5. Does the autistic child have a “theory of mind”? Cognition, 21(1):37–46, 1985.
  6. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  7. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  8. Innovative bert-based reranking language models for speech recognition. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp.  266–271. IEEE, 2021.
  9. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  10. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
  11. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  12. Modular networks for compositional instruction following. arXiv preprint arXiv:2010.12764, 2020.
  13. A survey for in-context learning. arXiv preprint arXiv:2301.00234, 2022.
  14. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998, 2012.
  15. Unified pragmatic models for generating and following instructions. arXiv preprint arXiv:1711.04987, 2017.
  16. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574–9586, 2021.
  17. Amortized inference in probabilistic reasoning. In Proceedings of the annual meeting of the cognitive science society, volume 36, 2014.
  18. Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences, 20(11):818–829, 2016.
  19. Children’s understanding of representational change and its relation to the understanding of false belief and the appearance-reality distinction. Child development, pp.  26–37, 1988.
  20. Probabilistic models of cognition: Exploring representations and inductive biases. Trends in cognitive sciences, 14(8):357–364, 2010.
  21. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023.
  22. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
  23. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019.
  24. Kahneman, D. Thinking, fast and slow. macmillan, 2011.
  25. Rl with kl penalties is better viewed as bayesian inference. arXiv preprint arXiv:2205.11275, 2022.
  26. Passive learning of active causal strategies in agents and language models. arXiv preprint arXiv:2305.16183, 2023.
  27. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07182, 2016.
  28. Levine, S. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909, 2018.
  29. Lewis, D. K. Convention: A Philosophical Study. Cambridge, MA, USA: Wiley-Blackwell, 1969.
  30. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097, 2022.
  31. Neurologic a* esque decoding: Constrained text generation with lookahead heuristics. arXiv preprint arXiv:2112.08726, 2021.
  32. Dissociating language and thought in large language models: a cognitive perspective. arXiv preprint arXiv:2301.06627, 2023.
  33. Interactive learning from activity description. In International Conference on Machine Learning, pp.  8096–8108. PMLR, 2021.
  34. Lever: Learning to verify language-to-code generation with execution. arXiv preprint arXiv:2302.08468, 2023.
  35. OpenAI. Chatgpt. https://openai.com/blog/chatgpt, 2022.
  36. OpenAI. Gpt-4 technical report. 2023.
  37. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  38. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023.
  39. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4):515–526, 1978.
  40. Bayesian brains without probabilities. Trends in cognitive sciences, 20(12):883–893, 2016.
  41. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022.
  42. Simon, H. A. Models of man; social and rational. 1957.
  43. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020.
  44. How to talk so ai will learn: Instructions, descriptions, and autonomy. Advances in Neural Information Processing Systems, 35:34762–34775, 2022.
  45. How to grow a mind: Statistics, structure, and abstraction. science, 331(6022):1279–1285, 2011.
  46. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  47. One and done? optimal decisions from very few samples. Cognitive science, 38(4):599–637, 2014.
  48. Voyager: An open-ended embodied agent with large language models. 2023.
  49. Calibrate your listeners! robust communication-based training for pragmatic speakers. arXiv preprint arXiv:2110.05422, 2021.
  50. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
  51. Learning to refer informatively by amortizing pragmatic reasoning. arXiv preprint arXiv:2006.00418, 2020.
  52. Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children’s understanding of deception. Cognition, 13(1):103–128, 1983.
  53. From word models to world models: Translating from natural language to the probabilistic language of thought. 2023.
  54. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023.
  55. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022a.
  56. Coder reviewer reranking for code generation. arXiv preprint arXiv:2211.16490, 2022b.
  57. Define, evaluate, and improve task-oriented cognitive capabilities for instruction generation models. arXiv preprint arXiv:2301.05149, 2023a.
  58. Large language models as commonsense knowledge for large-scale task planning. arXiv preprint arXiv:2305.14078, 2023b.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Paper to Video (Beta)

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Authors (1)

  1. Khanh Nguyen

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up