Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
51 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Economics Arena for Large Language Models (2401.01735v1)

Published 3 Jan 2024 in cs.GT

Abstract: LLMs have been extensively used as the backbones for general-purpose agents, and some economics literature suggest that LLMs are capable of playing various types of economics games. Following these works, to overcome the limitation of evaluating LLMs using static benchmarks, we propose to explore competitive games as an evaluation for LLMs to incorporate multi-players and dynamicise the environment. By varying the game history revealed to LLMs-based players, we find that most of LLMs are rational in that they play strategies that can increase their payoffs, but not as rational as indicated by Nash Equilibria (NEs). Moreover, when game history are available, certain types of LLMs, such as GPT-4, can converge faster to the NE strategies, which suggests higher rationality level in comparison to other models. In the meantime, certain types of LLMs can win more often when game history are available, and we argue that the winning rate reflects the reasoning ability with respect to the strategies of other players. Throughout all our experiments, we observe that the ability to strictly follow the game rules described by natural languages also vary among the LLMs we tested. In this work, we provide an economics arena for the LLMs research community as a dynamic simulation to test the above-mentioned abilities of LLMs, i.e. rationality, strategic reasoning ability, and instruction-following capability.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (68)
  1. Do as i can, not as i say: Grounding language in robotic affordances, 2022.
  2. Can we trust the evaluation on chatgpt?, 2023.
  3. Playing repeated games with large language models, 2023.
  4. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  5. Anthropic. Releasing claude instant 1.2. https://www.anthropic.com/index/releasing-claude-instant-1-2, 2023a. Accessed: 2023-09-30.
  6. Anthropic. Claude 2. https://www.anthropic.com/index/claude-2, 2023b. Accessed: 2023-09-30.
  7. Emergent autonomous scientific research capabilities of large language models. arXiv preprint arXiv:2304.05332, 2023.
  8. Dialogue management in conversational systems: a review of approaches, challenges, and opportunities. IEEE Transactions on Cognitive and Developmental Systems, 2021.
  9. Playing games with gpt: What can we learn about a large language model from canonical strategic games? Available at SSRN 4493398, 2023.
  10. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  11. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  12. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  320–335, 2022.
  13. Improving language model negotiation with self-play and in-context learning from ai feedback. arXiv preprint arXiv:2305.10142, 2023.
  14. Strategic reasoning with language models, 2023.
  15. Challenges in evaluating AI systems, 2023. URL https://www.anthropic.com/index/evaluating-ai-systems.
  16. Fulin Guo. Gpt agents in game theory experiments, 2023.
  17. Measuring massive multitask language understanding, 2021a.
  18. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b.
  19. John J Horton. Large language models as simulated economic agents: What can we learn from homo silicus? Technical report, National Bureau of Economic Research, 2023.
  20. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models, 2023.
  21. Algorithmic cooperation. Available at SSRN 4389647, 2023.
  22. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84–90, 2017.
  23. Jonathan Levin. Auction theory. Manuscript available at www. stanford. edu/jdlevin/Econ, 20286, 2004.
  24. Holistic evaluation of language models, 2022.
  25. Learning to model the world with language, 2023.
  26. Chain of hindsight aligns language models with feedback, 2023a.
  27. Training socially aligned language models in simulated human society. arXiv preprint arXiv:2305.16960, 2023b.
  28. Norman Malcolm. Wittgenstein on language and rules. Philosophy, 64(247):5–28, 1989.
  29. Microeconomic theory, volume 1. Oxford university press New York, 1995.
  30. Herve Moulin. Game theory for the social sciences. NYU press, 1986.
  31. Rosemarie Nagel. Unraveling in guessing games: An experimental study. The American economic review, 85(5):1313–1326, 1995.
  32. Industrial engineering with large language models: A case study of chatgpt’s performance on oil & gas problems, 2023.
  33. OpenAI. Introducing chatgpt. https://openai.com/blog/chatgpt, 2022. Accessed: 2023-09-30.
  34. OpenAI. Gpt-4 technical report, 2023.
  35. Training language models to follow instructions with human feedback, 2022.
  36. Generative agents: Interactive simulacra of human behavior. In In the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23), UIST ’23, New York, NY, USA, 2023. Association for Computing Machinery.
  37. Investigating emergent goal-like behaviour in large language models using experimental economics. arXiv preprint arXiv:2305.07970, 2023.
  38. Communicative agents for software development, 2023.
  39. Probabilistic language understanding: An introduction to the rational speech act framework. Retrieved January, 17:2021, 2018.
  40. Moral Sentiments. Chapter 7: Economic behavior and rationality. Boston University, 2019.
  41. Reflexion: Language agents with verbal reinforcement learning, 2023.
  42. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016.
  43. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023.
  44. Cognitive architectures for language agents, 2023.
  45. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022.
  46. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  47. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  48. William Vickrey. Counterspeculation, auctions, and competitive sealed tenders. The Journal of finance, 16(1):8–37, 1961.
  49. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023a.
  50. A survey on large language model based autonomous agents, 2023b.
  51. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560, 2023c.
  52. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022a.
  53. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022b.
  54. Chain-of-thought prompting elicits reasoning in large language models, 2023a.
  55. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846, 2023b.
  56. Lilian Weng. Llm-powered autonomous agents. lilianweng.github.io, Jun 2023. URL https://lilianweng.github.io/posts/2023-06-23-agent/.
  57. Self-polish: Enhance reasoning in large language models via problem refinement, 2023.
  58. Towards autonomous system: flexible modular production system enhanced with large language model agents. arXiv preprint arXiv:2304.14721, 2023.
  59. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.04658, 2023.
  60. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023.
  61. Tree of thoughts: Deliberate problem solving with large language models, 2023a.
  62. React: Synergizing reasoning and acting in language models, 2023b.
  63. Muhamet Yildiz. Economic applications of game theory, chapter 5 rationalizability, 2012. URL https://ocw.mit.edu/courses/14-12-economic-applications-of-game-theory-fall-2012/pages/lecture-notes/.
  64. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022.
  65. Building cooperative embodied agents modularly with large language models. arXiv preprint arXiv:2307.02485, 2023.
  66. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
  67. Agieval: A human-centric benchmark for evaluating foundation models, 2023.
  68. Don’t make your llm an evaluation benchmark cheater, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Shangmin Guo (18 papers)
  2. Haoran Bu (2 papers)
  3. Haochuan Wang (8 papers)
  4. Yi Ren (215 papers)
  5. Dianbo Sui (19 papers)
  6. Yuming Shang (4 papers)
  7. Siting Lu (2 papers)
Citations (5)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets