Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Simulating the Economic Impact of Rationality through Reinforcement Learning and Agent-Based Modelling (2405.02161v2)

Published 3 May 2024 in cs.LG, cs.AI, cs.CE, cs.MA, econ.GN, and q-fin.EC

Abstract: Agent-based models (ABMs) are simulation models used in economics to overcome some of the limitations of traditional frameworks based on general equilibrium assumptions. However, agents within an ABM follow predetermined 'bounded rational' behavioural rules which can be cumbersome to design and difficult to justify. Here we leverage multi-agent reinforcement learning (RL) to expand the capabilities of ABMs with the introduction of 'fully rational' agents that learn their policy by interacting with the environment and maximising a reward function. Specifically, we propose a 'Rational macro ABM' (R-MABM) framework by extending a paradigmatic macro ABM from the economic literature. We show that gradually substituting ABM firms in the model with RL agents, trained to maximise profits, allows for studying the impact of rationality on the economy. We find that RL agents spontaneously learn three distinct strategies for maximising profits, with the optimal strategy depending on the level of market competition and rationality. We also find that RL agents with independent policies, and without the ability to communicate with each other, spontaneously learn to segregate into different strategic groups, thus increasing market power and overall profits. Finally, we find that a higher number of rational (RL) agents in the economy always improves the macroeconomic environment as measured by total output. Depending on the specific rational policy, this can come at the cost of higher instability. Our R-MABM framework allows for stable multi-agent learning, is available in open source, and represents a principled and robust direction to extend economic simulators.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (34)
  1. Multi-agent reinforcement learning: Foundations and modern approaches. Massachusetts Institute of Technology: Cambridge, MA, USA, 2023.
  2. Phantom-a rl-driven multi-agent framework to model complex systems. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pages 2742–2744, 2023.
  3. Emergent dynamics of a macroeconomic agent based model with capital and credit. Journal of Economic Dynamics and Control, 50:5–28, 2015.
  4. T. Atashbar and R. A. Shi. Deep reinforcement learning: emerging trends in macroeconomics and future prospects. 2022.
  5. T. Atashbar and R. A. Shi. AI and macroeconomic modeling: Deep reinforcement learning in an RBC model. International Monetary Fund, 2023.
  6. Agent-based modeling in economics and finance: Past, present, and future. Journal of Economic Literature, 2022.
  7. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009.
  8. Introduction to computer science and economic theory. Journal of Economic Theory, 156:1–13, 2015.
  9. Artificial intelligence, algorithmic pricing, and collusion. American Economic Review, 110(10):3267–3297, 2020.
  10. Reinforcement learning in economics and finance. Computational Economics, pages 1–38, 2021.
  11. Learning solutions in large economic networks using deep multi-agent reinforcement learning. In AAMAS, pages 2760–2762, 2023.
  12. H. Dawid and D. D. Gatti. Agent-based macroeconomics. Handbook of computational economics, 4:63–156, 2018.
  13. Abides-economist: Agent-based simulation of economic systems with learning agents. arXiv preprint arXiv:2402.09563, 2024.
  14. J. D. Farmer and D. Foley. The economy needs agent-based modelling. Nature, 460(7256):685–686, 2009.
  15. D. D. Gatti and J. Grazzini. Rising to the challenge: Bayesian estimation and forecasting techniques for macroeconomic agent based models. Journal of Economic Behavior & Organization, 178:875–902, 2020.
  16. Reinforcement learning for combining search methods in the calibration of economic abms. In Proceedings of the Fourth ACM International Conference on AI in Finance, pages 305–313, 2023.
  17. S. Gronauer and K. Diepold. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review, 55(2):895–943, 2022.
  18. Solving heterogeneous general equilibrium economic models with deep reinforcement learning. arXiv preprint arXiv:2103.16977, 2021.
  19. N. Hinterlang and A. Tänzer. Optimal monetary policy using reinforcement learning. 2021.
  20. Emergent bartering behaviour in multi-agent reinforcement learning. arXiv preprint arXiv:2205.06760, 2022.
  21. A. Kuriksha. An economy of neural networks: Learning from heterogeneous experiences. arXiv preprint arXiv:2110.11582, 2021.
  22. E. Lindau. Optimal taxation by two-agent reinforcement learning, 2023.
  23. E. Maskin and J. Tirole. Markov perfect equilibrium: I. observable actions. Journal of Economic Theory, 100(2):191–219, 2001.
  24. Taxai: A dynamic economic simulator and benchmark for multi-agent reinforcement learning. arXiv preprint arXiv:2309.16307, 2023.
  25. J. Rothe et al. Economics and computation, volume 4. Springer, 2015.
  26. L. S. Shapley. Stochastic games. Proceedings of the National Academy of Sciences, 39(10):1095–1100, 1953.
  27. Reinforcement learning: An introduction. MIT press, 2018.
  28. L. Tesfatsion. Agent-based computational economics: A constructive approach to economic theory. Handbook of computational economics, 2:831–880, 2006.
  29. L. Tesfatsion. Agent-based computational economics: Overview and brief history. Artificial Intelligence, Learning and Computation in Economics and Finance, pages 41–58, 2023.
  30. Towards multi-agent reinforcement learning-driven over-the-counter market simulations. Mathematical Finance, 34(2):262–347, 2024.
  31. J. van der Wal. Stochastic dynamic programming. 1981.
  32. C. Watkins and P. Dayan. Q-learning. Machine Learning, 8:279–292, 1992.
  33. M. Wooldridge. An introduction to multiagent systems. John wiley & sons, 2009.
  34. The ai economist: Taxation policy design via two-level deep multiagent reinforcement learning. Science advances, 8(18):eabk2607, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Simone Brusatin (1 paper)
  2. Tommaso Padoan (7 papers)
  3. Andrea Coletta (15 papers)
  4. Domenico Delli Gatti (3 papers)
  5. Aldo Glielmo (25 papers)
Citations (1)
View on Semantic Scholar