Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Modelling crypto markets by multi-agent reinforcement learning (2402.10803v1)

Published 16 Feb 2024 in q-fin.CP, cs.AI, cs.GT, and cs.MA

Abstract: Building on a previous foundation work (Lussange et al. 2020), this study introduces a multi-agent reinforcement learning (MARL) model simulating crypto markets, which is calibrated to the Binance's daily closing prices of $153$ cryptocurrencies that were continuously traded between 2018 and 2022. Unlike previous agent-based models (ABM) or multi-agent systems (MAS) which relied on zero-intelligence agents or single autonomous agent methodologies, our approach relies on endowing agents with reinforcement learning (RL) techniques in order to model crypto markets. This integration is designed to emulate, with a bottom-up approach to complexity inference, both individual and collective agents, ensuring robustness in the recent volatile conditions of such markets and during the COVID-19 era. A key feature of our model also lies in the fact that its autonomous agents perform asset price valuation based on two sources of information: the market prices themselves, and the approximation of the crypto assets fundamental values beyond what those market prices are. Our MAS calibration against real market data allows for an accurate emulation of crypto markets microstructure and probing key market behaviors, in both the bearish and bullish regimes of that particular time period.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (115)
  1. Modelling stock markets by multi-agent reinforcement learning. Computational Economics, pages 1–35, 2020a.
  2. Forecasting volatility in bitcoin market. Annals of Finance, 16(3):435–462, 2020.
  3. Cryptocurrency market contagion: Market uncertainty, market complexity, and dynamic portfolios. Journal of International Financial Markets, Institutions and Money, 61:37–51, 2019.
  4. Fundamental pricing of utility tokens. working paper, 2021.
  5. Multiscale characteristics of the emerging global cryptocurrency market. Physics Reports, 901:1–82, 2021.
  6. What keeps stablecoins stable? Journal of International Money and Finance, 131:102777, 2023.
  7. Hemang Subramanian. Security tokens: architecture, smart contract applications and illustrations using safe. Managerial Finance, 46(6):735–748, 2020.
  8. A survey of distributed consensus protocols for blockchain networks. IEEE Communications Surveys & Tutorials, 22(2):1432–1465, 2020.
  9. Tokenomics: when tokens beat equity. Management Science, 69(11):6568–6583, 2023.
  10. Tokenomics: Dynamic adoption and valuation. The Review of Financial Studies, 34(3):1105–1155, 2021.
  11. Satoshi Nakamoto et al. Bitcoin. A peer-to-peer electronic cash system, 21260, 2009.
  12. Distributed ledger technology. Internet computing: Principles of distributed systems and emerging internet-based technologies, pages 265–299, 2020.
  13. A survey of blockchain consensus protocols. ACM Computing Surveys, 2023.
  14. A survey of layer-two blockchain protocols. Journal of Network and Computer Applications, 209:103539, 2023.
  15. Arman Eshraghi. Approaches to cryptocurrency valuation. In The Emerald Handbook on Cryptoassets: Investment Opportunities and Challenges, pages 171–184. Emerald Publishing Limited, 2023.
  16. Investors’ beliefs and cryptocurrency prices. The Review of Asset Pricing Studies, page raad015, 2024.
  17. Dynamic principal market determination: Fair value measurement of cryptocurrency. Journal of Accounting, Auditing & Finance, 38(4):731–748, 2023.
  18. Herding and anchoring in cryptocurrency markets: Investor reaction to fear and uncertainty. Journal of Behavioral and Experimental Finance, 25:100271, 2020.
  19. Learning and cognition in financial markets: A paradigm shift for agent-based models. Proceedings of SAI Intelligent Systems Conference, pages 241–255, 2020b.
  20. An agent-based computational model for china’s stock market and stock index futures market. Mathematical Problems in Engineering, 2014:563912, 2014.
  21. Agent-based models of the economy, from theories to applications. Palgrave Macmillan, 2015.
  22. A fractional reaction–diffusion description of supply and demand. The European Physical Journal B, 91(23), 2018.
  23. E. Way and M. P. Wellman. Latency arbitrage, market fragmentation, and efficiency: a two-market model. Proceedings of the fourteenth ACM conference on Electronic commerce, pages 855–872, 2013.
  24. M. Aloud. Agent-based simulation in finance: design and choices. Proceedings in Finance and Risk Perspectives ‘14, 2014.
  25. F. H. Westerhoff. The use of agent-based financial market models to test the effectiveness of regulatory policies. Jahrbucher Fur Nationalokonomie Und Statistik, 228(2):195, 2008.
  26. Tipping points in macroeconomic agent-based models. Journal of Economic Dynamics and Control, 50:29–61, 2015.
  27. S. Barde. A practical, universal, information criterion over nth order markov processes. University of Kent, School of Economics Discussion Papers, 04, 2015.
  28. G. Kim and H. M. Markowitz. Investment rules, margin and market volatility. Journal of Portfolio Management, 16:45–52, 1989.
  29. M. Levy and S. Solomon. Power laws are logarithmic boltzmann laws. International Journal of Modern Physics C, 7:595–601, 1996a.
  30. A microscopic model of the stock market: cycles, booms, and crashes. Economics Letters, 45:103–111, 1994.
  31. Microscopic simulation of the stock market: the effect of microscopic diversity. Journal de Physique I, 5:1087–1107, 1995.
  32. M. Levy and S. Solomon. Dynamical explanation for the emergence of power law in a stock market model. International Journal of Modern Physics C, 7:65–72, 1996b.
  33. The complex dynamics of a simple stock market model. International Journal of High Speed Computing, 8:93–113, 1996.
  34. New evidence for the power-law distribution of wealth. Physica A, 242:90–94, 1997.
  35. Microscopic simulation of financial markets: from investor behavior to market phenomena. Academic Press, New York, 2000.
  36. R. Cont and J. P. Bouchaud. Herd behavior and aggregate fluctuations in financial markets. Macroeconomic Dynamics, 4:170–196, 2000.
  37. Social percolation models. Physica A, 277(1):239–247, 2000.
  38. T. Lux and M. Marchesi. Scaling and criticality in a stochastic multi-agent model of a financial market. Nature, 397:498–500, 1999.
  39. T. Lux and M. Marchesi. Volatility clustering in financial markets: a microsimulation of interacting agents. Journal of Theoretical and Applied Finance, 3:67–70, 2000.
  40. Modelling an imperfect market. Physica A, 283:469–478, 2000.
  41. R. Donangelo and K. Sneppen. Self-organization of value and demand. Physica A, 276:572–580, 2000.
  42. Dynamics of money. Physical Review E, 60:2528–2532, 1999.
  43. Money and goldstone modes. Quantitative Finance, 1:186–190, 2001.
  44. Z. F. Huang and S. Solomon. Power, lévy, exponential and gaussian-like regimes in autocatalytic financial systems. European Physical Journal B, 20:601–607, 2000.
  45. J.A. Lipski and R. Kutner. Agent-based stock market model with endogenous agents’ impact. arXiv:1310.0762, 2013.
  46. M. Potters and J.-P. Bouchaud. More stylized facts of financial markets: Leverage effect and downside correlations. Physica A, 299:60–70, 2001.
  47. Scaling of the distribution of fluctuations of financial market indices. Physical Review E, 60(6):6519, 1999.
  48. M. Cristelli. Complexity in Financial Markets. Springer, 2014.
  49. R. Cont. Empirical properties of asset returns: stylized facts and statistical issues. Quantitative Finance, 1:223–236, 2001.
  50. Scale Invariance and Beyond, Proc. CNRS Workshop on Scale Invariance, Les Houches. Springer, 1997.
  51. A long memory property of stock market returns and a new model. Journal of Empirical Finance, 1:83–106, 1993.
  52. Real and spurious long-memory properties of stock-market data. Journal of Business and Economics Statistics, 16:261–283, 1998.
  53. N. Vandewalle and M. Ausloos. Coherent and random sequences in financial fluctuations. Physica A, 246:454–459, 1997.
  54. A multifractal model of asset returns. Cowles Foundation for Research and Economics, 1997.
  55. Robert F. Engle. Autoregressive conditional heteroscedasticity with estimates of the variance of united kingdom inflation. Econometrica, 50(4):987–1007, 1982.
  56. Stylized facts of nominal exchange rate returns. Working Papers from Purdue University, Krannert School of Management - Center for International Business Education and Research (CIBER), 1994.
  57. A. Pagan. The econometrics of financial markets. Journal of Empirical Finance, 3:15–102, 1996.
  58. B. Mandelbrot. The variation of certain speculative prices. The Journal of Business, pages 394–419, 1963.
  59. Rama Cont. Chapter 7 - Agent-Based Models for Market Impact and Volatility. A Kirman and G Teyssiere: Long memory in economics, Springer, 2005.
  60. E. Fama. Efficient capital markets: A review of theory and empirical work. Journal of Finance, 25:383–417, 1970.
  61. Financial Econometrics and Empirical Market Microstructure. Springer, 2015.
  62. Reverse engineering financial markets with majority and minority games using genetic algorithms. Computational Economics, 41:475–492, 2013.
  63. Toward reverse engineering to economic analysis: An overview of tools and methodology. Journal of the Knowledge Economy, 13(2):1414–1432, 2022.
  64. Generative agent-based modeling with actions grounded in physical, social, or digital space using concordia. arXiv preprint arXiv:2312.03664, 2023.
  65. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583–589, 2021.
  66. A general reinforcement learning algorithm that masters chess, shogi and go through self-play. Science, 362(6419):1140–1144, 2018a. ISSN 0036-8075.
  67. Mastering the game of go without human knowledge. Nature, 550:354–359, 2018b.
  68. Deep reinforcement learning in agent based financial market simulation. Journal of Risk and Financial Management, 13(4), 2020. ISSN 1911-8074. doi: 10.3390/jrfm13040071. URL https://www.mdpi.com/1911-8074/13/4/71.
  69. Replicating Financial Markets using Reinforcement Learning: An Agent-Based Approach. Master Thesis, NTNU, 2019.
  70. A. V. Rutkauskas and T. Ramanauskas. Building an artificial stock market populated by reinforcement?learning agents. Journal of Business Economics and Management, 10(4):329–341, 2009. doi: https://doi.org/10.3846/1611-1699.2009.10.329-341.
  71. The successor representation in human reinforcement learning. Nature Human Behavior, 1:680–692, 2017.
  72. Behavioural and neural characterization of optimistic reinforcement learning. Nature Human Behaviour, 1(4), 2017.
  73. Contextual modulation of value signals in reward and punishment learning. Nature communications, pages 1–14, 2015.
  74. More than the sum of its parts: A role for the hippocampus in configural reinforcement learning. Neuron, 98:645–657, 2018.
  75. Reinforcement learning for market making in a multi-agent dealer market. arXiv:1911.05892, 2019.
  76. Deep reinforcement learning for optimizing portfolio management. 2019 Amity International Conference on Artificial Intelligence, 2019.
  77. R. Neuneier. Enhancing q-learning for optimal asset allocation. Proc. of the 10th International Conference on Neural Information Processing Systems, 1997.
  78. Deep direct reinforcement learning for financial signal representation and trading. IEEE Trans. on Neural Networks and Learning Systems, 28(3), 2017.
  79. Using an artificial financial market for studying a cryptocurrency market. Journal of Economic Interaction and Coordination, 12:345–365, 2017.
  80. Market making via reinforcement learning. Proceedings of the 17th AAMAS, 2018.
  81. Alessio Emanuele Biondo. Order book modeling and financial stability. Journal of Economic Interaction and Coordination, 14(3), 2019.
  82. Universal features of price formation in financial markets: perspectives from deep learning. Quantitative Finance, 19(9), 2019.
  83. Stock price formation: Precepts from a multi-agent reinforcement learning model. Computational Economics, pages 1–22, 2022.
  84. A. Dodonova and Y. Khoroshilov. Private information in futures markets: An experimental study. Manag Decis Econ, 39, 2018.
  85. The relationship between stock market volatility and trading volume: evidence from south africa. J Dev Areas, 52(1), 2018.
  86. Symba code repository, 2023. URL https://github.com/johannlussange/symba_crypto. Accessed: 2023-12-25.
  87. R. Sutton and A. Barto. Reinforcement Learning, second edition: An Introduction. Bradford Books, 2018.
  88. Marco Wiering and Martijn van Otterlo. Reinforcement Learning: State-of-the-Art. Springer, Berlin, Heidelberg, 2012.
  89. Csaba Szepesvari. Algorithms for Reinforcement Learning. Morgan and Claypool Publishers, 2010.
  90. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12:1057–1063, 2000.
  91. Deterministic policy gradient algorithms. Proceedings of the 31st International Conference on Machine Learning, 32, 2014.
  92. Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279–292, 1992.
  93. Mastering the game of go with deep neural networks and tree search. Nature, 529:484–489, 2016.
  94. A bayesian approach for learning and planning in partially observable markov decision processes. Journal of Machine Learning Research, 12:1729–1770, 2011.
  95. Learning in pomdps with monte carlo tree search. Proceedings of the 34th International Conference on Machine Learning, 2017.
  96. Robust adversarial reinforcement learning. arXiv:1703.02702, 2017.
  97. Prefrontal cortex as a meta-reinforcement learning system. Nature Neuroscience, 21:860–868, 2018.
  98. A survey of actor-critic reinforcement learning: standard and natural policy gradients. IEEE Transactions on Systems Man and Cybernetics, 42:1291–1307, 2012.
  99. Johannes Heinrich. Deep RL from Self-Play in Imperfect-Information Games. PhD thesis, University College London, 2017.
  100. Asynchronous methods for deep reinforcement learning. arXiv:1602.01783, 2016.
  101. Modular multitask reinforcement learning with policy sketches. International Conference on Machine Learning, 2017.
  102. A deep hierarchical approach to lifelong learning in minecraft. arXiv:1604.07255, 2016.
  103. Actor-critic algorithms for hierarchical decision processes. Automatica, 42, 2006.
  104. Theory and application to reward shaping. International Conference on Machine Learning, 1999.
  105. Autonomous helicopter aerobatics through apprenticeship learning. The International Journal of Robotics Research, 2010.
  106. M. Keramati and B. Gutkin. Homeostatic reinforcement learning for integrating reward collection and physiological stability. Elife, 3, 2014.
  107. M. Keramati and B. Gutkin. A reinforcement learning theory for homeostatic regulation. NIPS, 2011.
  108. Reference-point centering and range-adaptation enhance human reinforcement learning at the cost of irrational preferences. Nature Communications, 4503, 2018.
  109. A detailed heterogeneous agent model for a single asset financial market with trading via an order book. arXiv:1601.00229, 2016.
  110. Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing. PLoS computational biology, 13(8), 2017.
  111. What is a free lunch? Notices of the AMS, 51(5), 2011.
  112. Simulating and analyzing order book data: the queue-reactive model. Journal of the American Statistical Association, 110:509, 2015.
  113. Market microstructure and order book dynamics in cryptocurrency exchanges. In Crypto Valley Conference on Blockchain Technology, 2018.
  114. Eduard Silantyev. Order flow analysis of cryptocurrency markets. Digital Finance, 1(1-4):191–218, 2019.
  115. Allocative efficiency of markets with zero-intelligence traders: Market as a partial substitute for individual rationality. Journal of Political Economy, 101(1), 1993.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now