Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Explainable Post hoc Portfolio Management Financial Policy of a Deep Reinforcement Learning agent (2407.14486v1)

Published 19 Jul 2024 in cs.CE, cs.AI, and q-fin.PM

Abstract: Financial portfolio management investment policies computed quantitatively by modern portfolio theory techniques like the Markowitz model rely on a set on assumptions that are not supported by data in high volatility markets. Hence, quantitative researchers are looking for alternative models to tackle this problem. Concretely, portfolio management is a problem that has been successfully addressed recently by Deep Reinforcement Learning (DRL) approaches. In particular, DRL algorithms train an agent by estimating the distribution of the expected reward of every action performed by an agent given any financial state in a simulator. However, these methods rely on Deep Neural Networks model to represent such a distribution, that although they are universal approximator models, they cannot explain its behaviour, given by a set of parameters that are not interpretable. Critically, financial investors policies require predictions to be interpretable, so DRL agents are not suited to follow a particular policy or explain their actions. In this work, we developed a novel Explainable Deep Reinforcement Learning (XDRL) approach for portfolio management, integrating the Proximal Policy Optimization (PPO) with the model agnostic explainable techniques of feature importance, SHAP and LIME to enhance transparency in prediction time. By executing our methodology, we can interpret in prediction time the actions of the agent to assess whether they follow the requisites of an investment policy or to assess the risk of following the agent suggestions. To the best of our knowledge, our proposed approach is the first explainable post hoc portfolio management financial policy of a DRL agent. We empirically illustrate our methodology by successfully identifying key features influencing investment decisions, which demonstrate the ability to explain the agent actions in prediction time.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. The basics of finance: An introduction to financial markets, business finance, and portfolio management, volume 192. John Wiley & Sons, 2010.
  2. Myles E Mangram. A simplified perspective of the markowitz portfolio theory. Global journal of business research, 7(1):59–70, 2013.
  3. D Sykes Wilford. True markowitz or assumptions we break and why it matters. Review of Financial Economics, 21(3):93–101, 2012.
  4. Joanne M Hill. The different faces of volatility exposure in portfolio management. The Journal of Alternative Investments, 15(3):9, 2013.
  5. Stock market prediction and portfolio selection models: a survey. Opsearch, 54:558–579, 2017.
  6. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26–38, 2017.
  7. Reinforcement learning: An introduction, 2nd ed. MIT press, Cambridge, MA, 2020.
  8. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
  9. Deep learning. Nature, 521(7553):436–444, 2015.
  10. A universal approximation theorem of deep neural networks for expressing probability distributions. Advances in neural information processing systems, 33:3094–3105, 2020.
  11. Davide Castelvecchi. Can we open the black box of AI? Nature News, 538(7623):20, 2016.
  12. Louis Lowenstein. Financial transparency and corporate governance: you manage what you measure. Columbia Law Review, 96(5):1335–1362, 1996.
  13. Intelligible and explainable machine learning: Best practices and practical challenges. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3511–3512, 2020.
  14. Explainable artificial intelligence: an analytical review. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 11(5):e1424, 2021.
  15. Explainable AI: Foundations, methodologies and applications. Springer, 2023.
  16. Alphastock: A buying-winners-and-selling-losers investment strategy using interpretable deep reinforcement attention networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1900–1908, 2019.
  17. Towards interpretable reinforcement learning with state abstraction driven by external knowledge. IEICE TRANSACTIONS on Information and Systems, 103(10):2143–2153, 2020.
  18. Explainable deep reinforcement learning for portfolio management: an empirical approach. In Proceedings of the second ACM international conference on AI in finance, pages 1–9, 2021.
  19. XPM: An explainable deep reinforcement learning framework for portfolio management. In Proceedings of the 30th ACM international conference on information & knowledge management, pages 1661–1670, 2021.
  20. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  21. Principles and practice of explainable machine learning. Frontiers in big Data, 4:688969, 2021.
  22. The shapley value in machine learning. arXiv preprint arXiv:2202.05594, 2022.
  23. Statistical stability indices for lime: Obtaining reliable explanations for machine learning models. Journal of the Operational Research Society, 73(1):91–101, 2022.
  24. Recent advances in reinforcement learning in finance. Mathematical Finance, 33(3):437–503, 2023.
  25. Finrl: Deep reinforcement learning framework to automate trading in quantitative finance. In Proceedings of the second ACM international conference on AI in finance, pages 1–9, 2021.
  26. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015.
  27. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
  28. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
  29. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018.
  30. Twin-delayed ddpg: A deep reinforcement learning technique to model a continuous movement of an intelligent robot agent. In Proceedings of the 3rd international conference on vision, image and signal processing, pages 1–5, 2019.
  31. Practical deep reinforcement learning approach for stock trading. arXiv preprint arXiv:1811.07522, 2018.
  32. Finrl: A deep reinforcement learning library for automated stock trading in quantitative finance. arXiv preprint arXiv:2011.09607, 2020.
  33. Adversarial deep reinforcement learning in portfolio management. arXiv preprint arXiv:1808.09940, 2018.
  34. Cryptocurrency portfolio management with deep reinforcement learning. In 2017 Intelligent systems conference (IntelliSys), pages 905–913. IEEE, 2017.
  35. Jonathan Sadighian. Deep reinforcement learning in cryptocurrency market making. arXiv preprint arXiv:1911.08647, 2019.
  36. Recommending cryptocurrency trading points with deep reinforcement learning approach. Applied Sciences, 10(4):1506, 2020.
  37. Deep reinforcement learning for trading. arXiv preprint arXiv:1911.10107, 2019.
  38. Deep hedging: hedging derivatives under generic market frictions using reinforcement learning. Swiss Finance Institute Research Paper, (19-80), 2019.
  39. Deep hedging of derivatives using reinforcement learning. arXiv preprint arXiv:2103.16409, 2021.
  40. Alexandre Carbonneau. Deep hedging of long-term financial derivatives. Insurance: Mathematics and Economics, 99:327–340, 2021.
  41. Multi-agent reinforcement learning approach for hedging portfolio problem. Soft Computing, 25(12):7877–7885, 2021.
  42. Modelling stock markets by multi-agent reinforcement learning. Computational Economics, 57(1):113–147, 2021.
  43. Mspm: A modularized and scalable multi-agent reinforcement learning-based system for financial portfolio management. Plos one, 17(2):e0263689, 2022.
  44. Maps: Multi-agent reinforcement learning-based portfolio management system. arXiv preprint arXiv:2007.05402, 2020.
  45. Cost-sensitive portfolio selection via deep reinforcement learning. IEEE Transactions on Knowledge and Data Engineering, 34(1):236–248, 2020.
  46. Gpm: A graph convolutional network based reinforcement learning framework for portfolio management. Neurocomputing, 498:14–27, 2022.
  47. Stock portfolio management by using fuzzy ensemble deep reinforcement learning algorithm. Journal of Risk and Financial Management, 16(3):201, 2023.
  48. Deep reinforcement learning for stock portfolio optimization by connecting with modern portfolio theory. Expert Systems with Applications, 218:119556, 2023.
  49. Market sentiment-aware deep reinforcement learning approach for stock portfolio allocation. Engineering Science and Technology, an International Journal, 24(4):848–859, 2021.
  50. Optimistic bull or pessimistic bear: Adaptive deep reinforcement learning for stock portfolio allocation. arXiv preprint arXiv:1907.01503, 2019.
  51. Leveraging deep learning and online source sentiment for financial portfolio management. arXiv preprint arXiv:2309.16679, 2023.
  52. Stock trading bot using deep reinforcement learning. In Innovations in Computer Science and Engineering: Proceedings of the Fifth ICICSE 2017, pages 41–49. Springer, 2019.
  53. Deep reinforcement learning for the control of robotic manipulation: a focussed mini-review. Robotics, 10(1):22, 2021.
  54. A survey of deep reinforcement learning in video games. arXiv preprint arXiv:1912.10944, 2019.
  55. George A Vouros. Explainable deep reinforcement learning: state of the art and challenges. ACM Computing Surveys, 55(5):1–39, 2022.
  56. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
  57. ”Why should I trust you?” explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now