Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash 90 tok/s
Gemini 2.5 Pro 41 tok/s Pro
GPT-5 Medium 20 tok/s
GPT-5 High 23 tok/s Pro
GPT-4o 93 tok/s
GPT OSS 120B 441 tok/s Pro
Kimi K2 212 tok/s Pro
2000 character limit reached

Interpretable and Editable Programmatic Tree Policies for Reinforcement Learning (2405.14956v1)

Published 23 May 2024 in cs.AI and cs.LG

Abstract: Deep reinforcement learning agents are prone to goal misalignments. The black-box nature of their policies hinders the detection and correction of such misalignments, and the trust necessary for real-world deployment. So far, solutions learning interpretable policies are inefficient or require many human priors. We propose INTERPRETER, a fast distillation method producing INTerpretable Editable tRee Programs for ReinforcEmenT lEaRning. We empirically demonstrate that INTERPRETER compact tree programs match oracles across a diverse set of sequential decision tasks and evaluate the impact of our design choices on interpretability and performances. We show that our policies can be interpreted and edited to correct misalignments on Atari games and to explain real farming strategies.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Information Fusion, 2020.
  2. Model interpretability through the lens of computational complexity. In Neural Information Processing Systems (NeurIPS), 2020.
  3. Model interpretability through the lens of computational complexity. Advances in neural information processing systems, 2020.
  4. Assessing the interpretability of programmatic policies with large language models, 2024.
  5. Verifiable reinforcement learning via policy extraction. In Neural Information Processing Systems, 2018.
  6. The arcade learning environment: An evaluation platform for general agents (extended abstract). In International Joint Conference on Artificial Intelligence, 2012.
  7. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=PczQtTsTIX.
  8. Classification And Regression Trees. Taylor and Francis, New York, 1984.
  9. Ocatari: Object-centric atari 2600 reinforcement learning environments. ArXiv, 2023a.
  10. Interpretable and explainable logical policies via neurally guided symbolic abstraction. 2023b.
  11. Boosting object representation learning via motion and object continuity. In Danai Koutra, Claudia Plant, Manuel Gomez Rodriguez, Elena Baralis, and Francesco Bonchi, editors, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML), 2023c.
  12. Interpretable concept bottlenecks to align reinforcement learning agents. arXiv, 2024.
  13. Murtree: Optimal decision trees via dynamic programming and search. Journal of Machine Learning Research, 23(26):1–47, 2022. URL http://jmlr.org/papers/v23/20-520.html.
  14. Goal misgeneralization in deep reinforcement learning. In International Conference on Machine Learning ICML, 2022.
  15. Introduction to evolutionary computing. Springer, 2015.
  16. Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning. In Proceedings of the 42nd acm sigplan international conference on programming language design and implementation, 2021.
  17. Learning crop management by reinforcement: gym-dssat. In AIAFS 2023-2nd AAAI Workshop on AI for Agriculture and Food Systems, 2023.
  18. A survey on interpretable reinforcement learning. ArXiv, 2021.
  19. A survey of methods for explaining black box models. ACM Comput. Surv., 2018.
  20. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, 2018.
  21. Interpretable decision tree search as a markov decision process, 2024.
  22. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=wRXzOa2z5T.
  23. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, 2012.
  24. Discovering symbolic policies with deep reinforcement learning. In International Conference on Machine Learning, pages 5979–5989. PMLR, 2021.
  25. SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations, 2020.
  26. Zachary Chase Lipton. The mythos of model interpretability. ArXiv, 2016.
  27. Insight: End-to-end neuro-symbolic visual reinforcement learning with language explanations. ArXiv, 2024.
  28. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents (extended abstract). In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI, 2018.
  29. Quant-BnB: A scalable branch-and-bound method for optimal decision trees with continuous features. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 15255–15277. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/mazumder22a.html.
  30. A survey of explainable reinforcement learning. ArXiv, 2022.
  31. Playing atari with deep reinforcement learning. ArXiv, 2013.
  32. Human-level control through deep reinforcement learning. Nature, 2015.
  33. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences, 2019.
  34. Lookahead and pathology in decision tree induction. In Proceedings of the 14th International Joint Conference on Artificial Intelligence - Volume 2, IJCAI’95, page 1025–1031, San Francisco, CA, USA, 1995. Morgan Kaufmann Publishers Inc. ISBN 1558603638.
  35. A system for induction of oblique decision trees. Journal of artificial intelligence research, 1994.
  36. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2019.
  37. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011.
  38. Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
  39. Wenjie Qiu and He Zhu. Programmatic reinforcement learning without oracles. In International Conference on Learning Representations, 2022.
  40. Antonin Raffin. Rl baselines3 zoo, 2020.
  41. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 2021.
  42. Explainable deep learning: A field guide for the uninitiated. Journal of Artificial Intelligence Research, 2022.
  43. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics, 2010.
  44. Explainability via causal self-talk. 2022.
  45. Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities. Knowledge-Based Systems, 2023.
  46. Computational Complexity Analysis of Decision Tree Algorithms: 38th SGAI International Conference on Artificial Intelligence, AI 2018, Cambridge, UK, December 11–13, 2018, Proceedings, pages 191–197. 11 2018. ISBN 978-3-030-04190-8. doi: 10.1007/978-3-030-04191-5_17.
  47. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2020.
  48. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, 2015.
  49. Proximal policy optimization algorithms. ArXiv, 2017.
  50. Reinforcement learning: An introduction. MIT press, 2018.
  51. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots andSystems (IROS), pages 5026–5033. IEEE, 2012.
  52. Iterative bounding mdps: Learning interpretable policies via non-interpretable methods. Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
  53. Gymnasium, 2023.
  54. Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, 2016.
  55. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  56. Programmatically interpretable reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, ICML, 2018.
  57. Pix2code: Learning to compose neural visual concepts as programs. ArXiv, 2024.
  58. Efficient decompositional rule extraction for deep neural networks. In eXplainable AI approaches for debugging and diagnosis., 2021.
  59. Fast segment anything. arXiv, 2023.
Citations (11)
View on Semantic Scholar
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Ai Generate Text Spark Streamline Icon: https://streamlinehq.com

Paper Prompts

Sign up for free to create and run prompts on this paper using GPT-5.

List Streamline Icon: https://streamlinehq.com Explore 10 Community Prompts
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now