Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Action Q-Transformer: Visual Explanation in Deep Reinforcement Learning with Encoder-Decoder Model using Action Query (2306.13879v1)

Published 24 Jun 2023 in cs.LG

Abstract: The excellent performance of Transformer in supervised learning has led to growing interest in its potential application to deep reinforcement learning (DRL) to achieve high performance on a wide variety of problems. However, the decision making of a DRL agent is a black box, which greatly hinders the application of the agent to real-world problems. To address this problem, we propose the Action Q-Transformer (AQT), which introduces a transformer encoder-decoder structure to Q-learning based DRL methods. In AQT, the encoder calculates the state value function and the decoder calculates the advantage function to promote the acquisition of different attentions indicating the agent's decision-making. The decoder in AQT utilizes action queries, which represent the information of each action, as queries. This enables us to obtain the attentions for the state value and for each action. By acquiring and visualizing these attentions that detail the agent's decision-making, we achieve a DRL model with high interpretability. In this paper, we show that visualization of attention in Atari 2600 games enables detailed analysis of agents' decision-making in various game tasks. Further, experimental results demonstrate that our method can achieve higher performance than the baseline in some games.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. Agent57: Outperforming the atari human benchmark. In International conference on machine learning, 2020.
  2. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, 2017.
  3. End-to-end object detection with transformers. In European conference on computer vision, pp.  213–229. Springer, 2020.
  4. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021.
  5. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
  6. Noisy networks for exploration. In International conference on learning representations, 2018.
  7. Visualizing and understanding Atari agents. Proceedings of international conference on machine learning (ICML), 80:1792–1801, 2018.
  8. Rainbow: Combining improvements in deep reinforcement learning. In the AAAI conference on artificial intelligence, 2018.
  9. Scaling laws for neural language models. arXiv preprint, arXiv:2001.08361, 2020.
  10. Human-level atari 200x faster. In International conference on learning representations, 2023.
  11. Towards interpretable deep reinforcement learning with human-friendly prototypes. In International conference on learning representations, 2023.
  12. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23:4909–4926, 2020.
  13. Multi-game decision transformers. Advances in neural information processing systems, abs/2205.15241, 2022.
  14. Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection. The International Journal of Robotics Research, 37(4–5):421–436, 2018.
  15. Reinforcement learning with attention that works: A self-supervised approach. In In International Conference on Neural Information Processing. Springer, Cham., pp.  223–230, 2019.
  16. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
  17. Towards interpretable reinforcement learning using attention augmented agents. Advances in neural information processing systems, 32:12350–12359, 2019.
  18. Finding and visualizing weaknesses of deep reinforcement learning agents. In International conference on learning representations, 2020.
  19. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015.
  20. Proximal policy optimization algorithms. In arXiv preprint arXiv:1707.06347, 2017.
  21. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. Proceedings of the IEEE international conference on computer vision, pp.  618–626, 2017.
  22. A survey of deep reinforcement learning in video games. arXiv preprint arXiv:1912.10944, 2019.
  23. Self-supervised discovering of interpretable features for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
  24. Reinforcement learning: An introduction. MIT press, 1988.
  25. Deep Reinforcement Learning with Double Q-Learning. In the AAAI conference on artificial intelligence, 2016.
  26. Attention is all you need. In Proceedings of neural information processing systems, volume 30, 2017.
  27. Dueling Network Architectures for Deep Reinforcement Learning. In International conference on machine learning, pp. 1995–2003, 2016.
  28. Q-Learning. Machine learning, 8(3-4):279–292, 1992.
  29. Visual rationalizations in deep reinforcement learning for atari games. In Benelux conference on artificial intelligence, pp. 151–165, 2018.
  30. AGIL: Learning Attention from Human for Visuomotor Tasks. In Proceedings of the European conference on computer vision, pp.  663–679, 2018.
  31. Online decision transformer. In Proceedings of international conference on machine learning, pp.  27042–27059, 2022.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now