Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Neural architecture impact on identifying temporally extended Reinforcement Learning tasks (2310.03161v1)

Published 4 Oct 2023 in cs.LG and cs.AI

Abstract: Inspired by recent developments in attention models for image classification and natural language processing, we present various Attention based architectures in reinforcement learning (RL) domain, capable of performing well on OpenAI Gym Atari-2600 game suite. In spite of the recent success of Deep Reinforcement learning techniques in various fields like robotics, gaming and healthcare, they suffer from a major drawback that neural networks are difficult to interpret. We try to get around this problem with the help of Attention based models. In Attention based models, extracting and overlaying of attention map onto images allows for direct observation of information used by agent to select actions and easier interpretation of logic behind the chosen actions. Our models in addition to playing well on gym-Atari environments, also provide insights on how agent perceives its environment. In addition, motivated by recent developments in attention based video-classification models using Vision Transformer, we come up with an architecture based on Vision Transformer, for image-based RL domain too. Compared to previous works in Vision Transformer, our model is faster to train and requires fewer computational resources. 3

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (24)
  1. Abien Fred Agarap “Deep Learning using Rectified Linear Units (ReLU)” cite arxiv:1803.08375Comment: 7 pages, 11 figures, 9 tables, 2018 URL: http://arxiv.org/abs/1803.08375
  2. Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey E. Hinton “Layer Normalization”, 2016 arXiv:1607.06450 [stat.ML]
  3. Gedas Bertasius, Heng Wang and Lorenzo Torresani “Is Space-Time Attention All You Need for Video Understanding?” In CoRR abs/2102.05095, 2021 arXiv: https://arxiv.org/abs/2102.05095
  4. G. Bradski “The OpenCV Library” In Dr. Dobb’s Journal of Software Tools, 2000
  5. “OpenAI Gym”, 2016 arXiv:1606.01540 [cs.LG]
  6. “Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context” In CoRR abs/1901.02860, 2019 arXiv: http://arxiv.org/abs/1901.02860
  7. “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale” In International Conference on Learning Representations, 2021 URL: https://openreview.net/forum?id=YicbFdNTTy
  8. “IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures” In CoRR abs/1802.01561, 2018 arXiv: http://arxiv.org/abs/1802.01561
  9. “Visualizing and Understanding Atari Agents” In CoRR abs/1711.00138, 2017 arXiv: http://arxiv.org/abs/1711.00138
  10. “Deep Residual Learning for Image Recognition” In CoRR abs/1512.03385, 2015 arXiv: http://arxiv.org/abs/1512.03385
  11. “Deep Residual Learning for Image Recognition” In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778 DOI: 10.1109/CVPR.2016.90
  12. “Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units” In CoRR abs/1606.08415, 2016 arXiv: http://arxiv.org/abs/1606.08415
  13. Shakti Kumar, Jerrod Parker and Panteha Naderian “Adaptive Transformers in RL” In CoRR abs/2004.03761, 2020 arXiv: https://arxiv.org/abs/2004.03761
  14. “TorchBeast: A PyTorch Platform for Distributed RL” In arXiv preprint arXiv:1910.03552, 2019 URL: https://github.com/facebookresearch/torchbeast
  15. “Reproducing “Towards Interpretable ReinforcementLearning Using Attention Augmented Agents”” Submitted to NeurIPS 2019 Reproducibility Challenge, 2020 URL: https://openreview.net/forum?id=BJgtDa9GaH
  16. “Human-level control through deep reinforcement learning” In Nature 518.7540, 2015, pp. 529–533 DOI: 10.1038/nature14236
  17. “Towards Interpretable Reinforcement Learning Using Attention Augmented Agents” In CoRR abs/1906.02500, 2019 arXiv: http://arxiv.org/abs/1906.02500
  18. “Stabilizing Transformers for Reinforcement Learning” In CoRR abs/1910.06764, 2019 arXiv: http://arxiv.org/abs/1910.06764
  19. Scipy “Savitzky-Golay filter” URL: https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.savgol_filter.html
  20. “Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting” In CoRR abs/1506.04214, 2015 arXiv: http://arxiv.org/abs/1506.04214
  21. Richard S. Sutton and Andrew G. Barto “Reinforcement Learning: An Introduction” The MIT Press, 2018 URL: http://incompleteideas.net/book/the-book-2nd.html
  22. TensorFlow “Transformer model for language understanding” URL: https://www.tensorflow.org/text/tutorials/transformer
  23. “Attention is All you Need” In Advances in Neural Information Processing Systems 30 Curran Associates, Inc., 2017 URL: https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  24. Xiaofeng Yuan, Lin Li and Yalin Wang “Nonlinear Dynamic Soft Sensor Modeling With Supervised Long Short-Term Memory Network” In IEEE Transactions on Industrial Informatics 16.5, 2020, pp. 3168–3176 DOI: 10.1109/TII.2019.2902129
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (1)