Variational Autoencoders for exteroceptive perception in reinforcement learning-based collision avoidance (2404.00623v1)
Abstract: Modern control systems are increasingly turning to machine learning algorithms to augment their performance and adaptability. Within this context, Deep Reinforcement Learning (DRL) has emerged as a promising control framework, particularly in the domain of marine transportation. Its potential for autonomous marine applications lies in its ability to seamlessly combine path-following and collision avoidance with an arbitrary number of obstacles. However, current DRL algorithms require disproportionally large computational resources to find near-optimal policies compared to the posed control problem when the searchable parameter space becomes large. To combat this, our work delves into the application of Variational AutoEncoders (VAEs) to acquire a generalized, low-dimensional latent encoding of a high-fidelity range-finding sensor, which serves as the exteroceptive input to a DRL agent. The agent's performance, encompassing path-following and collision avoidance, is systematically tested and evaluated within a stochastic simulation environment, presenting a comprehensive exploration of our proposed approach in maritime control systems.
- “Improving Sample Efficiency in Model-Free Reinforcement Learning from Images.” (2020). arXiv:1910.01741 [cs.LG]
- “The Dreaming Variational Autoencoder for Reinforcement Learning Environments.” (2018). arXiv:1810.01112 [cs.LG]
- “Risk-based Convolutional Perception Models for Collision Avoidance in Autonomous Marine Surface Vessels using Deep Reinforcement Learning.” IFAC-PapersOnLine Vol. 56 No. 2 (2023): pp. 10033–10038.
- “Nomenclature for treating the motion of a submerged body through a fluid: Report of the American towing tank conference”. (1950). Technical and research bulletin, Society of Naval Architects and Marine Engineers.
- “Modeling, identification, and adaptive maneuvering of CyberShip II: A complete design with experiments.” IFAC Proceedings Volumes Vol. 37 No. 10 (2024) : pp. 203–208.
- “A Ship Heading and Speed Control Concept Inherently Satisfying Actuator Constraints.” IEEE Conference on Control Technology and Applications (CCTA), Maui, HI, USA, (2017), pp. 323-330.
- “Mastering the game of Go with deep neural networks and tree search.” Nature Vol. 529 No. 7587 (2016): pp. 484–489.
- “Agent57: Outperforming the Atari Human Benchmark.” International Conference on Machine Learning. (2020).
- “Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates.” IEEE International Conference on Robotics and Automation (ICRA): (2017). pp. 3389–3396.
- “Reaching the limit in autonomous racing: Optimal control versus reinforcement learning.” Science Robotics Vol. 8 No. 82 (2023): p. eadg1462.
- “Playing Atari with Deep Reinforcement Learning.” (2013).
- “Proximal Policy Optimization Algorithms.” (2017). ArXiv:1707.06347 [cs].
- “Auto-Encoding Variational Bayes.” (2013). arXiv:1312.6114
- “beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework.” International Conference on Learning Representations. (2017).
- “Generating Diverse High-Fidelity Images with VQ-VAE-2.” Advances in Neural Information Processing Systems, Vol. 32. (2019). Curran Associates, Inc.
- “Joint Autoregressive and Hierarchical Priors for Learned Image Compression.” Advances in Neural Information Processing Systems, Vol. 31. (2018). Curran Associates, Inc.
- “Neural Discrete Representation Learning.” Advances in Neural Information Processing Systems, Vol. 30. (2017). Curran Associates, Inc.
- “Taming an Autonomous Surface Vehicle for Path Following and Collision Avoidance Using Deep Reinforcement Learning.” IEEE Access Vol. 8 (2020): pp. 41466–41481.
- “Comparing Deep Reinforcement Learning Algorithms’ Ability to Safely Navigate Challenging Waters.” Frontiers in Robotics and AI Vol. 8 (2021).
- “Modular Control Architecture for Safe Marine Navigation: Reinforcement Learning and Predictive Safety Filters.” (2023). arXiv:2312.01855 [cs.RO]. (2023).
- “COLREG-Compliant Collision Avoidance for Unmanned Surface Vehicle Using Deep Reinforcement Learning.” IEEE Access Vol. 8 (2020): pp. 165344–165364.
- “PyTorch: An Imperative Style, High-Performance Deep Learning Library.” Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché Buc, F., Fox, E. and Garnett, R. (eds.). Advances in Neural Information Processing Systems 32: pp. 8024–8035. (2019). Curran Associates, Inc.
- Chollet, François et al. “Keras.” (2015).
- “Circular Convolutional Neural Networks for Panoramic Images and Laser Data.” 2019 IEEE Intelligent Vehicles Symposium (IV): pp. 653–660. (2019). IEEE, Paris, France.
- “Stable-Baselines3: Reliable Reinforcement Learning Implementations.” Journal of Machine Learning Research Vol. 22 No. 268 (2021): pp. 1–8.
- “Adam: A Method for Stochastic Optimization.”. arXiv:1412.6980 [cs.LG]. (2014).
- “Posterior Collapse and Latent Variable Non-identifiability.” Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S. and Vaughan, J. Wortman (eds.). Advances in Neural Information Processing Systems, Vol. 34: pp. 5443–5455. (2021). Curran Associates, Inc.