Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Two-step dynamic obstacle avoidance (2311.16841v2)

Published 28 Nov 2023 in cs.RO and cs.AI

Abstract: Dynamic obstacle avoidance (DOA) is a fundamental challenge for any autonomous vehicle, independent of whether it operates in sea, air, or land. This paper proposes a two-step architecture for handling DOA tasks by combining supervised and reinforcement learning (RL). In the first step, we introduce a data-driven approach to estimate the collision risk (CR) of an obstacle using a recurrent neural network, which is trained in a supervised fashion and offers robustness to non-linear obstacle movements. In the second step, we include these CR estimates into the observation space of an RL agent to increase its situational awareness. We illustrate the power of our two-step approach by training different RL agents in a challenging environment that requires to navigate amid multiple obstacles. The non-linear movements of obstacles are exemplarily modeled based on stochastic processes and periodic patterns, although our architecture is suitable for any obstacle dynamics. The experiments reveal that integrating our CR metrics into the observation space doubles the performance in terms of reward, which is equivalent to halving the number of collisions in the considered environment. We also perform a generalization experiment to validate the proposal in an RL environment based on maritime traffic and real-world vessel trajectory data. Furthermore, we show that the architecture's performance improvement is independent of the applied RL algorithm.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Self-driving cars: A survey. Expert Systems with Applications 165, 113816.
  2. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature 588, 77–82.
  3. Scalable autonomous separation assurance with heterogeneous multi-agent reinforcement learning. IEEE Transactions on Automation Science and Engineering 19, 2837–2848.
  4. Deep reinforcement learning-based collision avoidance for an autonomous ship. Ocean Engineering 234, 109216.
  5. Convolutional neural network: a review of models, methodologies and applications to object detection. Progress in Artificial Intelligence 9, 85–112.
  6. Motion planning among dynamic, decision-making agents with deep reinforcement learning, in: International Conference on Intelligent Robots and Systems, IEEE. pp. 3052–3059.
  7. Collision avoidance in pedestrian-rich environments with deep reinforcement learning. IEEE Access 9, 10357–10377.
  8. Dynamic obstacle avoidance for quadrotors with event cameras. Science Robotics 5, eaaz9712.
  9. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature 610, 47–53.
  10. Dense reinforcement learning for safety validation of autonomous vehicles. Nature 615, 620–627.
  11. Handbook of Marine Craft Hydrodynamics and Motion Control, 2nd Edition. John Wiley & Sons.
  12. Addressing function approximation error in actor-critic methods, in: International Conference on Machine Learning, PMLR. pp. 1587–1596.
  13. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, in: International Conference on Machine Learning, PMLR. pp. 1861–1870.
  14. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 .
  15. Enhanced method for reinforcement learning based dynamic obstacle avoidance by assessment of collision risk. URL: https://arxiv.org/abs/2212.04123, doi:10.48550/ARXIV.2212.04123.
  16. Memory-based control with recurrent neural networks. arXiv preprint arXiv:1512.04455 .
  17. Long short-term memory. Neural computation 9, 1735–1780.
  18. Ship collision avoidance methods: State-of-the-art. Safety science 121, 451–473.
  19. International Maritime Organization, 1972. COLREG: Convention on the International Regulations for Preventing Collisions at Sea.
  20. Multi-UAV conflict resolution with graph convolutional reinforcement learning. Applied Sciences 12, 610.
  21. Planning and acting in partially observable stochastic domains. Artificial Intelligence 101, 99–134.
  22. Applying the unscented kalman filter for nonlinear state estimation. Journal of Process Control 18, 753–768.
  23. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 .
  24. A review of conflict detection and resolution modeling methods. IEEE Transactions On Intelligent Transportation Systems 1, 179–189.
  25. Deep learning. Nature 521, 436–444.
  26. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks, in: International Conference on Robotics and Automation, IEEE. pp. 8943–8950.
  27. Collision threat parameters for a new radar display and plot technique. The Journal of Navigation 36, 404–410.
  28. Reinforcement learning applications. arXiv preprint arXiv:1908.06973 .
  29. Unmanned surface vehicles: An overview of developments and challenges. Annual Reviews in Control 41, 71–93.
  30. Global and local path planning study in a ros-based research platform for autonomous vehicles. Journal of Advanced Transportation 2018.
  31. Memory-based deep reinforcement learning for pomdps, in: International Conference on Intelligent Robots and Systems, IEEE. pp. 5619–5626.
  32. Taming an autonomous surface vehicle for path following and collision avoidance using deep reinforcement learning. IEEE Access 8, 41466–41481.
  33. Human-level control through deep reinforcement learning. Nature 518, 529–533.
  34. Deep reinforcement learning: an overview, in: Proceedings of SAI Intelligent Systems Conference, Springer. pp. 426–440.
  35. Speech recognition using deep neural networks: A systematic review. IEEE access 7, 19143–19165.
  36. Autonomous ships are on the horizon: here’s what we need to know. Nature 615, 30–33.
  37. Recurrent model-free RL is a strong baseline for many POMDPs. arXiv preprint arXiv:2110.05038 .
  38. Mobile robot navigation and obstacle avoidance techniques: A review. International Robotics & Automation Journal 2, 96–105.
  39. Obstacle avoidance approaches for autonomous navigation of unmanned surface vehicles. The Journal of Navigation 71, 241–256.
  40. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons.
  41. A survey of the opportunities and challenges of supervised machine learning in maritime risk analysis. Transport Reviews 43, 108–130.
  42. Review of conflict resolution methods for manned and unmanned aviation. Aerospace 7, 79.
  43. Conditional affordance learning for driving in urban environments, in: Conference on Robot Learning, PMLR. pp. 237–252.
  44. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science 362, 1140–1144.
  45. Deep predictive models for collision risk assessment in autonomous driving, in: International Conference on Robotics and Automation, IEEE. pp. 4685–4692.
  46. Feature engineering for deep reinforcement learning based routing, in: International Conference on Communications, IEEE. pp. 1–6.
  47. Reinforcement Learning: An Introduction. Cambridge: The MIT Press.
  48. Traffic Flow Dynamics: Data, Models and Simulation. Springer-Verlag Berlin Heidelberg.
  49. Analysis of Financial Time Series. New Jersey: John Wiley & Sons.
  50. Path planning and collision avoidance for autonomous surface vehicles I: a review. Journal of Marine Science and Technology 26, 1292–1306.
  51. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature 575, 350–354.
  52. Spatial–temporal recurrent reinforcement learning for autonomous ships. Neural Networks 165, 634–653.
  53. RL Dresden Algorithm Suite. https://github.com/MarWaltz/TUD_RL.
  54. 2-level reinforcement learning for ships on inland waterways. arXiv preprint arXiv:2307.16769 .
  55. Autonomous navigation of uavs in large-scale complex environments: A deep reinforcement learning approach. IEEE Transactions on Vehicular Technology 68, 2124–2136.
  56. COLREGs-abiding hybrid collision avoidance algorithm based on deep reinforcement learning for usvs. Ocean Engineering 247, 110749.
  57. Towards a convolutional neural network model for classifying regional ship collision risk levels for waterway risk analysis. Reliability Engineering & System Safety 204, 107127.
  58. Physics informed deep reinforcement learning for aircraft conflict resolution. IEEE Transactions on Intelligent Transportation Systems 23, 8288–8301.
  59. A SVM based ship collision risk assessment algorithm. Ocean Engineering 202, 107062.
  60. A survey on epistemic (model) uncertainty in supervised learning: Recent advances and applications. Neurocomputing 489, 449–465.
Citations (1)
View on Semantic Scholar

Summary

  • The paper presents a two-step architecture that combines supervised learning for collision risk estimation with reinforcement learning to significantly reduce collision rates.
  • It leverages an LSTM model to predict non-linear obstacle trajectories, enabling precise calculation of collision risk for informed decision-making.
  • Empirical results demonstrate doubled rewards and halved collision rates in simulations, highlighting the method's robustness and scalability.

Overview

Dynamic obstacle avoidance (DOA) is essential for the safety of autonomous vehicles in environments like waterways, airspace, or on the road. The paper introduces a novel two-step approach that combines supervised and reinforcement learning to handle DOA tasks. The proposed method assesses collision risks and utilizes this information to improve the decision-making of an autonomous agent. The paper proves the approach's capability to substantially reduce the number of collisions, implying its potential for enhancing autonomous navigation systems.

Step 1: Estimating Collision Risk

The first step leverages supervised learning to predict the trajectory of an obstacle, accounting for potential non-linear movements. Specifically, a recurrent neural network (LSTM) estimates potential future paths based on past observations, allowing for calculation of collision risk metrics. This method captures complex movements and provides valuable insights into potential points of closest approach (CPA).

Step 2: Integration with Reinforcement Learning

With collision risk estimates at hand, the second step incorporates these metrics into the observation space of a reinforcement learning (RL) agent. By augmenting the observation space with the LSTM-generated estimates, the RL agent gains a richer understanding of the environment. This enhanced observation enables the agent to make better-informed decisions, effectively doubling the rewards and halving the collision rates in simulations.

Empirical Performance

The effectiveness of the two-step architecture was rigorously tested in a challenging environment, simulating the movement of multiple obstacles using both stochastic and periodic patterns. Independent of the choice of RL algorithm, integrating collision risk estimates consistently improved performance. The paper's findings emphasize the proposed method's algorithmic robustness and scalability, demonstrating its potential for a wide range of applications in autonomous systems.

Conclusion

The research concludes that autonomous systems could significantly benefit from employing the two-step architecture for dynamic obstacle avoidance. Future work will focus on enhancing the architecture with real-world data and exploring the influence of uncertainty in collision risk estimates on decision-making. The improvements in safety and efficiency brought by this approach contribute meaningfully towards the development of advanced autonomous transportation networks.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown