Modular Control Architecture for Safe Marine Navigation: Reinforcement Learning and Predictive Safety Filters (2312.01855v2)

Abstract: Many autonomous systems face safety challenges, requiring robust closed-loop control to handle physical limitations and safety constraints. Real-world systems, like autonomous ships, encounter nonlinear dynamics and environmental disturbances. Reinforcement learning is increasingly used to adapt to complex scenarios, but standard frameworks ensuring safety and stability are lacking. Predictive Safety Filters (PSF) offer a promising solution, ensuring constraint satisfaction in learning-based control without explicit constraint handling. This modular approach allows using arbitrary control policies, with the safety filter optimizing proposed actions to meet physical and safety constraints. We apply this approach to marine navigation, combining RL with PSF on a simulated Cybership II model. The RL agent is trained on path following and collision avpodance, while the PSF monitors and modifies control actions for safety. Results demonstrate the PSF's effectiveness in maintaining safety without hindering the RL agent's learning rate and performance, evaluated against a standard RL agent without PSF.

• The paper demonstrates a modular control framework that fuses RL with a Predictive Safety Filter to ensure collision avoidance and accurate path following in complex marine settings.
• It employs an Optimal Control Problem approach that minimally adjusts the RL agent's actions while meeting both physical and safety constraints, including dynamic obstacles.
• Simulation results validate that the PSF enhances safety without hindering the RL agent's learning performance, paving the way for robust autonomous surface vessel navigation.

Modular Control Architecture for Safe Marine Navigation

Introduction

The rapid advancement of AI has led to significant developments in autonomous systems, particularly in marine navigation. Autonomous Surface Vessels (ASVs), which operate in complex marine environments, must tackle non-linear and uncertain dynamics influenced by time-varying disturbances such as waves, currents, and wind. Ensuring the safety and stability of such systems, especially when using Reinforcement Learning (RL) for control, is a crucial challenge. Recently, Predictive Safety Filters (PSF) have surfaced as a method to ensure safety without explicitly integrating constraints within the RL algorithm.

Predictive Safety Filtering Combined with Reinforcement Learning

Innovatively, the research unites Reinforcement Learning (RL) with a Predictive Safety Filter to address path following and collision avoidance in marine navigation. The PSF assesses the control actions proposed by the RL agent, permitting only those that maintain the system's safety. The RL agent is trained using a simulated model of a supply ship, known as Cybership II, across a broad spectrum of environmental conditions. The effectiveness of the combined PSF/RL system is shown through simulations. It was found that integrating a PSF successfully retains safety without deterring the learning performance or progress of the RL agent.

Safe Marine Navigation Strategy

The paper formulates the PSF as an Optimal Control Problem, which ensures minimal alterations to the agent's proposed actions while satisfying both physical and safety constraints. Specifically for ASVs, the research innovates by also accounting for moving obstacles like other vessels. This is achieved by incorporating their projected motions within the PSF to compute safe trajectories. Additionally, the approach includes a collision risk metric, which utilizes a blend of ship motion predictions and environmental perceptions to achieve risk-aware autonomous navigation.

Results and Implications for Future Research

The safety performance of the combined PSF/RL scheme is assessed against traditional RL-based control strategies. Results from numerous simulated scenarios reveal that the PSF is highly effective in preventing collisions. This work exemplifies the potential of integrating PSFs with RL algorithms to create robust and safe autonomous navigation systems for ASVs. It opens new avenues for future research, such as improving the realism of training scenarios and exploring multi-agent collaborative behaviors for collective safety and efficiency in marine traffic management.