Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents (2410.11671v2)

Abstract: Reinforcement learning (RL) controllers are flexible and performant but rarely guarantee safety. Safety filters impart hard safety guarantees to RL controllers while maintaining flexibility. However, safety filters can cause undesired behaviours due to the separation between the controller and the safety filter, often degrading performance and robustness. In this paper, we analyze several modifications to incorporating the safety filter in training RL controllers rather than solely applying it during evaluation. The modifications allow the RL controller to learn to account for the safety filter, improving performance. This paper presents a comprehensive analysis of training RL with safety filters, featuring simulated and real-world experiments with a Crazyflie 2.0 drone. We examine how various training modifications and hyperparameters impact performance, sample efficiency, safety, and chattering. Our findings serve as a guide for practitioners and researchers focused on safety filters and safe RL.

• The paper introduces action filtering, which blocks unsafe actions during training to prevent violations in real-world systems.
• The paper implements correction penalization to reward policies that require fewer safety corrections, promoting inherently safe behavior.
• The paper employs safe environment resets to restart episodes from certified safe states, enhancing sample efficiency and reducing constraint violations.

Safety Filtering While Training: Enhancing Reinforcement Learning Performance and Efficiency

The research presented in this paper highlights the integration of safety filters within the training processes of reinforcement learning (RL) agents to improve their performance and sample efficiency. The authors aim to address a significant limitation of conventional RL, which is the lack of inherent safety guarantees, particularly in safety-critical applications like autonomous driving and medical procedures.

Key Contributions

The paper proposes three specific methodologies for incorporating safety filters during the training of RL controllers:

1. Action Filtering: By applying safety filters during training, potentially unsafe actions generated by the RL agent are either corrected or blocked, ensuring safety while the agent learns. The output actions applied are safe, thus preventing violations and allowing for training directly on physical systems.
2. Correction Penalization: The authors introduce a reward penalization mechanism based on the extent of corrections made by safety filters. This approach incentivizes the learned policy to generate actions that align with safety constraints, minimizing the filter’s intervention in execution.
3. Safe Environment Resets: Episodes are restarted from certifiably safe states by utilizing safety filters to identify these states. This ensures the RL agent is always in a feasible state space, enhancing learning efficiency by focusing on realistic scenarios.

Experimental Evaluation

The effectiveness of these proposed methods was evaluated through both simulation and real-world experiments using a Crazyflie 2.0 drone. Several benchmarks were employed, demonstrating significant improvements over standard RL training practices:

• Improved Sample Efficiency and Performance: In the experiments, these methods reduced the required environment interactions and enhanced the training speed. This is particularly crucial in real-world applications where data collection is expensive and time-consuming.
• Reduction in Constraint Violations: By integrating safety concepts from the very start of the training process, the proposed approach ensures that constraint violations are minimized if not entirely avoided. This is a major step forward for training RL directly on physical systems.
• Chattering Mitigation: The methodologies reduce the norm of the rate of change of inputs, addressing issues related to chattering and ensuring that the learned policies operate more smoothly and effectively.

Implications and Future Prospects

The research demonstrates that RL systems can be effectively adapted for safety-critical tasks without sacrificing their adaptability and performance. The integration of safety filters during training provides robustness not typically seen in traditional RL algorithms. This approach mitigates risk during both the training and execution phases, presenting a compelling case for deploying RL in environments where safety cannot be compromised.

Looking forward, this integration could be refined further, perhaps by exploring different types of safety filters such as control barrier functions or Hamilton-Jacobi-based methods to optimize computational efficiency. Additionally, applying these concepts in various dynamic environments and with different RL architectures could further verify and expand their applicability.

Conclusion

This paper contributes significant advancements in the intersection of RL and safety assurance, providing methodologies that enhance both sample efficiency and performance while ensuring compliance with safety constraints. These approaches hold promise for transforming the application scope of RL in domains where safety is paramount, paving the way for broader and more reliable real-world deployments.