Probabilistic Shielding for Safe Reinforcement Learning (2503.07671v3)

Abstract: In real-life scenarios, a Reinforcement Learning (RL) agent aiming to maximise their reward, must often also behave in a safe manner, including at training time. Thus, much attention in recent years has been given to Safe RL, where an agent aims to learn an optimal policy among all policies that satisfy a given safety constraint. However, strict safety guarantees are often provided through approaches based on linear programming, and thus have limited scaling. In this paper we present a new, scalable method, which enjoys strict formal guarantees for Safe RL, in the case where the safety dynamics of the Markov Decision Process (MDP) are known, and safety is defined as an undiscounted probabilistic avoidance property. Our approach is based on state-augmentation of the MDP, and on the design of a shield that restricts the actions available to the agent. We show that our approach provides a strict formal safety guarantee that the agent stays safe at training and test time. Furthermore, we demonstrate that our approach is viable in practice through experimental evaluation.

Probabilistic Shielding for Safe Reinforcement Learning

The research paper "Probabilistic Shielding for Safe Reinforcement Learning" authored by Edwin Hamel-De le Court, Francesco Belardinelli, and Alexander W. Goodall presents a methodological advancement in the domain of Safe Reinforcement Learning (Safe RL). This paper proposes a scalable approach to Safe RL that involves probabilistic shielding within Markov Decision Processes (MDPs), enhancing both training and deployment phases' safety guarantees.

Problem Context and Objectives

Safe RL concerns the development of RL agents that optimize performance within predefined safety constraints, a crucial requirement for real-world applications such as autonomous driving and industrial automation. Traditional approaches often resort to linear programming methods to impose safety constraints, which inherently limit scalability due to computational complexity. This paper emphasizes developing a technique that not only respects safety requirements but also scales with problem size and complexity.

Approach and Methodology

The paper introduces a novel probabilistic shielding method, fundamental to ensuring state safety characterized by probabilistic avoidance metrics within MDP frameworks. The approach hinges on three core components:

1. State-Augmentation: The authors propose enhancing each state with a "safety level," representing a probabilistic threshold of evading unsafe states.
2. Shield Construction: A shield is designed that systematically restricts actions available to the agent based on safety constraints. This shield enforces behavior that avoids entering unsafe states with probabilities above a dictated threshold.
3. Value Iteration Algorithms: Where previous methods might employ linear programming, this research shifts to sound value iteration techniques which afford scalability improvements. Techniques such as Interval Iteration and Sound Value Iteration are leveraged to compute safety costs with formal approximation guarantees.

The shield transforms the original MDP by augmenting each state with safety metrics and thereby allowing any RL algorithm like PPO or A2C to solve the shielded MDP without incurring constraint violations.

Main Contributions

The paper delineates four primary contributions:

• Design of a novel shield for finite MDPs, facilitating a state-augmented safety-aware exploration strategy.
• Formal proof of the shield's effectiveness in preserving agent safety throughout the RL exploration process.
• Demonstration that the quest for an optimal policy within safety constraints translates to solving a policy optimization problem within the constructed shield.
• A practical guide for implementing this shield as a gym environment for RL experiments, ensuring ease of replication and application.

Empirical Results

The empirical section evaluates the proposed approach across several benchmark environments. These include gridworld scenarios and a classic media streaming task, where safety equates to avoiding specific hazardous states. The experiments show that PPO-Shield effectively maintains safety constraints throughout training and final policy performance, consistently outperforming mix-in constraint-based RL algorithms such as PPO-Lagrangian and CPO in terms of safety adherence and reward optimization.

Implications and Future Work

Practically, the proposed probabilistic shielding adds a significant layer of reliability to RL systems used in safety-critical tasks. The development of efficient algorithms for calculating the safety measures through value iteration without compromising scalability opens new avenues for practical implementations of Safe RL in various domains.

Theoretically, this work can inspire further research into more nuanced safety constraints and more efficient computation strategies within different types of RL environments. The probabilistic approach demonstrates promise for more generalized safe-learning frameworks, potentially extending to multi-agent systems or partially observable environments.

Conclusion

This paper marks a step forward in addressing the dual objectives of safety and optimality in RL settings by proposing a scalable probabilistic shielding method. The approach's sound theoretical underpinning, coupled with empirical validation, accentuates its potential to influence future developments in the field of Safe RL and beyond.