Safe Reinforcement Learning: An Overview

Updated 2 December 2025

Safe RL is a reinforcement learning framework that integrates explicit safety constraints to generate policies avoiding unsafe actions during both training and deployment.
It employs methods like shielding, risk estimation, and control barrier functions within the CMDP framework to provide formal probabilistic and worst-case safety guarantees.
Practical implementations in robotics and simulation show that Safe RL balances exploration with safety, achieving higher rewards and reduced violations across complex tasks.

Safe Reinforcement Learning (Safe RL) is an area of reinforcement learning focused on synthesizing optimal or near-optimal policies that maintain strict safety guarantees during both learning and deployment. Unlike standard RL, which often tolerates or even depends on unconstrained exploration that may violate safety constraints, Safe RL frameworks integrate explicit mechanisms—algorithmic, optimization-based, or data-driven—for certifying and enforcing safety criteria, often in formal probabilistic or worst-case terms.

1. Mathematical Foundations and Problem Formulations

Safe RL is typically formalized using the Constrained Markov Decision Process (CMDP) framework, where the agent seeks to maximize expected cumulative reward while satisfying constraints on trajectory-level costs or state-occupancy (Li et al., 2023, Jansen et al., 2018, Chen et al., 2023, Jeddi et al., 2021, Wachi et al., 2020). Formally, for states $s \in S$ , actions $a \in A$ , transitions $P(s'|s,a)$ , reward $r(s,a)$ , constraint cost $c(s,a)\geq0$ , discount $\gamma$ : $\max_\pi\, J(\pi)\;=\;\mathbb{E}\bigl[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\bigr]\quad \text{s.t.}~\,C(\pi) = \mathbb{E}\left[\sum_{t=0}^\infty \gamma^t c(s_t,a_t)\right]\leq \kappa.$ Safety violations may be encoded as cost constraints on trajectories, state-action reachability, or chance-constraints (e.g., probability of entering an unsafe set $U$ limited to $\delta$ ) (Jansen et al., 2018, Chen et al., 2023). In partially observable or continuous domains, Safe RL augments this framework with additional modeling, such as POMDPs, predictive state representations, or constraint sets over beliefs or observed histories (Cheng et al., 2023, Jeddi et al., 2021).

2. Core Methodological Approaches

Safe RL literature develops a taxonomy of approaches for integrating safety:

Shielding and Safety Filters:
- Probabilistic Shields: Precompute, via formal verification or model checking, the risk of each action relative to entering unsafe states, then block actions (replacing $A(s)$ by $A_\mathrm{safe}(s) = \{a \mid p_{s,a} \le \delta\}$ ) (Jansen et al., 2018).
- Confidence-Based Filters: Construct a backup policy certified to be safe using probabilistic model uncertainty and minimally modify the agent's action online, ensuring high-probability satisfaction of state constraints (Curi et al., 2022).
- Data-Driven Predictive Control: Use real-time reachability or model-predictive safety layers based on data-driven linear model identification to project unsafe actions onto the closest safe alternatives (Selim et al., 2022, Selim et al., 2022).
- Control Barrier Functions: Embed RCBFs as a differentiable QP layer after the RL policy, guaranteeing forward invariance of the certified safe set under disturbances (Emam et al., 2021).
Risk Estimation and Penalization:
- Contrastive Risk Classification: Learn a classifier online to estimate the probability that $(s,a)$ leads to an unsafe event, using predicted risk for both early termination of rollouts and reward shaping via Lagrangian penalty (Zhang et al., 2022).
- Risk Predictive Value Functions and Critics: Model the advantage or value functions for safety cost, using hierarchical or surrogate chance constraints and adjust actions via projection onto safe sets (Chen et al., 2023).
Offline/Hybrid Safe RL:
- Demonstration-Guided Distillation: Bootstrap RL with high-capacity offline policies (e.g. Decision Transformers), then distill into lightweight, safe policies through guided online optimization (Li et al., 2023, Quessy et al., 8 Jan 2025).
- Skill-Based and PU-Learned Risk Estimation: Learn skill-level risk predictors (e.g., via positive-unlabeled learning) from offline data for skill-level safe exploration and risk-aware policy optimization (Zhang et al., 2 May 2025).
- Return-Conditioned Supervised Safe RL: Optimize a set of target returns (reward/cost) using supervised offline training, then safely adapt a small parameterization online via GP-UCB, with end-to-end high-probability guarantees (Wachi et al., 28 May 2025).
Model-Based and Formal Uncertainty Quantification:
- Lyapunov-Based Local Constraints: Convert trajectory-level constraints to local linear inequalities via Lyapunov functions, combined with epistemic uncertainty quantification (e.g., neural ensembles + dropout) to promote risk-averse actions (Jeddi et al., 2021).
- Safe RL in Tensor RKHS: Leverage predictive state embeddings and kernel Bayes rule to analytically estimate future (multi-step) costs/risks directly from histories, guaranteeing almost-sure safety under unknown POMDP dynamics (Cheng et al., 2023).
Safe Exploration via Safe-Set Expansion and Optimistic Planning:
- Safe-Set Learning and Expansion: Use Gaussian Process regression to first expand a certified safe set (via high-confidence lower bounds on unknown safety signals), then optimize within this set, never visiting unverified unsafe states (Wachi et al., 2020, Quessy et al., 8 Jan 2025).
- Optimistic Forgetting: Periodically remove poor-performing episodes from the buffer to prevent collapse of the estimated safe set under limited demonstrations (Quessy et al., 8 Jan 2025).

3. Safety Certification, Theoretical Guarantees, and Trade-offs

Rigorous guarantees—either probabilistic or worst-case—are central to Safe RL methodologies:

Action-Level Safety: Probabilistic shields guarantee $\Pr^\pi_s(\lozenge U) \leq \delta$ for all time and all learning policies, decoupling safety enforcement from the RL policy (Jansen et al., 2018); similar hard guarantees hold for Lyapunov and RCBF-based approaches under model and implementation assumptions (Emam et al., 2021, Jeddi et al., 2021).
Sample-Complexity and Optimality: Provided sufficient data/model coverage, safe-set expansion and return-conditioned methods guarantee that learned policies are both safe and $\epsilon$ -optimal within the certified safe region, with polynomial sample complexity in the size of the problem and regularity parameters (Wachi et al., 2020, Wachi et al., 28 May 2025, Cheng et al., 2023).
Expressivity–Safety Trade-offs:
- Shields' conservativeness can impede exploration, while more permissive thresholds increase risk (Jansen et al., 2018).
- Confidence-based and risk-predictive policies allow explicit risk–reward trade-offs by tuning filter parameters or surrogate risk budgets (Chen et al., 2023, Zhang et al., 2022).
- Minimal-intervention safety layers strive to minimally alter the nominal policy to preserve performance within safety.
Partial Observability & Out-of-Distribution Generalization:
- Transformer architectures and attention-based memory modules are employed for safety in partially observable or memory-based tasks, improving both return and constraint satisfaction (Jeddi et al., 2021).
- Predictive-state-based analytic safe RL (Cheng et al., 2023) circumvents belief over system states, directly embedding uncertain observation predictions via RKHS for provably analytic updates.

4. Practical Implementations and Empirical Evidence

Safe RL algorithms have been validated in a wide range of simulation and real-world environments:

Robotic Benchmarks: Simulated tasks in Safety Gym, Bullet-Safety-Gym, MuJoCo (Ant, Hopper, Cheetah, Humanoid), and real robot box-pushing with Franka Panda arms (Yang et al., 2023, Kovač et al., 2023, Li et al., 2023, Selim et al., 2022).
Performance Metrics: Studies report episode return, total constraint violation, reward/violation trade-off ratios, and in many cases, real-time feasibility (e.g., RAG control at 30 ms/step; BRSL layer at 30–70 ms/step) (Li et al., 2021, Selim et al., 2022, Selim et al., 2022).
Comparative Results: Across tasks, shielded and safety-filtered RL consistently achieves lower violation rates, higher final reward, and faster convergence relative to constraint-agnostic RL or Lagrangian CMDP baselines (Jansen et al., 2018, Li et al., 2021, Zhang et al., 2022, Li et al., 2023).
Offline Data Constraints: The quantity and diversity of offline demonstrations (or safe-set points) are empirically shown to be key for bootstrapping online safe RL in high-dimensional and spatially extended tasks, motivating unsupervised data collection and forgetting strategies under limited supervision (Quessy et al., 8 Jan 2025).

5. Distinctive Algorithmic Features

A cross-section of representative Safe RL frameworks is summarized below.

Method/Concept	Guarantee/Formulation	Design Principle/Integration
Probabilistic Shield (Jansen et al., 2018)	$\Pr^\pi_s(\lozenge U) \leq \delta$	Precompute per-action risk via model checking, filter action set per query
Data-Driven Predictive Control (Selim et al., 2022)	Hard invariance wrt offline safe set	On-policy reachability via zonotopes, safety QP filters risky actions
Skill Risk PU Learning (Zhang et al., 2 May 2025)	Risk-averse skill selection	PU-learned skill-level risk predictor from demos, skill selection via CEM risk minimization
Guided Online Distillation (Li et al., 2023)	CMDP constraint bounded	Offline expert (DT) guidance, online distillation ensures constraint adherence
Return-Conditioned Safe RL (Wachi et al., 28 May 2025)	High-probability safe deployment	Offline RL (RCSL), online GP-UCB safe/reward maximization in target-return space
Lyapunov-Uncertainty Safe RL (Jeddi et al., 2021)	Feasible at each step via local constraints	Lyapunov-transformed constraints, ensemble + dropout for risk estimation, GTrXL memory
Black-box Reachability Layer (Selim et al., 2022)	Hard reachability-based safety	Ensemble NN dynamics, differentiable collision LP, online safe projection

6. Limitations and Future Directions

Key limitations highlighted across the literature include:

Conservatism vs. Exploration: Overly restrictive shields or risk-averse policies can diminish exploration and slow convergence (Jansen et al., 2018, Chen et al., 2023).
Data/Efficient Coverage: Safe set learning is fundamentally limited by the quality and coverage of available (often offline) data; unsupervised skill discovery can partially address this but may incur additional sample complexity (Quessy et al., 8 Jan 2025, Zhang et al., 2 May 2025).
Model/Assumption Dependence: Some guarantees require linearity, known bounded disturbances, or accurate learned models; generalization to nonlinear, black-box, and high-dimensional settings often relies on local linearization, zonotopic approximations, or expansive function classes (Selim et al., 2022, Cheng et al., 2023, Li et al., 2021, Emam et al., 2021).
Computational Overhead: Certain formulations (e.g., MIQP, forward reachability, kernel Gram matrix inversion) can be computationally intensive, motivating approximate or amortized solutions (Selim et al., 2022, Cheng et al., 2023).
Adaptability to Distribution Shift: Online adaptation or bootstrapping (e.g., updating risk predictors or switching off shields) remains an active area for efficient transition from shielded to unconstrained policies (Zhang et al., 2 May 2025, Li et al., 2023).

Current directions include integrating risk quantification with more adaptive exploration, scaling safe RL to partially observable and multi-agent systems, developing efficient offline safe data collection paradigms, and bridging toward formal safety verification in continuous and uncertain domains.

7. Representative Empirical Outcomes and Comparative Performance

Empirical studies document substantial improvements in both safety and reward relative to unconstrained RL across diverse tasks. For example:

Probabilistic Shield in PAC-MAN: Shielded RL achieved average score $+340$ versus $-130$ for unshielded, with win rate $40-80\%$ (vs. $0\%$ unshielded) (Jansen et al., 2018).
Guided Distillation in Car-Circle (Safe RL):
- CVPO: $r\approx502,~c\approx7.4$ ;
- GOLD(DT-IQL): $r\approx688,~c\approx3.2$ (Li et al., 2023).
Safe Set Learning (Mars Exploration): SNO-MDP attained $0.81\times$ oracle reward with $0$ unsafe actions, outperforming nonoptimistic methods (Wachi et al., 2020).
Skill-Based Risk Planning: SSkP obtained higher reward for equal or fewer violations compared to CPQ/SMBPO/Recovery RL in MuJoCo (Zhang et al., 2 May 2025).
Test-Time Violation (Trajectory Optimization): Safety-Embedded MDP exhibited near-zero test-time cost versus substantial violations for CPO or PPO-Lagrangian in multi-agent Safety Gym (Yang et al., 2023).

In summary, modern Safe RL methodologies unite formal safety certification, probabilistic risk assessment, filtering and projection layers, and demonstration-guided learning to achieve near-optimal policy reward under strict safety constraints, with increasing maturity for deployment across complex robotic and real-world environments.