---
title: Offline Safe Reinforcement Learning (OSRL)
url: https://www.emergentmind.com/topics/offline-safe-reinforcement-learning-osrl
type: topic
---

# Offline Safe Reinforcement Learning (OSRL)

Offline Safe Reinforcement Learning (OSRL) studies how to learn a policy from a fixed dataset of prior experience while maximizing task return under explicit safety constraints, without further interaction with the environment. In the dominant formulation, the environment is a constrained Markov decision process (CMDP) with reward \(r\), cost \(c\), and safety threshold \(\kappa\); the learner must optimize return while keeping cumulative cost below the threshold, despite distribution shift and incomplete support in the offline data [2306.09303]. Recent work has expanded this core formulation in several directions: distribution-constrained CMDPs that require proximity to the behavior policy, trajectory-level and hard-safety formulations, reach-avoid specifications, risk-sensitive objectives such as CVaR, and deployment-time budget adaptation [2412.15429].

## 1. Formal setting and problem variants

The standard OSRL objective is
\[
\max_{\pi}\ \mathbb{E}_{\tau\sim\pi}[R(\tau)] \quad \text{s.t.} \quad \mathbb{E}_{\tau\sim\pi}[C(\tau)] \le \kappa,
\]
with \(R(\tau)=\sum_t \gamma^t r_t\) and \(C(\tau)=\sum_t \gamma^t c_t\), learned from a fixed offline dataset such as \(\mathcal D=\{(s_i,a_i,r_i,c_i,s'_i)\}_{i=1}^n\) or trajectory collections \(D=\{\tau_1,\tau_2,\dots\}\) [2306.09303]. Several papers further impose an offline-distribution restriction, for example \(D_{KL}(\pi\|\pi_\beta)\le \varepsilon\) or \(\mathcal D(\pi\|\pi_\beta)\le \epsilon\), to control extrapolation beyond the behavior policy that generated the data [2412.15429].

The safety notion is not uniform across the literature. Much of the benchmarked CMDP work uses discounted expected cost constraints, but other formulations require stricter guarantees. TREBI imposes a trajectory-level condition,
\[
\forall \tau\sim\pi,\ C(\tau)\le b,
\]
to support real-time budget adaptation [2306.00603]. SafeFQL uses a hard state-safety requirement,
\[
x_t \notin \mathcal F,\ \forall t\ge 0,
\]
where \(\mathcal F=\{x\mid \ell(x)>0\}\) is a failure set [2603.15136]. Shielding-based OSRL encodes safety as a reach-avoid specification \(\varphi=\neg S_U\ \mathsf U\ S_T\), meaning unsafe states must be avoided until target states are reached [2605.10293]. Risk-averse offline RL replaces expected return with a distortion objective, notably CVaR, to emphasize lower-tail behavior rather than only the mean [2102.05371].

These formulations are not interchangeable. Expected-cost CMDPs, hard-state safety, reach-avoid specifications, and CVaR all define different acceptable behaviors. This suggests that OSRL is better understood as a family of offline constrained decision problems than as a single optimization template.

## 2. Why offline safety is difficult

OSRL inherits the central offline RL problem of distribution shift, then compounds it with safety estimation. Because the learner cannot test actions online, errors in reward or cost estimation on unsupported state-action pairs can directly induce unsafe policies. The benchmark paper emphasizes that OSRL is difficult in both training and deployment: training is limited by fixed logged trajectories, extrapolation error, and incomplete coverage of the reward-cost trade-off surface, while deployment must tolerate unseen states, unseen safety thresholds, stochasticity, and noisy or outlier data [2306.09303].

A recurrent criticism of earlier methods is that they approximate safety locally when the relevant notion is often global. TraC states that safety is a trajectory-level property, whereas many OSRL algorithms enforce local per-step cost constraints derived from global constraints; this can produce either overly conservative policies or policies that violate safety constraints [2412.15429]. The same tension appears in hard-safety work: FISOR argues that soft expectation constraints can still permit violations on some trajectories or states, while SafeFQL argues that expected-cost objectives can be insufficient for safety-critical real-time control [2401.10700].

Another difficulty is dataset mismatch. OASIS formalizes “Safe Dataset Mismatch” for tempting datasets, which contain high-reward unsafe behavior, and conservative datasets, which contain low-cost but also low-reward behavior; direct regularization toward the empirical behavior policy can therefore pull learning toward the wrong part of the reward-cost frontier [2407.14653]. CPQ emphasizes an adjacent problem: mixed behavior data may contain both safe low-reward and unsafe high-reward trajectories, so simply staying close to the data distribution can preserve unsafe tendencies rather than eliminate them [2107.09003].

A further challenge is deployment-time variability. CAPS and TREBI both target settings where the cost threshold changes after training, a regime in which a policy specialized for one threshold may become unsafe under tighter budgets or unnecessarily conservative under looser ones [2412.18946]. Shielding-based offline RL identifies an additional structural issue: sparse data and coverage gaps make both transition dynamics and value estimates highly uncertain, while offline learning offers no online correction mechanism [2605.10293].

A common misconception is that OSRL is merely offline RL with an extra cost term. The benchmark study, CPQ, TraC, and hard-safety methods all reject that view: cost estimation, safety semantics, and support restrictions alter both the optimization objective and the failure modes [2306.09303].

## 3. Methodological paradigms

The literature spans several distinct design patterns.

| Paradigm | Representative methods | Defining mechanism |
|---|---|---|
| Penalized or pessimistic Q-learning | CPQ [2107.09003], WSAC [2401.00629] | Inflate or adversarially estimate cost to suppress unsafe/OOD actions |
| Sequence or conditional trajectory modeling | CDT [2302.07351], GAS [2602.05323] | Condition policies on desired reward and cost returns |
| Feasibility or hard-safety generative modeling | FISOR [2401.10700], SafeFQL [2603.15136] | Learn feasible or reachable safe regions, then optimize reward within them |
| Distribution shaping or generative data synthesis | OASIS [2407.14653] | Generate a shaped offline dataset closer to the desired safe-reward distribution |
| Trajectory-level classification | TraC [2412.15429] | Label full trajectories as desirable or undesirable and train a classifier-guided policy |
| Minimax and primal-dual offline optimization | O3SRL [2510.22027] | Solve a Lagrangian minimax problem with offline RL and no-regret online optimization |
| Adaptive deployment-time policy selection | CAPS [2412.18946], TREBI [2306.00603] | Switch policies or plan trajectories according to the current budget |
| Shielding and robust model checking | Shielded SPIBB [2605.10293] | Build an IMDP from offline data and filter actions by robust reach-avoid safety |
| Lexicographic safety hierarchy | LexiSafe [2602.17312] | Optimize safety first, then reward inside the safety-feasible set |

Penalized value-based methods explicitly treat unsupported actions as dangerous. CPQ learns a cost critic with an extra term that raises the estimated cost of OOD actions and then uses that critic to filter reward backups and actor improvement, effectively making OOD actions unsafe by construction [2107.09003]. WSAC instead frames OSRL as a two-player Stackelberg game between an actor and adversarial reward and cost critics, seeking improvement over a reference policy while maintaining the same safety level [2401.00629].

Sequence-modeling approaches condition on reward and cost targets rather than solving a constrained Bellman problem directly. CDT extends Decision Transformer with cost-return tokens and uses return relabeling so the policy can dynamically adjust the safety-performance trade-off at deployment without retraining [2302.07351]. GAS retains the goal-conditioned flavor but argues that transformer-style generative methods lack stitching ability; it therefore augments and relabels the dataset at the transition level and learns reward and cost goal functions via expectile regression [2602.05323].

Hard-safety and feasibility methods replace expected-cost reasoning with safe-set or reachability reasoning. FISOR uses Hamilton-Jacobi reachability to identify the largest feasible region, then derives an optimal policy of weighted behavior cloning form that maximizes reward in feasible states and minimizes risk in infeasible states [2401.10700]. SafeFQL uses a reachability-inspired safety critic with a max-backup recursion, a feasibility gate that prioritizes safety before reward, and conformal prediction to calibrate the learned safety boundary [2603.15136].

Trajectory-level and preference-style methods operate on whole episodes rather than local transitions. TraC partitions the offline dataset into desirable trajectories, defined as safe and high-reward, and undesirable trajectories, defined as unsafe or low-reward safe trajectories, then trains a classifier-guided policy with a trajectory score tied to maximum-entropy RL and regret-based preference modeling [2412.15429].

Adaptive and planning-based methods address changing budgets explicitly. CAPS learns a small family of policies spanning reward-cost trade-offs, then at test time filters candidate actions using a learned cost \(Q\)-function and selects the feasible action with the highest reward estimate [2412.18946]. TREBI instead treats the problem as trajectory-distribution optimization and performs budget-conditioned diffusion planning, aiming for per-trajectory budget satisfaction under changing real-time thresholds [2306.00603].

## 4. Safety notions and theoretical guarantees

The theoretical landscape is heterogeneous because the guarantees target different objects. Some methods guarantee safe policy improvement relative to a baseline, some guarantee approximate primal-dual optimality, some provide high-probability shielding soundness, and some provide deployment-time regret bounds.

WSAC gives a reference-policy guarantee: under its refined objective, the learned policy is no worse than the behavior policy in reward, \(J_r(\hat\pi^*)\ge J_r(\mu)\), while positive safety violation is bounded by \(\frac{1}{\lambda(1-\gamma)}+\{J_c(\mu)\}_+\) [2401.00629]. Shielded SPI provides a different type of statement. It builds an interval MDP from offline data, proves that the true MDP lies inside the interval model with high probability, and then shows that every action admitted by the shield is safe in the true MDP with probability at least \(1-\delta\), yielding a PAC-style safety filter for reach-avoid objectives [2605.10293].

LexiSafe derives safety-violation and performance-suboptimality bounds for its single-cost formulation and extends the analysis to multi-cost hierarchical safety. The stated sample-complexity result for LexiSafe-SC is
\[
N=\tilde{\mathcal O}\!\left(\frac{T^4}{\epsilon^2}\cdot \max\{c_m^2\mathcal C_{\text{safe}}d_\theta L,\ r_m^2\mathcal C_{\text{perf}}d_\theta L\}\right),
\]
with a corresponding linear-in-\(K\) extension for the multi-cost case [2602.17312]. O3SRL, by contrast, analyzes OSRL as a dual minimax problem over distributions over policies and Lagrange multipliers. Its averaged output is an \(\epsilon\)-approximate equilibrium with
\[
\epsilon=(n)+\frac{R_T(\Lambda)}{T},
\]
and its discrete-EXP3 approximation yields an error roughly \(\mathcal O((n)+\sqrt{K/T+1/K})\) while eliminating the need for offline policy evaluation [2510.22027].

SafeFQL’s guarantee is finite-sample probabilistic safety coverage rather than constrained-optimality. After conformal calibration, it states that with probability at least \(1-\beta_s\), states in the calibrated safe set satisfy \(\mathbb P(V_c(x_i)<0)\ge 1-\epsilon_s\) [2603.15136]. SOReL addresses a different question again: whether an offline policy is safe to deploy before any online interaction. It bounds regret in terms of posterior information loss and uses posterior predictive uncertainty to estimate deployment-time regret entirely offline [2505.22442].

These results are not directly comparable. Some depend on safe baseline policies, some on concentrability and function-class assumptions, some on known safe and unsafe state sets and transition graphs, and some on accurate world models. FISOR explicitly notes that it does not claim a formal global zero-violation theorem for the learned deep model in the full offline setting, despite its hard-constraint formulation [2401.10700]. A plausible implication is that OSRL theory currently offers a menu of conditional guarantees rather than a single dominant certification framework.

## 5. Benchmarks, metrics, and empirical regularities

A major step in consolidating OSRL as a field was the introduction of DSRL, a benchmark ecosystem consisting of FSRL for safe-policy generation, DSRL for datasets and environment wrappers, and OSRL for baseline implementations [2306.09303]. The benchmark reports over 75,000 trajectories across 38 tasks from SafetyGymnasium, BulletSafetyGym, and MetaDrive, with data post-processing filters for density, partial discarding, and noise injection [2306.09303]. Standard metrics are normalized reward and normalized cost, and a policy is operationally treated as safe when normalized cost is at or below \(1\) [2306.09303].

The original DSRL comparison already exposed a persistent empirical pattern. BC-All often attains higher reward but violates safety, BC-Safe is reliably safe but conservative, CDT usually balances reward and safety better than pure behavior cloning, and Q-learning-based methods such as BCQ-Lag, BEAR-Lag, and CPQ are inconsistent across tasks [2306.09303]. That qualitative pattern recurs in later studies, although the ranking among stronger recent methods depends on the safety notion and deployment protocol.

Later methods report improvements under different evaluation settings. TraC, evaluated on the DSRL benchmark with 38 tasks across SafetyGymnasium, BulletSafetyGym, and MetaDrive, reports higher rewards and better constraint satisfaction than competitive baselines, with averaged results of reward \(0.40\), cost \(0.92\) on SafetyGym, reward \(0.61\), cost \(0.52\) on BulletGym, and reward \(0.35\), cost \(0.05\) on MetaDrive [2412.15429]. CAPS, also evaluated on 38 DSRL tasks but targeting varying deployment-time thresholds, reports that CAPS(IQL) is safe on 34/38 tasks and achieves the highest reward on 18 tasks, whereas CDT is safe on 19/38 tasks [2412.18946]. O3SRL focuses on the DSRL Bullet benchmark with eight tasks under a stringent budget \(\kappa=5\) and reports being the only method that consistently satisfies the cost constraint across all eight tasks [2510.22027].

Hard-safety and generative methods use partially overlapping but not identical protocols. FISOR evaluates on 26 DSRL tasks from Safety-Gymnasium, Bullet-Safety-Gym, and MetaDrive and reports being the only method that achieves safety satisfaction in all evaluated tasks while attaining the highest return in most tasks [2401.10700]. OASIS, on Bullet-Safety-Gym datasets constructed to be full, tempting, conservative, or hybrid, reports safe normalized costs below threshold on all reported tasks and the highest reward among safe agents in most settings [2407.14653].

These results should be read with care. Fixed-threshold CMDP evaluation, zero-shot threshold adaptation, hard-state safety, reach-avoid safety, and trajectory-budget satisfaction do not measure the same property. This suggests that empirical superiority in OSRL is strongly benchmark- and safety-definition-dependent, even when the same environments are reused.

## 6. Applications, deployment variants, and unresolved issues

OSRL is motivated by domains in which online exploration is costly or unacceptable. The benchmark paper highlights robotics and autonomous driving [2306.09303]. LexiSafe explicitly targets cyber-physical systems, emphasizing that safety violations during training are unacceptable and that hierarchical safety requirements are common [2602.17312]. OGSRL specializes the offline-safe setting to healthcare, where unsupported generalization beyond clinician expertise can cause harmful recommendations; it combines an OOD guardian with physiological safety constraints and reports a 78.2% reduction in mortality estimate compared with standard care on MIMIC-III sepsis treatment data [2505.16242]. TREBI demonstrates a large-scale advertising application with 1,938,151,000 bidding records and reports 7.15% relative improvement in total pay with 14.69% violation rate over a 10-day deployment [2306.00603].

A separate line studies how offline-safe priors can support limited online improvement. GOLD trains a Decision Transformer from offline demonstrations, then distills it into a lightweight online policy through guided online safe RL; on MetaDrive scenes derived from the Waymo Open Motion Dataset, it reports about 15% higher reward than baselines while keeping cost below threshold [2309.09408]. Marvel asks why naive offline-to-online warm starts fail in safe RL, identifies erroneous Q-estimations and Lagrangian mismatch as the main causes, and proposes Value Pre-Alignment plus Adaptive PID Control for online finetuning [2412.04426]. These works sit adjacent to OSRL proper, but they reinforce one of its central lessons: offline data is often best treated as a safety and performance prior rather than as a complete substitute for all future adaptation.

Several unresolved issues recur across the literature. Conservatism remains a defining trade-off: shielded SPI, FISOR, OASIS, and LexiSafe all note that robust or pessimistic reasoning can exclude useful actions or depress reward when data are sparse [2605.10293]. Many methods depend critically on learned cost or safety value functions; CAPS states explicitly that inaccurate \(Q^c\) estimation can hurt safety, and OPE-style cost estimates can be too conservative or ineffective for switching [2412.18946]. Generative and sequence-model approaches can adapt across thresholds, but transformers and diffusion models increase computational cost, and OASIS identifies offline preprocessing expense as a limitation [2407.14653]. Hard-safety methods still rarely claim universal zero-violation guarantees under function approximation, partial coverage, and finite data [2401.10700].

The current OSRL landscape therefore combines a stable core problem definition with a plural set of safety semantics, algorithms, and guarantees. The field has moved beyond viewing safety as a single scalar penalty. Trajectory classification, reachability, shielding, lexicographic hierarchy, conditional distribution shaping, adaptive policy switching, and posterior-uncertainty-based deployment criteria all represent distinct answers to the same question: how to extract high-return behavior from fixed data without leaving the safe region supported by the task and the dataset.

Source: https://www.emergentmind.com/topics/offline-safe-reinforcement-learning-osrl