Offline Safe Reinforcement Learning (OSRL)

• Offline Safe Reinforcement Learning (OSRL) is a framework that learns policies from a fixed dataset while enforcing safety via constrained Markov decision processes (CMDP).
• Methods balance high reward and strict safety by incorporating diverse strategies such as pessimistic Q-learning, trajectory classification, and hard-safety modeling.
• Challenges include distribution shift, conservative bias in under-supported regions, and dynamic deployment constraints that demand adaptable safety and performance guarantees.

Offline Safe Reinforcement Learning (OSRL) studies how to learn a policy from a fixed dataset of prior experience while maximizing task return under explicit safety constraints, without further interaction with the environment. In the dominant formulation, the environment is a constrained Markov decision process (CMDP) with reward $r$, cost $c$, and safety threshold $\kappa$; the learner must optimize return while keeping cumulative cost below the threshold, despite distribution shift and incomplete support in the offline data (Liu et al., 2023). Recent work has expanded this core formulation in several directions: distribution-constrained CMDPs that require proximity to the behavior policy, trajectory-level and hard-safety formulations, reach-avoid specifications, risk-sensitive objectives such as CVaR, and deployment-time budget adaptation (Gong et al., 2024).

1. Formal setting and problem variants

The standard OSRL objective is

$$\max_{\pi}\ \mathbb{E}_{\tau\sim\pi}[R(\tau)] \quad \text{s.t.} \quad \mathbb{E}_{\tau\sim\pi}[C(\tau)] \le \kappa,$$

with $R(\tau)=\sum_t \gamma^t r_t$ and $C(\tau)=\sum_t \gamma^t c_t$, learned from a fixed offline dataset such as $\mathcal D=\{(s_i,a_i,r_i,c_i,s'_i)\}_{i=1}^n$ or trajectory collections $D=\{\tau_1,\tau_2,\dots\}$ (Liu et al., 2023). Several papers further impose an offline-distribution restriction, for example $D_{KL}(\pi\|\pi_\beta)\le \varepsilon$ or $\mathcal D(\pi\|\pi_\beta)\le \epsilon$, to control extrapolation beyond the behavior policy that generated the data (Gong et al., 2024).

The safety notion is not uniform across the literature. Much of the benchmarked CMDP work uses discounted expected cost constraints, but other formulations require stricter guarantees. TREBI imposes a trajectory-level condition,

$c$0

to support real-time budget adaptation (Lin et al., 2023). SafeFQL uses a hard state-safety requirement,

$c$1

where $c$2 is a failure set (Tayal et al., 16 Mar 2026). Shielding-based OSRL encodes safety as a reach-avoid specification $c$3, meaning unsafe states must be avoided until target states are reached (Galesloot et al., 11 May 2026). Risk-averse offline RL replaces expected return with a distortion objective, notably CVaR, to emphasize lower-tail behavior rather than only the mean (Urpí et al., 2021).

These formulations are not interchangeable. Expected-cost CMDPs, hard-state safety, reach-avoid specifications, and CVaR all define different acceptable behaviors. This suggests that OSRL is better understood as a family of offline constrained decision problems than as a single optimization template.

2. Why offline safety is difficult

OSRL inherits the central offline RL problem of distribution shift, then compounds it with safety estimation. Because the learner cannot test actions online, errors in reward or cost estimation on unsupported state-action pairs can directly induce unsafe policies. The benchmark paper emphasizes that OSRL is difficult in both training and deployment: training is limited by fixed logged trajectories, extrapolation error, and incomplete coverage of the reward-cost trade-off surface, while deployment must tolerate unseen states, unseen safety thresholds, stochasticity, and noisy or outlier data (Liu et al., 2023).

A recurrent criticism of earlier methods is that they approximate safety locally when the relevant notion is often global. TraC states that safety is a trajectory-level property, whereas many OSRL algorithms enforce local per-step cost constraints derived from global constraints; this can produce either overly conservative policies or policies that violate safety constraints (Gong et al., 2024). The same tension appears in hard-safety work: FISOR argues that soft expectation constraints can still permit violations on some trajectories or states, while SafeFQL argues that expected-cost objectives can be insufficient for safety-critical real-time control (Zheng et al., 2024).

Another difficulty is dataset mismatch. OASIS formalizes “Safe Dataset Mismatch” for tempting datasets, which contain high-reward unsafe behavior, and conservative datasets, which contain low-cost but also low-reward behavior; direct regularization toward the empirical behavior policy can therefore pull learning toward the wrong part of the reward-cost frontier (Yao et al., 2024). CPQ emphasizes an adjacent problem: mixed behavior data may contain both safe low-reward and unsafe high-reward trajectories, so simply staying close to the data distribution can preserve unsafe tendencies rather than eliminate them (Xu et al., 2021).

A further challenge is deployment-time variability. CAPS and TREBI both target settings where the cost threshold changes after training, a regime in which a policy specialized for one threshold may become unsafe under tighter budgets or unnecessarily conservative under looser ones (Chemingui et al., 2024). Shielding-based offline RL identifies an additional structural issue: sparse data and coverage gaps make both transition dynamics and value estimates highly uncertain, while offline learning offers no online correction mechanism (Galesloot et al., 11 May 2026).

A common misconception is that OSRL is merely offline RL with an extra cost term. The benchmark study, CPQ, TraC, and hard-safety methods all reject that view: cost estimation, safety semantics, and support restrictions alter both the optimization objective and the failure modes (Liu et al., 2023).

3. Methodological paradigms

The literature spans several distinct design patterns.

Paradigm	Representative methods	Defining mechanism
Penalized or pessimistic Q-learning	CPQ (Xu et al., 2021), WSAC (Wei et al., 2024)	Inflate or adversarially estimate cost to suppress unsafe/OOD actions
Sequence or conditional trajectory modeling	CDT (Liu et al., 2023), GAS (Liu et al., 5 Feb 2026)	Condition policies on desired reward and cost returns
Feasibility or hard-safety generative modeling	FISOR (Zheng et al., 2024), SafeFQL (Tayal et al., 16 Mar 2026)	Learn feasible or reachable safe regions, then optimize reward within them
Distribution shaping or generative data synthesis	OASIS (Yao et al., 2024)	Generate a shaped offline dataset closer to the desired safe-reward distribution
Trajectory-level classification	TraC (Gong et al., 2024)	Label full trajectories as desirable or undesirable and train a classifier-guided policy
Minimax and primal-dual offline optimization	O3SRL (Chemingui et al., 24 Oct 2025)	Solve a Lagrangian minimax problem with offline RL and no-regret online optimization
Adaptive deployment-time policy selection	CAPS (Chemingui et al., 2024), TREBI (Lin et al., 2023)	Switch policies or plan trajectories according to the current budget
Shielding and robust model checking	Shielded SPIBB (Galesloot et al., 11 May 2026)	Build an IMDP from offline data and filter actions by robust reach-avoid safety
Lexicographic safety hierarchy	LexiSafe (Yang et al., 19 Feb 2026)	Optimize safety first, then reward inside the safety-feasible set

Penalized value-based methods explicitly treat unsupported actions as dangerous. CPQ learns a cost critic with an extra term that raises the estimated cost of OOD actions and then uses that critic to filter reward backups and actor improvement, effectively making OOD actions unsafe by construction (Xu et al., 2021). WSAC instead frames OSRL as a two-player Stackelberg game between an actor and adversarial reward and cost critics, seeking improvement over a reference policy while maintaining the same safety level (Wei et al., 2024).

Sequence-modeling approaches condition on reward and cost targets rather than solving a constrained Bellman problem directly. CDT extends Decision Transformer with cost-return tokens and uses return relabeling so the policy can dynamically adjust the safety-performance trade-off at deployment without retraining (Liu et al., 2023). GAS retains the goal-conditioned flavor but argues that transformer-style generative methods lack stitching ability; it therefore augments and relabels the dataset at the transition level and learns reward and cost goal functions via expectile regression (Liu et al., 5 Feb 2026).

Hard-safety and feasibility methods replace expected-cost reasoning with safe-set or reachability reasoning. FISOR uses Hamilton-Jacobi reachability to identify the largest feasible region, then derives an optimal policy of weighted behavior cloning form that maximizes reward in feasible states and minimizes risk in infeasible states (Zheng et al., 2024). SafeFQL uses a reachability-inspired safety critic with a max-backup recursion, a feasibility gate that prioritizes safety before reward, and conformal prediction to calibrate the learned safety boundary (Tayal et al., 16 Mar 2026).

Trajectory-level and preference-style methods operate on whole episodes rather than local transitions. TraC partitions the offline dataset into desirable trajectories, defined as safe and high-reward, and undesirable trajectories, defined as unsafe or low-reward safe trajectories, then trains a classifier-guided policy with a trajectory score tied to maximum-entropy RL and regret-based preference modeling (Gong et al., 2024).

Adaptive and planning-based methods address changing budgets explicitly. CAPS learns a small family of policies spanning reward-cost trade-offs, then at test time filters candidate actions using a learned cost $c$4-function and selects the feasible action with the highest reward estimate (Chemingui et al., 2024). TREBI instead treats the problem as trajectory-distribution optimization and performs budget-conditioned diffusion planning, aiming for per-trajectory budget satisfaction under changing real-time thresholds (Lin et al., 2023).

4. Safety notions and theoretical guarantees

The theoretical landscape is heterogeneous because the guarantees target different objects. Some methods guarantee safe policy improvement relative to a baseline, some guarantee approximate primal-dual optimality, some provide high-probability shielding soundness, and some provide deployment-time regret bounds.

WSAC gives a reference-policy guarantee: under its refined objective, the learned policy is no worse than the behavior policy in reward, $c$5, while positive safety violation is bounded by $c$6 (Wei et al., 2024). Shielded SPI provides a different type of statement. It builds an interval MDP from offline data, proves that the true MDP lies inside the interval model with high probability, and then shows that every action admitted by the shield is safe in the true MDP with probability at least $c$7, yielding a PAC-style safety filter for reach-avoid objectives (Galesloot et al., 11 May 2026).

LexiSafe derives safety-violation and performance-suboptimality bounds for its single-cost formulation and extends the analysis to multi-cost hierarchical safety. The stated sample-complexity result for LexiSafe-SC is

$c$8

with a corresponding linear-in-$c$9 extension for the multi-cost case (Yang et al., 19 Feb 2026). O3SRL, by contrast, analyzes OSRL as a dual minimax problem over distributions over policies and Lagrange multipliers. Its averaged output is an $\kappa$0-approximate equilibrium with

$\kappa$1

and its discrete-EXP3 approximation yields an error roughly $\kappa$2 while eliminating the need for offline policy evaluation (Chemingui et al., 24 Oct 2025).

SafeFQL’s guarantee is finite-sample probabilistic safety coverage rather than constrained-optimality. After conformal calibration, it states that with probability at least $\kappa$3, states in the calibrated safe set satisfy $\kappa$4 (Tayal et al., 16 Mar 2026). SOReL addresses a different question again: whether an offline policy is safe to deploy before any online interaction. It bounds regret in terms of posterior information loss and uses posterior predictive uncertainty to estimate deployment-time regret entirely offline (Fellows et al., 28 May 2025).

These results are not directly comparable. Some depend on safe baseline policies, some on concentrability and function-class assumptions, some on known safe and unsafe state sets and transition graphs, and some on accurate world models. FISOR explicitly notes that it does not claim a formal global zero-violation theorem for the learned deep model in the full offline setting, despite its hard-constraint formulation (Zheng et al., 2024). A plausible implication is that OSRL theory currently offers a menu of conditional guarantees rather than a single dominant certification framework.

5. Benchmarks, metrics, and empirical regularities

A major step in consolidating OSRL as a field was the introduction of DSRL, a benchmark ecosystem consisting of FSRL for safe-policy generation, DSRL for datasets and environment wrappers, and OSRL for baseline implementations (Liu et al., 2023). The benchmark reports over 75,000 trajectories across 38 tasks from SafetyGymnasium, BulletSafetyGym, and MetaDrive, with data post-processing filters for density, partial discarding, and noise injection (Liu et al., 2023). Standard metrics are normalized reward and normalized cost, and a policy is operationally treated as safe when normalized cost is at or below $\kappa$5 (Liu et al., 2023).

The original DSRL comparison already exposed a persistent empirical pattern. BC-All often attains higher reward but violates safety, BC-Safe is reliably safe but conservative, CDT usually balances reward and safety better than pure behavior cloning, and Q-learning-based methods such as BCQ-Lag, BEAR-Lag, and CPQ are inconsistent across tasks (Liu et al., 2023). That qualitative pattern recurs in later studies, although the ranking among stronger recent methods depends on the safety notion and deployment protocol.

Later methods report improvements under different evaluation settings. TraC, evaluated on the DSRL benchmark with 38 tasks across SafetyGymnasium, BulletSafetyGym, and MetaDrive, reports higher rewards and better constraint satisfaction than competitive baselines, with averaged results of reward $\kappa$6, cost $\kappa$7 on SafetyGym, reward $\kappa$8, cost $\kappa$9 on BulletGym, and reward $$\max_{\pi}\ \mathbb{E}_{\tau\sim\pi}[R(\tau)] \quad \text{s.t.} \quad \mathbb{E}_{\tau\sim\pi}[C(\tau)] \le \kappa,$$0, cost $$\max_{\pi}\ \mathbb{E}_{\tau\sim\pi}[R(\tau)] \quad \text{s.t.} \quad \mathbb{E}_{\tau\sim\pi}[C(\tau)] \le \kappa,$$1 on MetaDrive (Gong et al., 2024). CAPS, also evaluated on 38 DSRL tasks but targeting varying deployment-time thresholds, reports that CAPS(IQL) is safe on 34/38 tasks and achieves the highest reward on 18 tasks, whereas CDT is safe on 19/38 tasks (Chemingui et al., 2024). O3SRL focuses on the DSRL Bullet benchmark with eight tasks under a stringent budget $$\max_{\pi}\ \mathbb{E}_{\tau\sim\pi}[R(\tau)] \quad \text{s.t.} \quad \mathbb{E}_{\tau\sim\pi}[C(\tau)] \le \kappa,$$2 and reports being the only method that consistently satisfies the cost constraint across all eight tasks (Chemingui et al., 24 Oct 2025).

Hard-safety and generative methods use partially overlapping but not identical protocols. FISOR evaluates on 26 DSRL tasks from Safety-Gymnasium, Bullet-Safety-Gym, and MetaDrive and reports being the only method that achieves safety satisfaction in all evaluated tasks while attaining the highest return in most tasks (Zheng et al., 2024). OASIS, on Bullet-Safety-Gym datasets constructed to be full, tempting, conservative, or hybrid, reports safe normalized costs below threshold on all reported tasks and the highest reward among safe agents in most settings (Yao et al., 2024).

These results should be read with care. Fixed-threshold CMDP evaluation, zero-shot threshold adaptation, hard-state safety, reach-avoid safety, and trajectory-budget satisfaction do not measure the same property. This suggests that empirical superiority in OSRL is strongly benchmark- and safety-definition-dependent, even when the same environments are reused.

6. Applications, deployment variants, and unresolved issues

OSRL is motivated by domains in which online exploration is costly or unacceptable. The benchmark paper highlights robotics and autonomous driving (Liu et al., 2023). LexiSafe explicitly targets cyber-physical systems, emphasizing that safety violations during training are unacceptable and that hierarchical safety requirements are common (Yang et al., 19 Feb 2026). OGSRL specializes the offline-safe setting to healthcare, where unsupported generalization beyond clinician expertise can cause harmful recommendations; it combines an OOD guardian with physiological safety constraints and reports a 78.2% reduction in mortality estimate compared with standard care on MIMIC-III sepsis treatment data (Yan et al., 22 May 2025). TREBI demonstrates a large-scale advertising application with 1,938,151,000 bidding records and reports 7.15% relative improvement in total pay with 14.69% violation rate over a 10-day deployment (Lin et al., 2023).

A separate line studies how offline-safe priors can support limited online improvement. GOLD trains a Decision Transformer from offline demonstrations, then distills it into a lightweight online policy through guided online safe RL; on MetaDrive scenes derived from the Waymo Open Motion Dataset, it reports about 15% higher reward than baselines while keeping cost below threshold (Li et al., 2023). Marvel asks why naive offline-to-online warm starts fail in safe RL, identifies erroneous Q-estimations and Lagrangian mismatch as the main causes, and proposes Value Pre-Alignment plus Adaptive PID Control for online finetuning (Chen et al., 2024). These works sit adjacent to OSRL proper, but they reinforce one of its central lessons: offline data is often best treated as a safety and performance prior rather than as a complete substitute for all future adaptation.

Several unresolved issues recur across the literature. Conservatism remains a defining trade-off: shielded SPI, FISOR, OASIS, and LexiSafe all note that robust or pessimistic reasoning can exclude useful actions or depress reward when data are sparse (Galesloot et al., 11 May 2026). Many methods depend critically on learned cost or safety value functions; CAPS states explicitly that inaccurate $$\max_{\pi}\ \mathbb{E}_{\tau\sim\pi}[R(\tau)] \quad \text{s.t.} \quad \mathbb{E}_{\tau\sim\pi}[C(\tau)] \le \kappa,$$3 estimation can hurt safety, and OPE-style cost estimates can be too conservative or ineffective for switching (Chemingui et al., 2024). Generative and sequence-model approaches can adapt across thresholds, but transformers and diffusion models increase computational cost, and OASIS identifies offline preprocessing expense as a limitation (Yao et al., 2024). Hard-safety methods still rarely claim universal zero-violation guarantees under function approximation, partial coverage, and finite data (Zheng et al., 2024).

The current OSRL landscape therefore combines a stable core problem definition with a plural set of safety semantics, algorithms, and guarantees. The field has moved beyond viewing safety as a single scalar penalty. Trajectory classification, reachability, shielding, lexicographic hierarchy, conditional distribution shaping, adaptive policy switching, and posterior-uncertainty-based deployment criteria all represent distinct answers to the same question: how to extract high-return behavior from fixed data without leaving the safe region supported by the task and the dataset.