Safe Flow Q-Learning (SafeFQL)

• Safe Flow Q-Learning (SafeFQL) is an offline safe reinforcement learning method defined on constrained MDPs, integrating Hamilton–Jacobi reachability-inspired safety values with flow-based behavior policies.
• It combines a behavior-cloned flow teacher, a distilled one-step actor, and separate reward and safety critics with a hard feasibility gate to ensure safe action generation.
• A post-hoc conformal calibration step adjusts safety thresholds under finite data, leading to robust empirical performance with rapid single-pass inference.

Searching arXiv for the cited papers to ground the article. Searching for the SafeFQL paper and key related work. Safe Flow Q-Learning (SafeFQL) is an offline safe reinforcement learning method for constrained decision-making from static datasets under strict safety constraints. It extends Flow Q-Learning (FQL) by combining a Hamilton–Jacobi reachability-inspired safety value function with a flow-based behavior policy teacher and a distilled one-step actor, and it adds a conformal prediction calibration step to adjust the learned safety threshold under finite-data approximation error. Its central objective is to maximize discounted reward while avoiding entry into a failure set, with deployment-time action generation performed in a single forward pass rather than through iterative denoising or rejection sampling (Tayal et al., 16 Mar 2026).

1. Problem formulation and safety semantics

SafeFQL is posed on a constrained Markov decision process

$$\mathcal{M} = (\mathcal{X}, \mathcal{A}, P, r, \ell, \gamma),$$

with offline dataset

$\mathcal D = \{(x_t,a_t,r_t,\ell_t,x_{t+1})\}.$

The reward $r$ and safety signal $\ell$ are state-based, and the failure set is defined as

$$\mathcal F := \{x \in \mathcal X \mid \ell(x) > 0\}.$$

A state is therefore safe iff $\ell(x)\le 0$, and unsafe iff $\ell(x)>0$ (Tayal et al., 16 Mar 2026).

The control objective is stricter than a standard expected-cost CMDP objective. SafeFQL seeks

$$\sup_{\pi} \Big[ \sum_{k=0}^{\infty} \gamma^k r(x_k) \,\Big|\, x_0 = x \Big] \quad \text{s.t.} \quad x_t \notin \mathcal F,\ \forall t\ge 0.$$

This is a trajectory-wise or state-wise safety requirement, not an expectation constraint. The method is explicitly motivated by the claim that expected cumulative cost constraints are soft constraints and may permit some unsafe trajectories so long as average cost remains acceptable (Tayal et al., 16 Mar 2026).

The paper frames this safety semantics through a Hamilton–Jacobi reachability-inspired value,

$V_\ell^*(x_0) := \min_{\pi}\max_{t\ge 0}\ell(x_t),$

with the interpretation that $V_\ell^*(x)<0$ indicates feasibility and $\mathcal D = \{(x_t,a_t,r_t,\ell_t,x_{t+1})\}.$0 indicates infeasibility. This substitutes worst-future safety margin for discounted cumulative cost as the central safety object. A plausible implication is that SafeFQL belongs to the reachability-oriented branch of safe RL rather than the Lagrangian CMDP branch (Tayal et al., 16 Mar 2026).

2. Learning architecture and optimization components

SafeFQL combines reward learning, safety learning, behavior modeling, and actor distillation in a single offline pipeline (Tayal et al., 16 Mar 2026).

Component	Role	Main object
Reward learner	IQL-style reward estimation	$\mathcal D = \{(x_t,a_t,r_t,\ell_t,x_{t+1})\}.$1
Safety learner	Reachability-style feasibility estimation	$\mathcal D = \{(x_t,a_t,r_t,\ell_t,x_{t+1})\}.$2
Behavior model	Flow-matching teacher trained by behavioral cloning	$\mathcal D = \{(x_t,a_t,r_t,\ell_t,x_{t+1})\}.$3
Deployment policy	Distilled one-step actor	$\mathcal D = \{(x_t,a_t,r_t,\ell_t,x_{t+1})\}.$4
Calibration module	Post-hoc threshold correction	$\mathcal D = \{(x_t,a_t,r_t,\ell_t,x_{t+1})\}.$5

The reward side follows IQL-style expectile regression: $\mathcal D = \{(x_t,a_t,r_t,\ell_t,x_{t+1})\}.$6

$\mathcal D = \{(x_t,a_t,r_t,\ell_t,x_{t+1})\}.$7

$\mathcal D = \{(x_t,a_t,r_t,\ell_t,x_{t+1})\}.$8

The expectile loss is

$\mathcal D = \{(x_t,a_t,r_t,\ell_t,x_{t+1})\}.$9

Thus reward optimization remains within the offline RL lineage of IQL, while safety is handled by a separate critic system (Tayal et al., 16 Mar 2026).

The actor is not trained directly from the behavior data alone. Instead, it is initialized and optimized against a teacher-student structure: a flow teacher $r$0 is learned by behavioral cloning, then a one-step actor $r$1 is distilled from that teacher and subsequently optimized with reward and safety critics. This separation is central to SafeFQL’s claim that expressive multimodal action modeling can be retained without iterative sampling at deployment (Tayal et al., 16 Mar 2026).

3. Reachability-inspired safety value and feasibility gate

The core safety recursion is not additive. SafeFQL defines the safety target

$r$2

This is a self-consistency Bellman recursion with a max operator: current unsafety dominates immediately, otherwise future worst-case safety margin is propagated backward. The corresponding losses are

$r$3

$r$4

The paper states that $r$5 is trained conservatively as a lower quantile estimate of the in-sample safety Q-distribution (Tayal et al., 16 Mar 2026).

Feasibility is then enforced by a hard gate on the actor output. For a latent $r$6,

$r$7

This induces a critic-defined feasible action condition $r$8. The actor objective is

$r$9

where $\ell$0. This implements a priority rule: if the action is predicted unsafe, optimization ignores reward and minimizes feasibility violation; if it is predicted safe, optimization ignores safety penalty and maximizes reward (Tayal et al., 16 Mar 2026).

This gate is one of the main conceptual distinctions of SafeFQL. The paper explicitly contrasts it with a naive soft-constrained actor objective

$\ell$1

arguing that SafeFQL should prioritize feasibility recovery rather than balance reward and safety by a single linear trade-off (Tayal et al., 16 Mar 2026).

A close conceptual antecedent is reach-avoid Q-learning, which also replaces additive reward backups with min/max-over-time safety-liveness recursions and conservative safe-set approximations (Hsu et al., 2021). SafeFQL differs in using offline function approximation, a learned one-step actor, and a reachability-style safety critic integrated into a flow-based policy architecture.

4. Flow teacher, one-step actor, and deployment-time action generation

The behavior policy is modeled by a conditional flow. A state-conditioned velocity field $\ell$2 defines the flow

$\ell$3

and the terminal policy sample

$\ell$4

With $\ell$5, this yields a stochastic state-conditional action model (Tayal et al., 16 Mar 2026).

The teacher is trained purely by behavioral cloning: $\ell$6 with

$\ell$7

The paper notes that this notation is slightly inconsistent with standard flow-matching notation, but the intended meaning is a conditional transport from latent Gaussian noise to dataset actions (Tayal et al., 16 Mar 2026).

The deployment actor is a one-step map $\ell$8, trained by distillation: $\ell$9 This is then combined with the feasibility-gated actor objective. At inference time, deployment uses only

$$\mathcal F := \{x \in \mathcal X \mid \ell(x) > 0\}.$$0

The paper emphasizes that this requires a single forward pass, with no denoising loop and no rejection sampling over multiple candidates (Tayal et al., 16 Mar 2026).

This positions SafeFQL relative to earlier flow and generative offline RL in a specific way. Vanilla FQL is described as unconstrained offline RL with a flow-based behavior teacher, distilled one-step actor, and reward optimization, whereas SafeFQL adds a second critic system for safety, a reachability-style max backup, feasibility-gated actor updates, and conformal calibration (Tayal et al., 16 Mar 2026). A nearby unconstrained line of work is one-step flow generation for offline RL, such as OFQL, which reformulates diffusion Q-learning within flow matching to enable efficient one-step action generation (Nguyen et al., 19 Aug 2025).

5. Conformal calibration and finite-sample safety coverage

SafeFQL treats approximation error near the learned safety boundary as a first-class issue. If $$\mathcal F := \{x \in \mathcal X \mid \ell(x) > 0\}.$$1 or $$\mathcal F := \{x \in \mathcal X \mid \ell(x) > 0\}.$$2 is inaccurate near the threshold $$\mathcal F := \{x \in \mathcal X \mid \ell(x) > 0\}.$$3, the actor may choose actions that are predicted safe but unsafe under rollout. To address this, the paper adds a post-hoc conformal calibration step that adjusts the safety threshold by a margin $$\mathcal F := \{x \in \mathcal X \mid \ell(x) > 0\}.$$4 (Tayal et al., 16 Mar 2026).

The calibrated safe set is

$$\mathcal F := \{x \in \mathcal X \mid \ell(x) > 0\}.$$5

The ideal correction is written as

$$\mathcal F := \{x \in \mathcal X \mid \ell(x) > 0\}.$$6

The notation is malformed in the paper, but the stated intent is to find the smallest predicted safety value among states that are actually unsafe under the learned policy, then use that as a stricter threshold (Tayal et al., 16 Mar 2026).

For a candidate $$\mathcal F := \{x \in \mathcal X \mid \ell(x) > 0\}.$$7, the calibration stage samples $$\mathcal F := \{x \in \mathcal X \mid \ell(x) > 0\}.$$8 i.i.d. states from $$\mathcal F := \{x \in \mathcal X \mid \ell(x) > 0\}.$$9 and uses conformal score

$\ell(x)\le 0$0

Let

$\ell(x)\le 0$1

If

$\ell(x)\le 0$2

then with probability at least $\ell(x)\le 0$3,

$\ell(x)\le 0$4

The paper describes this as a finite-sample probabilistic coverage guarantee under i.i.d. or exchangeability assumptions (Tayal et al., 16 Mar 2026).

Reported calibrated thresholds are environment-specific:

Environment	$\ell(x)\le 0$5
Boat	$\ell(x)\le 0$6
Hopper	$\ell(x)\le 0$7
HalfCheetah	$\ell(x)\le 0$8
Ant	$\ell(x)\le 0$9
Walker2D	$\ell(x)>0$0
Swimmer	$\ell(x)>0$1

The negative values for Hopper and Walker2D indicate stricter calibrated safe sets in those cases. This suggests that the learned safety boundary was already reasonably calibrated in most environments, but required additional pessimism in a subset of tasks (Tayal et al., 16 Mar 2026).

6. Empirical profile, related methods, and limitations

The empirical evaluation covers a custom Safe Boat Navigation domain and Safety Gymnasium MuJoCo safe-velocity tasks on Hopper, HalfCheetah, Ant, Walker2D, and Swimmer, using DSRL offline safe RL datasets (Tayal et al., 16 Mar 2026). In the boat domain, the state is $\ell(x)>0$2, the reward is

$\ell(x)>0$3

the dynamics are

$\ell(x)>0$4

$\ell(x)>0$5

with $\ell(x)>0$6, and the safety function is

$\ell(x)>0$7

The dataset contains about $\ell(x)>0$8M samples from 2500 random initial states with 400 random-control steps each (Tayal et al., 16 Mar 2026).

Across baselines including BEAR-Lag, COptiDICE, CPQ, C2IQL, FISOR, and SafeIFQL, the paper reports that SafeFQL achieves the lowest costs across all evaluated environments, zero violations across all evaluation episodes on boat navigation, and the highest reward among methods with comparable near-zero costs on Safety Gymnasium tasks (Tayal et al., 16 Mar 2026). On boat navigation, SafeFQL achieves top safety with $\ell(x)>0$9 action sample, whereas FISOR and SafeIFQL require larger candidate pools such as $$\sup_{\pi} \Big[ \sum_{k=0}^{\infty} \gamma^k r(x_k) \,\Big|\, x_0 = x \Big] \quad \text{s.t.} \quad x_t \notin \mathcal F,\ \forall t\ge 0.$$0 to become competitive. The conclusion also reports about $$\sup_{\pi} \Big[ \sum_{k=0}^{\infty} \gamma^k r(x_k) \,\Big|\, x_0 = x \Big] \quad \text{s.t.} \quad x_t \notin \mathcal F,\ \forall t\ge 0.$$1 inference speedup, reflecting the distinction between modestly higher offline training cost and substantially lower inference latency (Tayal et al., 16 Mar 2026).

Relative to prior safe offline RL, SafeFQL occupies a distinct position. CPQ treats support mismatch as a safety problem by inflating OOD cost estimates and gating reward propagation through safe actions (Xu et al., 2021). Safe-Support Q-Learning enforces safety by restricting learning and policy improvement to the support of a safe behavior policy through KL-regularized Bellman targets (Lim et al., 28 Apr 2026). Reach-avoid Q-learning replaces additive reward Bellman backups with discounted reach-avoid operators that conservatively under-approximate safe-and-live sets (Hsu et al., 2021). FlowQ, by contrast, is an unconstrained energy-guided flow-matching method for offline RL that targets expressive policies without inference-time guidance (Alles et al., 20 May 2025). SafeFQL can be read as synthesizing flow-based policy modeling with reachability-style safety semantics and statistical calibration (Tayal et al., 16 Mar 2026).

The method’s main limitations are also explicit. Offline data coverage remains decisive; if the dataset poorly covers critical safe actions, the critics and teacher can misgeneralize. Calibrated thresholds may shrink the safe set and reduce reward. The finite-sample guarantee is probabilistic rather than deterministic, and depends on calibration data and exchangeability assumptions. The hard indicator gate may create non-smooth optimization. The manuscript also contains several notation and implementation ambiguities, notably around whether pessimistic safety aggregation uses the maximum or minimum of two safety Q networks: the main text says the maximum is used to avoid optimistic feasibility estimates, while experimental details describe clipped double Q-learning as taking a minimum of the two Q values. The stated high-level intent is pessimistic safety estimation, but the discrepancy remains a manuscript-level caveat (Tayal et al., 16 Mar 2026).

In sum, SafeFQL is best understood as a reachability-aware, flow-distilled offline safe RL method whose defining features are a max-backup safety critic, a feasibility-gated actor, a behavior-cloned flow teacher, a one-step deployed actor, and conformal post-hoc calibration. Its technical contribution is not merely adding a safety cost to a flow-based offline RL algorithm, but re-specifying safety in terms of future feasibility and coupling that feasibility estimate to policy extraction and deployment efficiency (Tayal et al., 16 Mar 2026).