---
title: Safe Flow Q-Learning (SafeFQL)
url: https://www.emergentmind.com/topics/safe-flow-q-learning-safefql
type: topic
---

# Safe Flow Q-Learning (SafeFQL)

Searching arXiv for the cited papers to ground the article.
Searching for the SafeFQL paper and key related work.
Safe Flow Q-Learning (SafeFQL) is an offline safe reinforcement learning method for constrained decision-making from static datasets under strict safety constraints. It extends Flow Q-Learning (FQL) by combining a Hamilton–Jacobi reachability-inspired safety value function with a flow-based behavior policy teacher and a distilled one-step actor, and it adds a conformal prediction calibration step to adjust the learned safety threshold under finite-data approximation error. Its central objective is to maximize discounted reward while avoiding entry into a failure set, with deployment-time action generation performed in a single forward pass rather than through iterative denoising or rejection sampling [2603.15136].

## 1. Problem formulation and safety semantics

SafeFQL is posed on a constrained Markov decision process
\[
\mathcal{M} = (\mathcal{X}, \mathcal{A}, P, r, \ell, \gamma),
\]
with offline dataset
\[
\mathcal D = \{(x_t,a_t,r_t,\ell_t,x_{t+1})\}.
\]
The reward \(r\) and safety signal \(\ell\) are state-based, and the failure set is defined as
\[
\mathcal F := \{x \in \mathcal X \mid \ell(x) > 0\}.
\]
A state is therefore safe iff \(\ell(x)\le 0\), and unsafe iff \(\ell(x)>0\) [2603.15136].

The control objective is stricter than a standard expected-cost CMDP objective. SafeFQL seeks
\[
\sup_{\pi} \Big[ \sum_{k=0}^{\infty} \gamma^k r(x_k) \,\Big|\, x_0 = x \Big]
\quad \text{s.t.} \quad x_t \notin \mathcal F,\ \forall t\ge 0.
\]
This is a trajectory-wise or state-wise safety requirement, not an expectation constraint. The method is explicitly motivated by the claim that expected cumulative cost constraints are soft constraints and may permit some unsafe trajectories so long as average cost remains acceptable [2603.15136].

The paper frames this safety semantics through a Hamilton–Jacobi reachability-inspired value,
\[
V_\ell^*(x_0) := \min_{\pi}\max_{t\ge 0}\ell(x_t),
\]
with the interpretation that \(V_\ell^*(x)<0\) indicates feasibility and \(V_\ell^*(x)\ge 0\) indicates infeasibility. This substitutes worst-future safety margin for discounted cumulative cost as the central safety object. A plausible implication is that SafeFQL belongs to the reachability-oriented branch of safe RL rather than the Lagrangian CMDP branch [2603.15136].

## 2. Learning architecture and optimization components

SafeFQL combines reward learning, safety learning, behavior modeling, and actor distillation in a single offline pipeline [2603.15136].

| Component | Role | Main object |
|---|---|---|
| Reward learner | IQL-style reward estimation | \(Q_r, V_r\) |
| Safety learner | Reachability-style feasibility estimation | \(Q_c, V_c\) |
| Behavior model | Flow-matching teacher trained by behavioral cloning | \(\mu_\theta\) |
| Deployment policy | Distilled one-step actor | \(\mu_\omega\) |
| Calibration module | Post-hoc threshold correction | \(\delta^*\) |

The reward side follows IQL-style expectile regression:
\[
\mathcal L_{V_r}(\psi_r) = \mathbb E_{(x,a)\sim\mathcal D} \left[ \mathcal L_\tau(Q_r(x,a;\phi_r)-V_r(x;\psi_r)) \right],
\]
\[
y_r = r + \gamma\, \bar V_r(x'),
\]
\[
\mathcal L_{Q_r}(\phi_r) = \mathbb E_{(x,a,r,x')\sim\mathcal D} \left[ \left(Q_r(x,a;\phi_r)-y_r\right)^2 \right].
\]
The expectile loss is
\[
\mathcal L_\tau(u)=|\tau-\mathbb I(u<0)|\,u^2.
\]
Thus reward optimization remains within the offline RL lineage of IQL, while safety is handled by a separate critic system [2603.15136].

The actor is not trained directly from the behavior data alone. Instead, it is initialized and optimized against a teacher-student structure: a flow teacher \(\mu_\theta\) is learned by behavioral cloning, then a one-step actor \(\mu_\omega\) is distilled from that teacher and subsequently optimized with reward and safety critics. This separation is central to SafeFQL’s claim that expressive multimodal action modeling can be retained without iterative sampling at deployment [2603.15136].

## 3. Reachability-inspired safety value and feasibility gate

The core safety recursion is not additive. SafeFQL defines the safety target
\[
y_c(x,a,x') = \max\left\{\ell(x),\ \gamma \bar V_c(x')\right\}.
\]
This is a self-consistency Bellman recursion with a max operator: current unsafety dominates immediately, otherwise future worst-case safety margin is propagated backward. The corresponding losses are
\[
\mathcal L_{Q_c}(\phi_c) = \mathbb E_{(x,a,\ell,x')\sim\mathcal D} \left[ \left(Q_c(x,a;\phi_c)-y_c\right)^2 \right],
\]
\[
\mathcal L_{V_c}(\psi_c) = \mathbb E_{(x,a)\sim\mathcal D} \left[ \mathcal L_\tau\!\left(Q_c(x,a;\phi_c)-V_c(x;\psi_c)\right) \right].
\]
The paper states that \(V_c\) is trained conservatively as a lower quantile estimate of the in-sample safety Q-distribution [2603.15136].

Feasibility is then enforced by a hard gate on the actor output. For a latent \(z\),
\[
\zeta(x,z)=\mathbb I\left\{Q_c\!\left(x,\mu_\omega(x,z)\right)<0\right\}.
\]
This induces a critic-defined feasible action condition \(Q_c(x,a)<0\). The actor objective is
\[
\mathcal L_{\text{actor}(\omega) = \lambda\,\mathcal L_{\text{distill}(\omega) + \mathbb E_{(x,z)} \left[ \zeta(x,z)\cdot \big(-Q_r(x,a_\omega)\big) + \big(1-\zeta(x,z)\big)\cdot \max(0,Q_c(x,a_\omega)) \right],
\]
where \(a_\omega=\mu_\omega(x,z)\). This implements a priority rule: if the action is predicted unsafe, optimization ignores reward and minimizes feasibility violation; if it is predicted safe, optimization ignores safety penalty and maximizes reward [2603.15136].

This gate is one of the main conceptual distinctions of SafeFQL. The paper explicitly contrasts it with a naive soft-constrained actor objective
\[
\mathcal L_{\text{actor}^{\text{naive}(\omega) = \mathbb E_{x,z} \left[ -Q_r(x,a_\omega)+\eta \max(0,Q_c(x,a_\omega)) \right] +\lambda \mathcal L_{\text{distill}(\omega),
\]
arguing that SafeFQL should prioritize feasibility recovery rather than balance reward and safety by a single linear trade-off [2603.15136].

A close conceptual antecedent is reach-avoid Q-learning, which also replaces additive reward backups with min/max-over-time safety-liveness recursions and conservative safe-set approximations [2112.12288]. SafeFQL differs in using offline function approximation, a learned one-step actor, and a reachability-style safety critic integrated into a flow-based policy architecture.

## 4. Flow teacher, one-step actor, and deployment-time action generation

The behavior policy is modeled by a conditional flow. A state-conditioned velocity field \(v_\theta\) defines the flow
\[
\frac{d}{dt}\psi_\theta(t,x,z) = v_\theta\big(t,x,\psi_\theta(t,x,z)\big), \qquad \psi_\theta(0,x,z)=z,
\]
and the terminal policy sample
\[
\mu_\theta(x,z) = \psi_\theta(1,x,z) = z+\int_0^1 v_\theta\big(t,x,\psi_\theta(t,x,z)\big)\,dt.
\]
With \(z\sim\mathcal N(0,I)\), this yields a stochastic state-conditional action model [2603.15136].

The teacher is trained purely by behavioral cloning:
\[
\mathcal L_{\text{flow}(\theta) = \mathbb E_{(x,a)\sim\mathcal D,\ z\sim\mathcal N(0,I),\ t\sim\mathcal U([0,1])} \left[ \left\| \mu_\theta(x,x_t,t) - (a-z) \right\|_2^2 \right],
\]
with
\[
x_t = (1-t)z + ta.
\]
The paper notes that this notation is slightly inconsistent with standard flow-matching notation, but the intended meaning is a conditional transport from latent Gaussian noise to dataset actions [2603.15136].

The deployment actor is a one-step map \(\mu_\omega(x,z)\), trained by distillation:
\[
\mathcal L_{\text{distill}(\omega) = \mathbb E_{(x,z)\sim \mathcal D \times \mathcal N(0,I)} \left[ \|\mu_\omega(x,z)-\tilde\mu_\theta(x,z)\|_2^2 \right].
\]
This is then combined with the feasibility-gated actor objective. At inference time, deployment uses only
\[
a = \mu_\omega(x,z),\qquad z\sim \mathcal N(0,I).
\]
The paper emphasizes that this requires a single forward pass, with no denoising loop and no rejection sampling over multiple candidates [2603.15136].

This positions SafeFQL relative to earlier flow and generative offline RL in a specific way. Vanilla FQL is described as unconstrained offline RL with a flow-based behavior teacher, distilled one-step actor, and reward optimization, whereas SafeFQL adds a second critic system for safety, a reachability-style max backup, feasibility-gated actor updates, and conformal calibration [2603.15136]. A nearby unconstrained line of work is one-step flow generation for offline RL, such as OFQL, which reformulates diffusion Q-learning within flow matching to enable efficient one-step action generation [2508.13904].

## 5. Conformal calibration and finite-sample safety coverage

SafeFQL treats approximation error near the learned safety boundary as a first-class issue. If \(V_c\) or \(Q_c\) is inaccurate near the threshold \(Q_c=0\), the actor may choose actions that are predicted safe but unsafe under rollout. To address this, the paper adds a post-hoc conformal calibration step that adjusts the safety threshold by a margin \(\delta\) [2603.15136].

The calibrated safe set is
\[
\mathcal S_\delta = \{x:V_c(x)\le \delta\}, \qquad \mathcal S_{\delta^*}=\{x:V_c(x)<\delta^*\}.
\]
The ideal correction is written as
\[
\delta^* := \min_{\hat x \in \mathcal X} \{V_c(x): V_c^\pi(x)\ge 0\}.
\]
The notation is malformed in the paper, but the stated intent is to find the smallest predicted safety value among states that are actually unsafe under the learned policy, then use that as a stricter threshold [2603.15136].

For a candidate \(\delta\), the calibration stage samples \(N_s\) i.i.d. states from \(\mathcal S_\delta\) and uses conformal score
\[
s(X)=V_c^\pi(x), \qquad \forall x\in \mathcal S_{\tilde\delta}.
\]
Let
\[
l = \lfloor (N_s+1)\alpha_\delta \rfloor.
\]
If
\[
\sum_{i=0}^{l-1} \binom{N_s}{i} \epsilon_s^i (1-\epsilon_s)^{N_s-i} \le \beta_s,
\]
then with probability at least \(1-\beta_s\),
\[
\mathbb P_{x_i\in \mathcal S_\delta}\big(V_c(x_i)<0\big)\ge 1-\epsilon_s.
\]
The paper describes this as a finite-sample probabilistic coverage guarantee under i.i.d. or exchangeability assumptions [2603.15136].

Reported calibrated thresholds are environment-specific:

| Environment | \(\delta^*\) |
|---|---|
| Boat | \(0.0\) |
| Hopper | \(-0.07\) |
| HalfCheetah | \(0.0\) |
| Ant | \(0.0\) |
| Walker2D | \(-0.04\) |
| Swimmer | \(0.0\) |

The negative values for Hopper and Walker2D indicate stricter calibrated safe sets in those cases. This suggests that the learned safety boundary was already reasonably calibrated in most environments, but required additional pessimism in a subset of tasks [2603.15136].

## 6. Empirical profile, related methods, and limitations

The empirical evaluation covers a custom Safe Boat Navigation domain and Safety Gymnasium MuJoCo safe-velocity tasks on Hopper, HalfCheetah, Ant, Walker2D, and Swimmer, using DSRL offline safe RL datasets [2603.15136]. In the boat domain, the state is \(x=[x_1,x_2]^\top \in [-3,2]\times[-2,2]\), the reward is
\[
r(x)=C\cdot\Big(-\|[x_1,x_2]^\top-[x_{g1},x_{g2}]^\top\|\Big),\quad C=0.1,
\]
the dynamics are
\[
x_{1,t+1}=x_{1,t}+\big(a_{1,t}+2-0.5x_{2,t}^2\big)\Delta t,
\]
\[
x_{2,t+1}=x_{2,t}+a_{2,t}\Delta t,
\]
with \(a_1^2+a_2^2\le 1\), and the safety function is
\[
\ell(x):=
\max\!\big(
0.4-\|x-[-0.5,0.5]^\top\|,
\;0.5-\|x-[-1.0,-1.2]^\top\|
\big).
\]
The dataset contains about \(1\)M samples from 2500 random initial states with 400 random-control steps each [2603.15136].

Across baselines including BEAR-Lag, COptiDICE, CPQ, C2IQL, FISOR, and SafeIFQL, the paper reports that SafeFQL achieves the lowest costs across all evaluated environments, zero violations across all evaluation episodes on boat navigation, and the highest reward among methods with comparable near-zero costs on Safety Gymnasium tasks [2603.15136]. On boat navigation, SafeFQL achieves top safety with \(N=1\) action sample, whereas FISOR and SafeIFQL require larger candidate pools such as \(N=16\) to become competitive. The conclusion also reports about \(2.5\times\) inference speedup, reflecting the distinction between modestly higher offline training cost and substantially lower inference latency [2603.15136].

Relative to prior safe offline RL, SafeFQL occupies a distinct position. CPQ treats support mismatch as a safety problem by inflating OOD cost estimates and gating reward propagation through safe actions [2107.09003]. Safe-Support Q-Learning enforces safety by restricting learning and policy improvement to the support of a safe behavior policy through KL-regularized Bellman targets [2604.25379]. Reach-avoid Q-learning replaces additive reward Bellman backups with discounted reach-avoid operators that conservatively under-approximate safe-and-live sets [2112.12288]. FlowQ, by contrast, is an unconstrained energy-guided flow-matching method for offline RL that targets expressive policies without inference-time guidance [2505.14139]. SafeFQL can be read as synthesizing flow-based policy modeling with reachability-style safety semantics and statistical calibration [2603.15136].

The method’s main limitations are also explicit. Offline data coverage remains decisive; if the dataset poorly covers critical safe actions, the critics and teacher can misgeneralize. Calibrated thresholds may shrink the safe set and reduce reward. The finite-sample guarantee is probabilistic rather than deterministic, and depends on calibration data and exchangeability assumptions. The hard indicator gate may create non-smooth optimization. The manuscript also contains several notation and implementation ambiguities, notably around whether pessimistic safety aggregation uses the maximum or minimum of two safety Q networks: the main text says the maximum is used to avoid optimistic feasibility estimates, while experimental details describe clipped double Q-learning as taking a minimum of the two Q values. The stated high-level intent is pessimistic safety estimation, but the discrepancy remains a manuscript-level caveat [2603.15136].

In sum, SafeFQL is best understood as a reachability-aware, flow-distilled offline safe RL method whose defining features are a max-backup safety critic, a feasibility-gated actor, a behavior-cloned flow teacher, a one-step deployed actor, and conformal post-hoc calibration. Its technical contribution is not merely adding a safety cost to a flow-based offline RL algorithm, but re-specifying safety in terms of future feasibility and coupling that feasibility estimate to policy extraction and deployment efficiency [2603.15136].

Source: https://www.emergentmind.com/topics/safe-flow-q-learning-safefql