---
title: 'RSGF-RL: Anytime Safe Constrained RL'
url: https://www.emergentmind.com/topics/rsgf-rl
type: topic
---

# RSGF-RL: Anytime Safe Constrained RL

Searching arXiv for the exact method name and primary source paper.
Searching arXiv for related acronym collisions and nearby methods to support disambiguation.
RSGF-RL is an off-policy method for constrained reinforcement learning derived from the **Robust Safe Gradient Flow (RSGF)**, a continuous-time dynamics for nonlinear constrained optimization. Its defining objective is to provide **anytime safety guarantees**: with a user-specified probability and sufficient data, each policy update yields a constraint-satisfying policy, and a mildly unsafe policy can be driven back to safety in one step with high probability. The method estimates value functions and their gradients from episodic data and computes each policy update by solving a convex quadratically constrained quadratic program (QCQP) [2510.01492].

## 1. Constrained RL setting and meaning of anytime safety

RSGF-RL is formulated for a finite-horizon discounted constrained Markov decision process
$$
\mathcal{M} = (\mathcal{S},\mathcal{A},P,R_0,\{R_j\}_{j=1}^q),
$$
with task reward $R_0$, safety-cost functions $R_j$, discount factor $\gamma\in(0,1)$, horizon $T\in\mathbb{Z}_{>0}$, and parameterized policy $\pi_\theta$, $\theta\in\mathbb{R}^d$. The paper writes the optimization problem as
$$
\begin{aligned}
\min_{\theta\in\mathbb{R}^d}\quad  V_0(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \\ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} -\gamma^k R_0(s_k, a_k, s_{k+1}) \biggr] \\
\text{s.t.}\quad  V_j(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \\ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} \gamma^k R_j(s_k, a_k, s_{k+1}) \biggr] \le 0,\qquad j\in[q].
\end{aligned}
$$
The sign of the reward is flipped so that the task is written as a minimization problem [2510.01492].

The corresponding feasible parameter set is
$$
C = \{\theta\in\mathbb{R}^d : V_j(\theta)\le 0,\ \forall j\in[q]\}.
$$
Because the value functions induced by a parameterized policy are generally nonconvex in $\theta$, the method targets convergence to KKT points rather than global optima [2510.01492].

In this framework, **anytime safety** means that if the algorithm generates policy iterates $\theta_1,\theta_2,\dots$, then constraint satisfaction is preserved at every training iterate with prescribed probability. The finite-horizon statement is
$$
\mathbb{P}\Big( \bigcap_{i=1}^{H+1}\{V_j(\theta_i)\le 0,\ \forall j\in[q]\}\Big)\ge 1-2qH\delta,
$$
provided the paper’s per-iteration sample conditions hold [2510.01492]. This is a training-time property, not merely a guarantee on the terminal policy.

The paper also notes that suitable cumulative constraints can encode state-space safety with confidence. If
$$
R_j(s_t) = 1-\mathbbm{1}_{C_j}(s_t)+\frac{\gamma^T\delta_j}{\sum_{t=0}^{T-1}\gamma^t},
$$
then satisfying $V_j(\theta)\le 0$ implies
$$
\mathbb{P}\Big(\bigcap_{t=0}^{T-1}\{s_t\in C_j\}\Big)\ge 1-\delta_j.
$$
This links CMDP constraints to high-probability state safety [2510.01492].

## 2. Continuous-time foundation: Robust Safe Gradient Flow

RSGF-RL is built from a continuous-time constrained optimization dynamics called the **Robust Safe Gradient Flow**. For the generic nonlinear program
$$
\begin{aligned}
\min_{\theta\in\mathbb{R}^d}\quad &V_0(\theta) \\
\text{s.t.}\quad &V_j(\theta)\le 0,\qquad j\in[\tilde q],
\end{aligned}
$$
the flow direction is defined by the QCQP
$$
\begin{aligned}
R_{\alpha,\beta}(\theta) &= \arg\min_{\xi\in\mathbb{R}^d} \frac12\|\xi+\nabla V_0(\theta)\|^2 \\
\text{s.t.}\quad &\alpha V_j(\theta)+\nabla V_j(\theta)^\top \xi +\frac{\beta(\theta)}{2}\|\xi\|^2 \le 0,\qquad j\in[\tilde q].
\end{aligned}
$$
The associated continuous-time dynamics is
$$
\dot\theta = R_{\alpha,\beta}(\theta).
$$
The construction is called **gradient flow** because the objective drives $\xi$ toward $-\nabla V_0(\theta)$, **safe** because the admissible directions are constrained by the safety functions, and **robust** because the quadratic term $\frac{\beta(\theta)}{2}\|\xi\|^2$ creates explicit margin against perturbations and estimation errors [2510.01492].

Under MFCQ, if $u_j(\theta)$ denotes the Lagrange multiplier of the $j$-th constraint, the solution has the closed form
$$
R_{\alpha,\beta}(\theta) = -\frac{\nabla V_0(\theta)+\sum_{j=1}^{\tilde q}u_j(\theta)\nabla V_j(\theta)}
{1+\beta(\theta)\sum_{j=1}^{\tilde q}u_j(\theta)}.
$$
This makes the direction a scaled combination of the objective gradient and active constraint gradients [2510.01492].

A key dissipation inequality follows from feasibility of the QCQP constraints:
$$
\nabla V_j(\theta)^\top R_{\alpha,\beta}(\theta) \le -\alpha V_j(\theta)-\frac{\beta(\theta)}{2}\|R_{\alpha,\beta}(\theta)\|^2.
$$
On the boundary $V_j(\theta)=0$, this yields
$$
\nabla V_j(\theta)^\top R_{\alpha,\beta}(\theta)\le 0,
$$
so the vector field lies in the tangent cone of the feasible set. By Nagumo’s theorem, the paper proves that the feasible set is forward invariant under the flow [2510.01492].

The equilibrium structure is equally central. The paper proves that if $R_{\alpha,\beta}(\theta^*)=0$, then $\theta^*\in C$, and under MFCQ, $R_{\alpha,\beta}(\theta^*)=0$ if and only if $\theta^*$ is a KKT point. Thus, the equilibria of the flow coincide exactly with KKT points of the constrained problem [2510.01492].

## 3. Discrete algorithm and off-policy estimation

The practical algorithm is the forward-Euler discretization
$$
\theta_{i+1}=\theta_i+h_i R_{\alpha,\beta}(\theta_i),
$$
with step sizes $h_i>0$. Since $V_j(\theta_i)$ and $\nabla V_j(\theta_i)$ are unknown, RSGF-RL replaces them with data-driven estimates and computes the sampled direction by solving
$$
\begin{aligned}
\hat{R}_{\alpha,\beta}(\theta) &= \arg\min_{\xi\in\mathbb{R}^d} \frac12\|\xi+\widehat{\nabla V_0}(\theta)\|^2 \\
\text{s.t.}\quad &\alpha \widehat V_j(\theta)+\widehat{\nabla V_j}(\theta)^\top \xi +\frac{\beta(\theta)}{2}\|\xi\|^2 \le 0,\qquad j\in[q],\\
&\alpha V_{q+1}(\theta)+\nabla V_{q+1}(\theta)^\top \xi +\frac{\beta(\theta)}{2}\|\xi\|^2 \le 0.
\end{aligned}
$$
The actual update is
$$
\theta_{i+1}=\theta_i+h_i\hat R_{\alpha,\beta}(\theta_i).
$$
The additional constraint
$$
V_{q+1}(\theta)=\|\theta\|^2-C
$$
is introduced to enforce a compact parameter set $\Theta=\{\theta:\|\theta\|^2\le C\}$ [2510.01492].

The algorithm is explicitly **off-policy**. At iteration $i$, estimation may use an offline batch $I_0$ generated by policies in a set $\Lambda$ together with previous and current batches $I_1,\dots,I_i$, selecting any subset
$$
J_i\subset \bigcup_{j=0}^{i}I_j.
$$
If the $n$-th episode in $J_i$ was generated by behavior policy $\zeta_n$, then the value estimator is
$$
\widehat{V_j}(\theta_i) = \frac{\sigma_j}{|J_i|} \bigg( \sum_{n=1}^{|J_i|} \prod_{t=0}^T \frac{\pi_{\theta_i}(a_t^n|s_t^n)}{\zeta_n(a_t^n|s_t^n)} \sum_{t=0}^T \gamma^t R_j(s_t^n,a_t^n,s_{t+1}^n) \bigg),
$$
where $\sigma_0=-1$ and $\sigma_j=1$ for $j\in[q]$ [2510.01492].

The gradient estimator is REINFORCE-style with baseline:
$$
\widehat{\nabla V_j}(\theta_i) = \frac{\sigma_j}{|J_i|} \bigg( \sum_{n=1}^{|J_i|} \prod_{t=0}^T \frac{\pi_{\theta_i}(a_t^n|s_t^n)}{\zeta_n(a_t^n|s_t^n)} \sum_{t=0}^T \gamma^t \nabla \chi_{a_t^n,s_t^n}(\theta_i)\, D_{j,t}^n \bigg),
$$
with
$$
D_{j,t}^n = \sum_{t'=t}^{T}\gamma^{t'-t}R_j(s_{t'}^n,a_{t'}^n,s_{t'+1}^n)-b(s_t^n),
$$
and $\chi_{a,s}(\theta)=\log \pi_\theta(a|s)$ [2510.01492].

The paper does not formulate RSGF-RL as an actor-critic method. Its formal estimation theory is based on trajectory-wise importance sampling and score-function gradients, with the baseline $b(s)$ as the explicit variance-reduction device [2510.01492].

## 4. Finite-sample safety guarantees and recovery mechanism

The safety theorem is based on a step-size condition and a robustness margin. The step size must satisfy
$$
h_i < \min\Big\{\frac1\alpha,\frac{\beta(\theta_i)}{L_1},\dots,\frac{\beta(\theta_i)}{L_q},\frac{\beta(\theta_i)}{L_{q+1}}\Big\},
$$
where $L_j$ are Lipschitz constants of $\nabla V_j$ [2510.01492].

For each constraint, the paper defines
$$
\hat M_{i,j} = \frac{ -(1-\alpha h_i)\widehat V_j(\theta_i) + \frac{h_i}{2}\big(\beta(\theta_i)-L_j h_i\big)\|\hat R_{\alpha,\beta}(\theta_i)\|^2 }{ 1+h_i\|\hat R_{\alpha,\beta}(\theta_i)\| }.
$$
This is the finite-sample robustness buffer: simultaneous errors in value and gradient estimation smaller than this quantity can be tolerated while preserving one-step safety [2510.01492].

The required sample conditions are
$$
\frac{|J_i|^2}{\bar N_i\phi_j^2+\tilde N_i\bar\phi_j^2} \ge -\frac{2}{\hat M_{i,j}^2}\log\frac{\delta}{2},
\qquad
\frac{|J_i|^2}{\bar N_i\psi_j^2+\tilde N_i\bar\psi_j^2} \ge -\frac{2d}{\hat M_{i,j}^2}\log\frac{\delta}{2d},
$$
with $\bar N_i$ the number of episodes in $J_i$ generated by the current policy, $\tilde N_i=|J_i|-\bar N_i$, and constants $\phi_j,\bar\phi_j,\psi_j,\bar\psi_j$ determined by the reward bounds, horizon, discount factor, and action-probability lower bound [2510.01492].

Under these conditions, if $\widehat V_j(\theta_i)\le 0$, then
$$
\mathbb P\big(V_j(\theta_{i+1})\le 0\big)\ge 1-2\delta.
$$
Thus, a policy estimated safe updates to a truly safe next policy with high probability [2510.01492].

The same theorem also yields a **recovery guarantee**. If $\widehat V_j(\theta_i)>0$ but $\hat M_{i,j}>0$ and the sample conditions hold, then
$$
\mathbb P\big(V_j(\theta_{i+1})\le 0\big)\ge 1-2\delta.
$$
This is the one-step recovery statement for a mildly unsafe policy [2510.01492].

The mechanism is based on combining the sampled QCQP constraints with the descent lemma
$$
V_j(\theta_{i+1}) \le V_j(\theta_i) +\nabla V_j(\theta_i)^\top(\theta_{i+1}-\theta_i) +\frac{L_j}{2}\|\theta_{i+1}-\theta_i\|^2.
$$
The quadratic robustness term in the QCQP is therefore not merely regularization; in the paper’s analysis it is part of the margin that compensates for estimation uncertainty [2510.01492].

## 5. Statistical analysis and asymptotic convergence

RSGF-RL’s statistical analysis begins with unbiasedness:
$$
\mathbb E[\widehat V_j(\theta_i)] = V_j(\theta_i), \qquad \mathbb E[\widehat{\nabla V_j}(\theta_i)] = \nabla V_j(\theta_i).
$$
The paper also gives variance expressions for the off-policy estimators and Hoeffding-type concentration inequalities for both values and gradient components [2510.01492].

For the value estimator,
$$
\operatorname{Var}[\widehat V_j(\theta_i)] = \frac{\bar N_i\phi_j^2+\tilde N_i\bar\phi_j^2}{|J_i|^2},
$$
and the tail bound is
$$
\mathbb P\big(|\widehat V_j(\theta_i)-V_j(\theta_i)|\le \epsilon\big) \ge 1-2\exp\Big( -\frac{\epsilon^2|J_i|^2}{2\bar N_i\phi_j^2+2\tilde N_i\bar\phi_j^2} \Big).
$$
For the gradient estimator, the componentwise variance and concentration bounds are given analogously in terms of $\psi_j$ and $\bar\psi_j$ [2510.01492].

The asymptotic convergence argument rewrites the algorithm as stochastic approximation:
$$
\theta_{i+1} = \theta_i +h_i R_{\alpha,\beta}(\theta_i) +h_i\big(\hat R_{\alpha,\beta}(\theta_i)-R_{\alpha,\beta}(\theta_i)\big).
$$
Under regularity assumptions, feasibility of the sampled QCQP, initialization in the bounded set, asymptotically vanishing QCQP error
$$
\lim_{i\to\infty}\|R_{\alpha,\beta}(\theta_i)-\hat R_{\alpha,\beta}(\theta_i)\|=0
\quad\text{a.s.},
$$
and step-size conditions
$$
h_i\to 0,\qquad \sum_{i=1}^{\infty} h_i = \infty,
$$
the iterates converge almost surely to the set of KKT points of the constrained RL problem in $\Theta$ [2510.01492].

The continuous-time convergence result provides the template. Inside the feasible set, the objective $V_0$ is nonincreasing along the flow, and for global convergence the paper introduces the exact-penalty-like function
$$
V_{\epsilon_*}(\theta)=V_0(\theta)+\frac1{\epsilon_*}\sum_{j=1}^q [V_j(\theta)]_+,
$$
which is nonincreasing along the flow on compact sets for small enough $\epsilon_*$ [2510.01492]. This suggests that the robust flow is designed not only to preserve safety but also to organize descent toward KKT points from both safe and mildly unsafe initial conditions.

## 6. Empirical behavior, positioning, and disambiguation

The paper evaluates RSGF-RL on a **navigation example** and the **cart-pole system**. In the navigation problem, a single-integrator robot in $[0,10]^2$ must reach a goal while avoiding obstacles; in cart-pole, the state is $(x,\theta,\dot x,\dot\theta)$, the action is a continuous force in $[-3,3]$, the reward is $R_0(s,a)=1$, and the safety set requires $x<0.5$ [2510.01492].

The baselines are **off-policy RSGF-RL**, **on-policy RSGF-RL from the conference version**, and **CPO**. The evaluation reports task performance through $V_0(\theta)$ and safety through $V_1(\theta)$. Across both tasks, the paper reports that RSGF-RL maintains safety during training or recovers quickly from unsafe initial conditions, that both on-policy and off-policy variants outperform CPO in reward performance, and that **off-policy RSGF-RL with clipped importance weights performs best** in practice [2510.01492]. The clipped importance weights are described as an empirical modification; they are not part of the theoretical estimator.

Within safe RL, RSGF-RL is positioned as a policy-parameter update rule that solves a convex QCQP at each iteration rather than using dual ascent or action-level shielding. This suggests a design centered on directly computing a feasible local descent direction in parameter space. The paper’s own comparison emphasizes superior performance with respect to the state of the art on its reported experiments [2510.01492].

The acronym is potentially confusable. It should not be conflated with **SG-RL**, a hierarchical path planning framework combining Simple Subgoal Graphs and LSPI for rational pathfinding in continuous and uncertain environments [1811.01700]; with **RSG**, the Robot Skill Graph framework for organizing and inferring quadruped skills [2311.06015]; or with **RSF-GLLM**, whose core module is Recurrent Soft-Flow for multi-hop KGQA and which does **not** use reinforcement learning [2607.06527]. In the literature represented here, **RSGF-RL** refers specifically to **“Off-Policy Reinforcement Learning with Anytime Safety Guarantees via Robust Safe Gradient Flow”** [2510.01492].

Source: https://www.emergentmind.com/topics/rsgf-rl