RSGF-RL: Anytime Safe Constrained RL

• RSGF-RL is an off-policy constrained RL method that guarantees anytime safety by ensuring every policy update satisfies predefined constraints via solving a convex QCQP.
• It is derived from Robust Safe Gradient Flow, leveraging continuous-time dynamics to combine objective descent with safety constraints and robustness margins against estimation errors.
• The algorithm employs off-policy estimation through importance sampling and finite-sample safety buffers, with convergence guarantees to KKT points under proper step-size conditions.

Searching arXiv for the exact method name and primary source paper. Searching arXiv for related acronym collisions and nearby methods to support disambiguation. RSGF-RL is an off-policy method for constrained reinforcement learning derived from the Robust Safe Gradient Flow (RSGF), a continuous-time dynamics for nonlinear constrained optimization. Its defining objective is to provide anytime safety guarantees: with a user-specified probability and sufficient data, each policy update yields a constraint-satisfying policy, and a mildly unsafe policy can be driven back to safety in one step with high probability. The method estimates value functions and their gradients from episodic data and computes each policy update by solving a convex quadratically constrained quadratic program (QCQP) (Mestres et al., 1 Oct 2025).

1. Constrained RL setting and meaning of anytime safety

RSGF-RL is formulated for a finite-horizon discounted constrained Markov decision process

$$\mathcal{M} = (\mathcal{S},\mathcal{A},P,R_0,\{R_j\}_{j=1}^q),$$

with task reward $R_0$, safety-cost functions $R_j$, discount factor $\gamma\in(0,1)$, horizon $T\in\mathbb{Z}_{>0}$, and parameterized policy $\pi_\theta$, $\theta\in\mathbb{R}^d$. The paper writes the optimization problem as

$$\begin{aligned} \min_{\theta\in\mathbb{R}^d}\quad V_0(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} -\gamma^k R_0(s_k, a_k, s_{k+1}) \biggr] \ \text{s.t.}\quad V_j(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} \gamma^k R_j(s_k, a_k, s_{k+1}) \biggr] \le 0,\qquad j\in[q]. \end{aligned}$$

The sign of the reward is flipped so that the task is written as a minimization problem (Mestres et al., 1 Oct 2025).

The corresponding feasible parameter set is

$$C = \{\theta\in\mathbb{R}^d : V_j(\theta)\le 0,\ \forall j\in[q]\}.$$

Because the value functions induced by a parameterized policy are generally nonconvex in $\theta$, the method targets convergence to KKT points rather than global optima (Mestres et al., 1 Oct 2025).

In this framework, anytime safety means that if the algorithm generates policy iterates $R_0$0, then constraint satisfaction is preserved at every training iterate with prescribed probability. The finite-horizon statement is

$R_0$1

provided the paper’s per-iteration sample conditions hold (Mestres et al., 1 Oct 2025). This is a training-time property, not merely a guarantee on the terminal policy.

The paper also notes that suitable cumulative constraints can encode state-space safety with confidence. If

$R_0$2

then satisfying $R_0$3 implies

$R_0$4

This links CMDP constraints to high-probability state safety (Mestres et al., 1 Oct 2025).

2. Continuous-time foundation: Robust Safe Gradient Flow

RSGF-RL is built from a continuous-time constrained optimization dynamics called the Robust Safe Gradient Flow. For the generic nonlinear program

$R_0$5

the flow direction is defined by the QCQP

$R_0$6

The associated continuous-time dynamics is

$R_0$7

The construction is called gradient flow because the objective drives $R_0$8 toward $R_0$9, safe because the admissible directions are constrained by the safety functions, and robust because the quadratic term $R_j$0 creates explicit margin against perturbations and estimation errors (Mestres et al., 1 Oct 2025).

Under MFCQ, if $R_j$1 denotes the Lagrange multiplier of the $R_j$2-th constraint, the solution has the closed form

$R_j$3

This makes the direction a scaled combination of the objective gradient and active constraint gradients (Mestres et al., 1 Oct 2025).

A key dissipation inequality follows from feasibility of the QCQP constraints:

$R_j$4

On the boundary $R_j$5, this yields

$R_j$6

so the vector field lies in the tangent cone of the feasible set. By Nagumo’s theorem, the paper proves that the feasible set is forward invariant under the flow (Mestres et al., 1 Oct 2025).

The equilibrium structure is equally central. The paper proves that if $R_j$7, then $R_j$8, and under MFCQ, $R_j$9 if and only if $\gamma\in(0,1)$0 is a KKT point. Thus, the equilibria of the flow coincide exactly with KKT points of the constrained problem (Mestres et al., 1 Oct 2025).

3. Discrete algorithm and off-policy estimation

The practical algorithm is the forward-Euler discretization

$\gamma\in(0,1)$1

with step sizes $\gamma\in(0,1)$2. Since $\gamma\in(0,1)$3 and $\gamma\in(0,1)$4 are unknown, RSGF-RL replaces them with data-driven estimates and computes the sampled direction by solving

$\gamma\in(0,1)$5

The actual update is

$\gamma\in(0,1)$6

The additional constraint

$\gamma\in(0,1)$7

is introduced to enforce a compact parameter set $\gamma\in(0,1)$8 (Mestres et al., 1 Oct 2025).

The algorithm is explicitly off-policy. At iteration $\gamma\in(0,1)$9, estimation may use an offline batch $T\in\mathbb{Z}_{>0}$0 generated by policies in a set $T\in\mathbb{Z}_{>0}$1 together with previous and current batches $T\in\mathbb{Z}_{>0}$2, selecting any subset

$T\in\mathbb{Z}_{>0}$3

If the $T\in\mathbb{Z}_{>0}$4-th episode in $T\in\mathbb{Z}_{>0}$5 was generated by behavior policy $T\in\mathbb{Z}_{>0}$6, then the value estimator is

$T\in\mathbb{Z}_{>0}$7

where $T\in\mathbb{Z}_{>0}$8 and $T\in\mathbb{Z}_{>0}$9 for $\pi_\theta$0 (Mestres et al., 1 Oct 2025).

The gradient estimator is REINFORCE-style with baseline:

$\pi_\theta$1

with

$\pi_\theta$2

and $\pi_\theta$3 (Mestres et al., 1 Oct 2025).

The paper does not formulate RSGF-RL as an actor-critic method. Its formal estimation theory is based on trajectory-wise importance sampling and score-function gradients, with the baseline $\pi_\theta$4 as the explicit variance-reduction device (Mestres et al., 1 Oct 2025).

4. Finite-sample safety guarantees and recovery mechanism

The safety theorem is based on a step-size condition and a robustness margin. The step size must satisfy

$\pi_\theta$5

where $\pi_\theta$6 are Lipschitz constants of $\pi_\theta$7 (Mestres et al., 1 Oct 2025).

For each constraint, the paper defines

$\pi_\theta$8

This is the finite-sample robustness buffer: simultaneous errors in value and gradient estimation smaller than this quantity can be tolerated while preserving one-step safety (Mestres et al., 1 Oct 2025).

The required sample conditions are

$\pi_\theta$9

with $\theta\in\mathbb{R}^d$0 the number of episodes in $\theta\in\mathbb{R}^d$1 generated by the current policy, $\theta\in\mathbb{R}^d$2, and constants $\theta\in\mathbb{R}^d$3 determined by the reward bounds, horizon, discount factor, and action-probability lower bound (Mestres et al., 1 Oct 2025).

Under these conditions, if $\theta\in\mathbb{R}^d$4, then

$\theta\in\mathbb{R}^d$5

Thus, a policy estimated safe updates to a truly safe next policy with high probability (Mestres et al., 1 Oct 2025).

The same theorem also yields a recovery guarantee. If $\theta\in\mathbb{R}^d$6 but $\theta\in\mathbb{R}^d$7 and the sample conditions hold, then

$\theta\in\mathbb{R}^d$8

This is the one-step recovery statement for a mildly unsafe policy (Mestres et al., 1 Oct 2025).

The mechanism is based on combining the sampled QCQP constraints with the descent lemma

$\theta\in\mathbb{R}^d$9

The quadratic robustness term in the QCQP is therefore not merely regularization; in the paper’s analysis it is part of the margin that compensates for estimation uncertainty (Mestres et al., 1 Oct 2025).

5. Statistical analysis and asymptotic convergence

RSGF-RL’s statistical analysis begins with unbiasedness:

$$\begin{aligned} \min_{\theta\in\mathbb{R}^d}\quad V_0(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} -\gamma^k R_0(s_k, a_k, s_{k+1}) \biggr] \ \text{s.t.}\quad V_j(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} \gamma^k R_j(s_k, a_k, s_{k+1}) \biggr] \le 0,\qquad j\in[q]. \end{aligned}$$0

The paper also gives variance expressions for the off-policy estimators and Hoeffding-type concentration inequalities for both values and gradient components (Mestres et al., 1 Oct 2025).

For the value estimator,

$$\begin{aligned} \min_{\theta\in\mathbb{R}^d}\quad V_0(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} -\gamma^k R_0(s_k, a_k, s_{k+1}) \biggr] \ \text{s.t.}\quad V_j(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} \gamma^k R_j(s_k, a_k, s_{k+1}) \biggr] \le 0,\qquad j\in[q]. \end{aligned}$$1

and the tail bound is

$$\begin{aligned} \min_{\theta\in\mathbb{R}^d}\quad V_0(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} -\gamma^k R_0(s_k, a_k, s_{k+1}) \biggr] \ \text{s.t.}\quad V_j(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} \gamma^k R_j(s_k, a_k, s_{k+1}) \biggr] \le 0,\qquad j\in[q]. \end{aligned}$$2

For the gradient estimator, the componentwise variance and concentration bounds are given analogously in terms of $$\begin{aligned} \min_{\theta\in\mathbb{R}^d}\quad V_0(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} -\gamma^k R_0(s_k, a_k, s_{k+1}) \biggr] \ \text{s.t.}\quad V_j(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} \gamma^k R_j(s_k, a_k, s_{k+1}) \biggr] \le 0,\qquad j\in[q]. \end{aligned}$$3 and $$\begin{aligned} \min_{\theta\in\mathbb{R}^d}\quad V_0(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} -\gamma^k R_0(s_k, a_k, s_{k+1}) \biggr] \ \text{s.t.}\quad V_j(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} \gamma^k R_j(s_k, a_k, s_{k+1}) \biggr] \le 0,\qquad j\in[q]. \end{aligned}$$4 (Mestres et al., 1 Oct 2025).

The asymptotic convergence argument rewrites the algorithm as stochastic approximation:

$$\begin{aligned} \min_{\theta\in\mathbb{R}^d}\quad V_0(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} -\gamma^k R_0(s_k, a_k, s_{k+1}) \biggr] \ \text{s.t.}\quad V_j(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} \gamma^k R_j(s_k, a_k, s_{k+1}) \biggr] \le 0,\qquad j\in[q]. \end{aligned}$$5

Under regularity assumptions, feasibility of the sampled QCQP, initialization in the bounded set, asymptotically vanishing QCQP error

$$\begin{aligned} \min_{\theta\in\mathbb{R}^d}\quad V_0(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} -\gamma^k R_0(s_k, a_k, s_{k+1}) \biggr] \ \text{s.t.}\quad V_j(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} \gamma^k R_j(s_k, a_k, s_{k+1}) \biggr] \le 0,\qquad j\in[q]. \end{aligned}$$6

and step-size conditions

$$\begin{aligned} \min_{\theta\in\mathbb{R}^d}\quad V_0(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} -\gamma^k R_0(s_k, a_k, s_{k+1}) \biggr] \ \text{s.t.}\quad V_j(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} \gamma^k R_j(s_k, a_k, s_{k+1}) \biggr] \le 0,\qquad j\in[q]. \end{aligned}$$7

the iterates converge almost surely to the set of KKT points of the constrained RL problem in $$\begin{aligned} \min_{\theta\in\mathbb{R}^d}\quad V_0(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} -\gamma^k R_0(s_k, a_k, s_{k+1}) \biggr] \ \text{s.t.}\quad V_j(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} \gamma^k R_j(s_k, a_k, s_{k+1}) \biggr] \le 0,\qquad j\in[q]. \end{aligned}$$8 (Mestres et al., 1 Oct 2025).

The continuous-time convergence result provides the template. Inside the feasible set, the objective $$\begin{aligned} \min_{\theta\in\mathbb{R}^d}\quad V_0(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} -\gamma^k R_0(s_k, a_k, s_{k+1}) \biggr] \ \text{s.t.}\quad V_j(\theta) &= \mathbb{E}_{\substack{a \sim \pi_{\theta}(\cdot|s) \ s_0 \sim \eta}} \biggl[ \sum_{k=0}^{T} \gamma^k R_j(s_k, a_k, s_{k+1}) \biggr] \le 0,\qquad j\in[q]. \end{aligned}$$9 is nonincreasing along the flow, and for global convergence the paper introduces the exact-penalty-like function

$$C = \{\theta\in\mathbb{R}^d : V_j(\theta)\le 0,\ \forall j\in[q]\}.$$0

which is nonincreasing along the flow on compact sets for small enough $$C = \{\theta\in\mathbb{R}^d : V_j(\theta)\le 0,\ \forall j\in[q]\}.$$1 (Mestres et al., 1 Oct 2025). This suggests that the robust flow is designed not only to preserve safety but also to organize descent toward KKT points from both safe and mildly unsafe initial conditions.

6. Empirical behavior, positioning, and disambiguation

The paper evaluates RSGF-RL on a navigation example and the cart-pole system. In the navigation problem, a single-integrator robot in $$C = \{\theta\in\mathbb{R}^d : V_j(\theta)\le 0,\ \forall j\in[q]\}.$$2 must reach a goal while avoiding obstacles; in cart-pole, the state is $$C = \{\theta\in\mathbb{R}^d : V_j(\theta)\le 0,\ \forall j\in[q]\}.$$3, the action is a continuous force in $$C = \{\theta\in\mathbb{R}^d : V_j(\theta)\le 0,\ \forall j\in[q]\}.$$4, the reward is $$C = \{\theta\in\mathbb{R}^d : V_j(\theta)\le 0,\ \forall j\in[q]\}.$$5, and the safety set requires $$C = \{\theta\in\mathbb{R}^d : V_j(\theta)\le 0,\ \forall j\in[q]\}.$$6 (Mestres et al., 1 Oct 2025).

The baselines are off-policy RSGF-RL, on-policy RSGF-RL from the conference version, and CPO. The evaluation reports task performance through $$C = \{\theta\in\mathbb{R}^d : V_j(\theta)\le 0,\ \forall j\in[q]\}.$$7 and safety through $$C = \{\theta\in\mathbb{R}^d : V_j(\theta)\le 0,\ \forall j\in[q]\}.$$8. Across both tasks, the paper reports that RSGF-RL maintains safety during training or recovers quickly from unsafe initial conditions, that both on-policy and off-policy variants outperform CPO in reward performance, and that off-policy RSGF-RL with clipped importance weights performs best in practice (Mestres et al., 1 Oct 2025). The clipped importance weights are described as an empirical modification; they are not part of the theoretical estimator.

Within safe RL, RSGF-RL is positioned as a policy-parameter update rule that solves a convex QCQP at each iteration rather than using dual ascent or action-level shielding. This suggests a design centered on directly computing a feasible local descent direction in parameter space. The paper’s own comparison emphasizes superior performance with respect to the state of the art on its reported experiments (Mestres et al., 1 Oct 2025).

The acronym is potentially confusable. It should not be conflated with SG-RL, a hierarchical path planning framework combining Simple Subgoal Graphs and LSPI for rational pathfinding in continuous and uncertain environments (Zeng et al., 2018); with RSG, the Robot Skill Graph framework for organizing and inferring quadruped skills (Zhang et al., 2023); or with RSF-GLLM, whose core module is Recurrent Soft-Flow for multi-hop KGQA and which does not use reinforcement learning (Bandyopadhyay et al., 7 Jul 2026). In the literature represented here, RSGF-RL refers specifically to “Off-Policy Reinforcement Learning with Anytime Safety Guarantees via Robust Safe Gradient Flow” (Mestres et al., 1 Oct 2025).