Rectified Robust Policy Optimization (RRPO)

• The paper's main contribution is the introduction of a primal-only algorithm (RRPO) for robust constrained reinforcement learning that addresses the failure of strong duality.
• RRPO sequentially rectifies constraint violations and improves the objective using robust policy evaluation and a natural policy gradient update.
• Empirical results show that RRPO outperforms traditional methods in safety and robustness in environments like gridworld, Mountain Car, and HalfCheetah.

Searching arXiv for the specified RRPO-related papers to ground the article and clarify naming/disambiguation. Rectified Robust Policy Optimization (RRPO) denotes a primal-only algorithm for model-uncertain constrained reinforcement learning in which both performance and constraints are evaluated under worst-case transition uncertainty. In the formulation developed in "Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality" (Ma et al., 24 Aug 2025), RRPO is motivated by a negative result: strong duality does not generally hold in robust constrained RL, so traditional primal-dual methods may fail to find optimal feasible policies. The method therefore operates directly on the primal problem, using robust policy evaluation, a rectification rule that alternates between improving violated constraints and improving the objective, and a threshold-tracking mechanism that preserves the best feasible policy found so far. The acronym RRPO has also been used in other contexts, notably robust reward modeling for emotional text-to-speech (Wang et al., 4 Dec 2025), so the term requires domain-specific disambiguation.

1. Scope and nomenclature

In its most specific and technically developed sense, RRPO refers to the algorithm proposed for robust constrained RL under model uncertainty in (Ma et al., 24 Aug 2025). The paper studies a robust constrained Markov decision process in which the transition kernel is not fixed, but lies in an uncertainty set $\mathcal P$, and it frames the central difficulty as the failure of strong duality in this setting. The resulting method is explicitly described as a primal-only algorithm that does not rely on dual formulations.

The acronym is not unique across the literature. "RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTS" (Wang et al., 4 Dec 2025) uses RRPO to denote a differentiable RL framework for LLM-based emotional text-to-speech, where the central intervention is a robust reward model rather than a robust constrained MDP solver. A broader naming vicinity also includes Rectified Policy Optimization (RePO) for LLM safety alignment (Peng et al., 2024), which replaces expected safety constraints with prompt-wise strict safety penalties, and the earlier baseline-guarantee framework in "Robust Policy Optimization with Baseline Guarantees" (Chow et al., 2015), whose final robust-baseline combination is described as being “in the spirit of rectified robust policy optimization.” This suggests that the phrase “rectified robust policy optimization” has acquired multiple, partially overlapping meanings, with (Ma et al., 24 Aug 2025) being the paper that makes it the explicit name of a robust constrained RL algorithm.

2. Robust constrained RL formulation

The problem setting in (Ma et al., 24 Aug 2025) is a robust MDP described by

$(\mathcal S,\mathcal A,\mathcal P,r,\gamma),$

with finite state space $\mathcal S$, finite action space $\mathcal A$, reward $r$, discount $\gamma\in[0,1)$, and initial distribution $\mu\in\Delta(\mathcal S)$. For a policy $\pi:\mathcal S\to\Delta(\mathcal A)$, the robust value function is

$$V^\pi(s) = \inf_{P\in\mathcal P} \mathbb E_{\pi,P}\!\left[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\,\middle|\, s_0=s\right],$$

with corresponding initial-state value

$V^\pi(\mu)=\mathbb E_{s\sim \mu}[V^\pi(s)].$

The formulation is extended to multiple constraints by introducing reward functions $(\mathcal S,\mathcal A,\mathcal P,r,\gamma),$0, $(\mathcal S,\mathcal A,\mathcal P,r,\gamma),$1, and corresponding worst-case values

$(\mathcal S,\mathcal A,\mathcal P,r,\gamma),$2

The robust constrained RL problem is then

$(\mathcal S,\mathcal A,\mathcal P,r,\gamma),$3

A defining property of this formulation is that both the objective and the constraints are evaluated in the worst case over the uncertainty set. The paper keeps $(\mathcal S,\mathcal A,\mathcal P,r,\gamma),$4 general, while emphasizing $(\mathcal S,\mathcal A,\mathcal P,r,\gamma),$5-rectangular and $(\mathcal S,\mathcal A,\mathcal P,r,\gamma),$6-rectangular uncertainty, and explicitly instantiating $(\mathcal S,\mathcal A,\mathcal P,r,\gamma),$7-norm and IPM uncertainty sets: $(\mathcal S,\mathcal A,\mathcal P,r,\gamma),$8 and

$(\mathcal S,\mathcal A,\mathcal P,r,\gamma),$9

This suggests that RRPO is intended as a general optimization layer above an available robust evaluation procedure rather than as a method tied to a single uncertainty geometry.

3. Duality failure and the rationale for rectification

A standard constrained RL route uses the Lagrangian

$\mathcal S$0

and studies

$\mathcal S$1

The central negative result of (Ma et al., 24 Aug 2025) is that strong duality does not generally hold in robust constrained RL. The paper defines the duality gap as

$\mathcal S$2

and gives an explicit 2-state, 2-action counterexample showing a strictly positive gap. For

$\mathcal S$3

the paper reports

$\mathcal S$4

The explanation given is structural: the “mixing of policies implies mixing of visitation distributions” property used in classic duality proofs does not carry over cleanly under worst-case transitions. The robust state-visitation distribution and value structure are therefore not convex in the policy in the way required by the standard proof route. This matters algorithmically because a primal-dual method may optimize the wrong saddle problem and fail to recover the true robust constrained optimum.

The “rectified” character of RRPO follows from this diagnosis. Rather than relying on a global dual variable to balance objective and constraints, the method selectively updates whichever component currently blocks primal progress. In that sense, rectification means local corrective emphasis on the violated constraint or lagging objective, rather than dual compensation across terms.

4. Algorithmic structure

RRPO reformulates the primal problem as

$\mathcal S$5

The algorithm maintains the current policy $\mathcal S$6, a threshold variable $\mathcal S$7 for the objective, a set $\mathcal S$8 of policies that are feasible within tolerance and achieve the best known objective, and a tolerance $\mathcal S$9 for constraint violation (Ma et al., 24 Aug 2025).

At each iteration, RRPO first estimates robust Q-functions $\mathcal A$0 for all $\mathcal A$1, and then computes approximate robust values $\mathcal A$2. It next performs a feasibility test. If

$\mathcal A$3

the policy is treated as feasible up to tolerance; RRPO adds $\mathcal A$4 to $\mathcal A$5, tracks the best feasible policy as $\mathcal A$6, and updates the objective threshold via

$\mathcal A$7

If instead some constraint is violated,

$\mathcal A$8

the method performs constraint rectification. If the objective is below the current threshold,

$\mathcal A$9

it performs objective rectification.

The policy improvement rule is a robust natural policy gradient step: $r$0 with

$r$1

Under softmax parameterization $r$2, this is equivalent to

$r$3

The index $r$4 is chosen according to whether the step is rectifying a violated constraint or the objective.

A key practical distinction from CRPO is that RRPO explicitly keeps the best feasible policy and updates the objective threshold $r$5. The paper states that CRPO may oscillate near the constraint boundary because it does not “remember” the best feasible objective, whereas RRPO reduces output variance by preserving that information.

5. Theoretical properties and assumptions

The convergence result in (Ma et al., 24 Aug 2025) is formulated under mild regularity assumptions. These include policy evaluation accuracy,

$r$6

worst-case exploration,

$r$7

bounded rewards,

$r$8

and bounded uncertainty diameter,

$r$9

With learning rate $\gamma\in[0,1)$0, tolerance $\gamma\in[0,1)$1, and evaluation error $\gamma\in[0,1)$2, RRPO outputs $\gamma\in[0,1)$3 such that

$\gamma\in[0,1)$4

and

$\gamma\in[0,1)$5

The fuller theorem gives

$\gamma\in[0,1)$6

The paper interprets these bounds as approximate optimality among feasible policies together with feasibility up to tolerance. It also states that the iteration complexity is

$\gamma\in[0,1)$7

matching the best-known lower bound for constrained RL in the non-robust setting up to the additional $\gamma\in[0,1)$8 term, and that for $\gamma\in[0,1)$9-norm uncertainty with robust TD evaluation, each evaluation can take $\mu\in\Delta(\mathcal S)$0 samples, giving total sample complexity

$\mu\in\Delta(\mathcal S)$1

These results are important because they replace strong-duality-based guarantees with a directly primal analysis. A plausible implication is that RRPO is best understood not as a correction to dual optimization, but as a re-specification of constrained robust RL around feasible-policy tracking and robust improvement steps.

6. Empirical behavior, limitations, and related variants

The experiments in (Ma et al., 24 Aug 2025) are designed to test whether RRPO remains safe under model mismatch and worst-case perturbations. In a FrozenLake-like $\mu\in\Delta(\mathcal S)$2 gridworld, training is deterministic but test dynamics are “slippery” with slip probability $\mu\in\Delta(\mathcal S)$3, modeled as uncertainty. The reward is

$\mu\in\Delta(\mathcal S)$4

and the constraint reward is

$\mu\in\Delta(\mathcal S)$5

With constraint requirement $\mu\in\Delta(\mathcal S)$6, RRPO learns the safer, longer path that avoids worst-case constraint violations, whereas CRPO tends to choose the shorter path and can violate the constraint in slippery or worst-case conditions.

In Mountain Car, the rewards are

$\mu\in\Delta(\mathcal S)$7

and the worst-case perturbation increases gravity from $\mu\in\Delta(\mathcal S)$8 to $\mu\in\Delta(\mathcal S)$9. The reported outcome is that RRPO obtains lower worst-case cost and better robustness than the non-robust baseline. In HalfCheetah-v4, with

$\pi:\mathcal S\to\Delta(\mathcal A)$0

RRPO respects the energy constraint and outperforms the robust primal-dual baseline under the feasible regime, while the robust policy gradient attains high reward but violates the constraint.

The limitations stated or implied in (Ma et al., 24 Aug 2025) are also central to the concept. RRPO requires sufficiently accurate robust value and Q estimation; its analysis assumes worst-case exploration; its optimality guarantee contains an additive $\pi:\mathcal S\to\Delta(\mathcal A)$1 uncertainty-diameter penalty; and its sample complexity can be high. The paper further indicates that the method is strongest when the uncertainty set is moderate or small, robust policy evaluation is available, and worst-case safety matters more than average-case performance.

The broader literature uses related rectification ideas in different ways. RePO for LLM safety alignment replaces expected safety constraints with prompt-wise strict penalties $\pi:\mathcal S\to\Delta(\mathcal A)$2, thereby preventing cross-prompt “safety compensation” (Peng et al., 2024). The baseline-guarantee framework in model-based RL shows that a policy can be “rectified” by preserving the baseline where uncertainty is large and improving only where the model is reliable (Chow et al., 2015). In emotional TTS, RRPO denotes robust reward policy optimization in which a robust reward model—trained with label smoothing, energy-adaptive mixup, and adversarial training—is used to prevent reward hacking during differentiable policy optimization (Wang et al., 4 Dec 2025). These uses are technically distinct, but they share a common pattern: optimization is corrected by an explicit mechanism that prevents hidden compensation, whether across prompts, across uncertain regions of the state space, or across exploitable reward-model blind spots.