RPO-AdaSharp: Reduced Policy Optimization in Constrained RL

• The paper's main contribution is introducing a GRG-inspired constrained RL method that enforces equality constraints by partitioning actions into basic and nonbasic components.
• It leverages implicit differentiation to compute reduced gradients, ensuring feasibility on hard, state-wise constraints through exact penalties and projection steps.
• Empirical results with RPO-DDPG and RPO-SAC outperform baselines on benchmarks like Safe CartPole and Spring Pendulum, achieving high rewards with near-zero constraint violations.

RPO-AdaSharp is not a standardized term in the arXiv literature. The label is most concretely associated with Reduced Policy Optimization for continuous control with hard constraints, a constrained reinforcement learning method inspired by the generalized reduced gradient (GRG) algorithm that partitions actions into basic and nonbasic components, enforces equality constraints by construction, and handles inequalities through exact-penalty optimization and reduced-gradient projection (Ding et al., 2023). At the same time, the acronym RPO is heavily overloaded: papers use it for Reward Partitioning Optimization, Relative Preference Optimization, Read-only Prompt Optimization, Robust Prompt Optimization, and several other unrelated methods, and many of those papers explicitly state that they do not define or use any component named “AdaSharp” (Faye et al., 16 Jun 2025, Yin et al., 2024).

1. Terminological scope and acronym ambiguity

Across recent work, RPO denotes multiple distinct methods rather than a single research line. In this landscape, “RPO-AdaSharp” is best understood as an ambiguous label whose most technically specified meaning here is the hard-constrained continuous-control method from constrained RL, while several other RPO papers explicitly reject any connection to “AdaSharp.” This suggests that the term functions more as a disambiguation problem than as a settled algorithmic name (Ding et al., 2023, Faye et al., 16 Jun 2025, Yin et al., 2024, Lee et al., 2023, Zhou et al., 2024).

Expansion of RPO	Domain	AdaSharp status
Reduced Policy Optimization	Continuous control with hard constraints	Presented as “RPO-AdaSharp” in the supplied details (Ding et al., 2023)
Reward Partitioning Optimization	Single-trajectory policy optimization	“The paper does not mention or use anything called ‘AdaSharp.’” (Faye et al., 16 Jun 2025)
Relative Preference Optimization	LLM alignment from identical and related prompts	“The paper does not introduce any component named ‘AdaSharp.’” (Yin et al., 2024)
Read-only Prompt Optimization	Vision-language few-shot learning	“The paper does not discuss or use any method named ‘AdaSharp.’” (Lee et al., 2023)
Robust Prompt Optimization	Jailbreak defense for LLMs	“The paper does not introduce or analyze any optimizer named ‘AdaSharp.’” (Zhou et al., 2024)

Within the hard-constrained RL usage, the method studies constrained reinforcement learning when actions must strictly satisfy general hard constraints at every step, including nonlinear and non-convex equalities and inequalities. Its objective is the infinite-horizon discounted return

$$J(\theta)=\mathbb{E}_{s_t\sim d^{\pi_\theta},\,a_t\sim \pi_\theta(\cdot|s_t)}\left[\sum_{t=0}^{\infty}\gamma^t r(s_t,a_t)\right],$$

subject to per-step equality constraints $g(s_t,a_t)=0$ and inequality constraints $h(s_t,a_t)\le 0$ (Ding et al., 2023).

2. Hard-constrained RL formulation

The method addresses a setting in which constraints are hard and instantaneous: they apply to the current state-action pair and must hold at all times. This is explicitly contrasted with typical safe RL methods that handle soft cumulative constraints over trajectories and “cannot guarantee exact satisfaction of equalities” (Ding et al., 2023).

The motivation is threefold. First, equality constraints require exact feasibility, so small approximation errors break feasibility. Second, non-convex, state-dependent constraints impede closed-form projections or convex solvers. Third, most existing hard-constrained RL methods are limited to specific linear or domain-specific settings, whereas the goal here is general hard constraints. The proposed response is to import GRG structure into RL: update only a subset of actions directly, solve the remaining action components from the equality constraints, and then differentiate through that construction (Ding et al., 2023).

The action vector $a\in\mathbb{R}^n$ is partitioned as

$a=[a_b; a_n],$

with $a_b\in\mathbb{R}^m$ and $a_n\in\mathbb{R}^{n-m}$, where $n-m$ equals the number of equality constraints. The equalities are written statewise as

$g(s,a_b,a_n)=0,$

which implicitly define

$a_n=\phi(s,a_b).$

Local solvability is guaranteed by the implicit function theorem when the Jacobian with respect to the nonbasic actions is nonsingular:

$g(s_t,a_t)=0$0

The policy network $g(s_t,a_t)=0$1 outputs only the basic actions $g(s_t,a_t)=0$2; a constraint solver then computes $g(s_t,a_t)=0$3 so that the equalities are satisfied exactly. The post-construction action is

$g(s_t,a_t)=0$4

(Ding et al., 2023)

3. GRG-inspired action construction and reduced gradients

The central technical mechanism is implicit differentiation through the equality-constrained action construction. Differentiating the equalities yields

$g(s_t,a_t)=0$5

For any scalar objective $g(s_t,a_t)=0$6, the reduced gradient with respect to the basic actions is

$g(s_t,a_t)=0$7

This is the GRG analogue inside the policy gradient pipeline (Ding et al., 2023).

In deterministic actor-critic form, with critic $g(s_t,a_t)=0$8, the reduced policy gradient is

$g(s_t,a_t)=0$9

The actor therefore receives gradient signal through both the direct dependence of $h(s_t,a_t)\le 0$0 on $h(s_t,a_t)\le 0$1 and the indirect dependence through the constructed $h(s_t,a_t)\le 0$2. In practice, the paper instantiates this through RPO-DDPG and RPO-SAC, with critics trained on the actually executed projected actions and actors updated on the pre-projection construction stage (Ding et al., 2023).

The paper states two principal theoretical properties. A proposition gives the gradient flow for the construction stage through

$h(s_t,a_t)\le 0$3

and a theorem states that when

$h(s_t,a_t)\le 0$4

the GRG projection steps lie in the tangent space of the equality manifold, exactly for linear equalities and locally for nonlinear equalities with sufficiently small step sizes (Ding et al., 2023).

4. Inequality handling, projection, and algorithmic structure

Equality satisfaction is enforced by construction, but inequalities are handled by a two-part mechanism: a modified Lagrangian relaxation in the actor loss and a reduced-gradient projection stage. After the equality-construction stage, the actor objective is augmented by a statewise exact penalty:

$h(s_t,a_t)\le 0$5

The dual-style update is

$h(s_t,a_t)\le 0$6

The paper invokes the Exact Penalty Theorem: if $h(s_t,a_t)\le 0$7, minimizing the penalized actor loss is equivalent to the constrained problem with per-state inequalities (Ding et al., 2023).

Projection is then performed with the aggregate violation objective

$h(s_t,a_t)\le 0$8

Using the reduced gradient based on $h(s_t,a_t)\le 0$9,

$a\in\mathbb{R}^n$0

The GRG projection updates are

$a\in\mathbb{R}^n$1

initialized at $a\in\mathbb{R}^n$2 and repeated until inequalities are satisfied or a maximum of $a\in\mathbb{R}^n$3 projection steps is reached (Ding et al., 2023).

Algorithmically, each environment step contains four stages: the actor proposes basic actions, the equality solver computes nonbasic actions, projection enforces inequalities, and the environment transition is stored in replay. Training is off-policy. For the DDPG-style actor update,

$a\in\mathbb{R}^n$4

The critic is updated by MSE with target

$a\in\mathbb{R}^n$5

where $a\in\mathbb{R}^n$6 includes both construction and projection (Ding et al., 2023).

The reported practical recipe uses actor/critic learning rates $a\in\mathbb{R}^n$7 and $a\in\mathbb{R}^n$8, discount $a\in\mathbb{R}^n$9, target smoothing $a=[a_b; a_n],$0, batch size $a=[a_b; a_n],$1, and projection step sizes $a=[a_b; a_n],$2 for Safe CartPole, $a=[a_b; a_n],$3 for Spring Pendulum, and $a=[a_b; a_n],$4 for OPF. Training typically uses $a=[a_b; a_n],$5 GRG updates, with larger $a=[a_b; a_n],$6 at evaluation (Ding et al., 2023).

5. Benchmarks, empirical behavior, and limitations

The method is evaluated on three benchmarks explicitly designed to stress hard constraints. Safe CartPole imposes the equality “zero net vertical force” and an inequality on bounded net horizontal force. Spring Pendulum imposes the equality $a=[a_b; a_n],$7 and a quadratic force-magnitude inequality. Optimal Power Flow with Battery Energy Storage imposes 28 nonlinear power-flow equalities and 58 inequality constraints in a 14-node system (Ding et al., 2023).

Across all three tasks, RPO-DDPG and RPO-SAC outperform prior constrained RL and penalty-only variants in both cumulative reward and constraint violation. On Safe CartPole, RPO achieves perfect episodic reward (200) with zero equality and inequality violations, whereas baselines such as CUP and Safety Layer have substantially lower rewards and non-zero equality violations. On Spring Pendulum, RPO-SAC reaches episodic rewards ≈182.4 with zero violations, outperforming SAC-L (≈149.7) and CPO (≈15.5). On OPF + Batteries, RPO achieves positive large cumulative rewards (≈42.1, ≈43.0) with near-zero violations, while safe RL baselines and penalty-only variants produce negative or small returns and fail to satisfy equalities (Ding et al., 2023).

The paper attributes this empirical behavior to the separation of roles within the architecture. Equalities are enforced by construction rather than penalized; inequalities are handled by the joint effect of exact penalties and projection; and the actor gradient respects the equality manifold through implicit differentiation. Sensitivity analysis indicates that small $a=[a_b; a_n],$8 suffices in training, larger $a=[a_b; a_n],$9 improves feasibility at evaluation, and adaptive penalty factors outperform fixed ones (Ding et al., 2023).

The limitations are equally explicit. The approach assumes local invertibility of $a_b\in\mathbb{R}^m$0; if this Jacobian is singular, $a_b\in\mathbb{R}^m$1 is not locally well-defined and the solver may diverge or stall. Solver design may require domain knowledge, ill-conditioned Jacobians can harm implicit differentiation, and iterative GRG projection adds inference and training overhead. Recommended mitigations include robust linear algebra such as LU or QR with pivoting, regularization, small $a_b\in\mathbb{R}^m$2, limited $a_b\in\mathbb{R}^m$3 during training, larger $a_b\in\mathbb{R}^m$4 for evaluation, and adaptive penalty factors to reduce violation before projection (Ding et al., 2023).

6. AdaSharp clarifications and relation to other RPO literatures

A recurrent misconception is that “RPO-AdaSharp” refers to a common optimizer family or a sharpness-aware training variant shared across RPO papers. The arXiv record represented here does not support that reading. In Reward Partitioning Optimization, the paper states that it “does not mention or use anything called ‘AdaSharp,’” that the actual optimizer is AdamW, and that no sharpness-aware training components such as SAM are used (Faye et al., 16 Jun 2025). In Relative Preference Optimization, the paper says there is no “RPO-AdaSharp” in the work and no SAM-like min-max objective or update rules are provided, although combining RPO with sharpness-aware minimization is mentioned only as a potential extension (Yin et al., 2024). In Read-only Prompt Optimization, a similar point appears: RPO is independent of AdaSharp, and any combination with sharpness-aware minimization is presented only as a conceptual future direction rather than part of the method (Lee et al., 2023).

The same pattern recurs elsewhere. Reflective Personalization Optimization explicitly says that the paper does not contain any mention or definition of “AdaSharp” (Hao et al., 7 Nov 2025). Reward-aware Preference Optimization states that there is no AdaSharp algorithm, objective, or empirical result in the work (Sun et al., 31 Jan 2025). Robust Preference Optimization presents RPO as an EM-based meta-framework for noisy preference data and gives an explicit recipe for building an “RPO-AdaSharp” variant only in the conditional sense that one would start from an existing AdaSharp loss and then apply the RPO robustification machinery; it does not identify AdaSharp as an existing component of the paper itself (Cao et al., 29 Sep 2025).

Taken together, this suggests a clear encyclopedic conclusion. RPO-AdaSharp is not a unified method family across arXiv. The strongest concrete referent in the supplied literature is the GRG-inspired Reduced Policy Optimization method for hard-constrained continuous control (Ding et al., 2023). In most other RPO papers, “AdaSharp” is either explicitly absent or mentioned only as a hypothetical extension rather than a defined algorithmic ingredient (Faye et al., 16 Jun 2025, Yin et al., 2024, Lee et al., 2023, Hao et al., 7 Nov 2025, Sun et al., 31 Jan 2025, Cao et al., 29 Sep 2025).