IP3O: Incrementally Penalized PPO

• The paper introduces IP3O, a constrained reinforcement learning algorithm that transforms cost signals into smooth incentives and penalties via a CELU-based mechanism.
• It integrates PPO-style trust-region updates with asymmetric clipping to balance reward maximization and constraint adherence effectively.
• Empirical results across safety-critical benchmarks demonstrate that IP3O improves safety compliance while maintaining competitive rewards.

Incrementally Penalized Proximal Policy Optimization (IP3O) is a constrained reinforcement learning algorithm that extends the proximal policy optimization lineage with an adaptive incentive mechanism and a progressively increasing penalty designed to stabilize training near constraint boundaries. It is formulated for settings in which a policy must maximize expected discounted reward while satisfying one or more cost budgets, and it combines PPO-style trust-region stabilization with a CELU-based transformation that behaves as an incentive within the safe region and as a penalty outside it (Hazra et al., 11 Sep 2025). Within the broader PPO family, the algorithm inherits the surrogate-objective, on-policy, minibatch-optimization pattern introduced by proximal policy optimization, whose defining appeal is to permit multiple epochs of minibatch updates while remaining simpler to implement than trust-region policy optimization (Schulman et al., 2017).

1. Lineage and formal problem formulation

IP3O is posed in the framework of a constrained Markov decision process (CMDP),

$$\langle \mathcal{S, A, R, C}, P, \rho, \gamma \rangle,$$

with state space $\mathcal{S}$, action space $\mathcal{A}$, reward function $\mathcal{R}$, cost function $\mathcal{C}$, transition kernel $P$, initial state distribution $\rho$, and discount factor $\gamma \in (0,1)$ (Hazra et al., 11 Sep 2025). The optimization target is to find a policy maximizing expected discounted reward subject to a cost budget:

$$\arg \max_{\pi} \mathcal{J}_{\mathcal{R}(\pi)} \quad \text{s.t. } \mathcal{J}_{\mathcal{C}(\pi)} \leq d.$$

For the multi-constraint case, the formulation is

$$\arg \max_{\pi} \mathcal{J}_{\mathcal{R}(\pi)} \quad \text{s.t. } \mathcal{J}_{\mathcal{C}_i}(\pi) \leq d_i.$$

The discounted reward and cost objectives are defined as

$\mathcal{S}$0

$\mathcal{S}$1

The paper also uses standard value, action-value, and advantage definitions for both reward and cost, together with the discounted state distribution

$\mathcal{S}$2

The connection to PPO is structural rather than incidental. PPO alternates between collecting data through interaction with the environment and optimizing a surrogate objective using stochastic gradient ascent, and its central design goal is to enable multiple epochs of minibatch updates while retaining conservative policy improvement behavior (Schulman et al., 2017). IP3O preserves that proximal backbone but makes the optimization explicitly constraint-aware.

2. Motivation: instability near constraint boundaries

The central motivation for IP3O is that standard PPO and several existing safe RL methods are not proactive about safety in constrained settings (Hazra et al., 11 Sep 2025). PPO stabilizes updates through clipping, but its objective is reward-only; it does not explicitly encourage the policy to remain within a safety region. In a CMDP, this can allow the policy to drift toward the feasible-set boundary, where small update noise may push it into violation.

The paper argues that standard penalty and barrier methods, including IPO and P3O, are often reactive rather than preventive. In that framing, a penalty is applied only after a constraint violation occurs. This produces abrupt transitions in the loss landscape, sharp gradients near the boundary, oscillatory updates, and suboptimal returns. The instability is especially acute when the policy is close to the threshold: a slight policy update may change the sign of the cost residual, causing the cost penalty to activate discontinuously or too sharply.

IP3O addresses this by converting the cost signal into an incentive inside the safe region and into a penalty outside it, using a smooth activation-like function. This design is intended to replace a hard boundary switch with a gradual transition. The paper’s claim is not merely that penalties should be stronger, but that the geometry of the cost transformation matters for stable learning. In this sense, IP3O differs from a purely reactive penalty method and also from ordinary PPO clipping: clipping controls the policy ratio, whereas IP3O additionally shapes the direction of policy improvement according to safety status (Hazra et al., 11 Sep 2025).

3. Objective construction and incremental penalty mechanism

The derivation begins from a performance-difference-based constrained update. Using the performance difference lemma, the paper writes the constrained objective in terms of reward advantage, with the constraint enforced through cost advantage:

$\mathcal{S}$3

subject to

$\mathcal{S}$4

A practical surrogate is then obtained using the importance sampling ratio

$\mathcal{S}$5

with expectations under $\mathcal{S}$6 and $\mathcal{S}$7. PPO-style clipping is retained to keep updates in a trust region (Hazra et al., 11 Sep 2025).

The paper introduces two distinct clipped ratios. The reward loss is

$\mathcal{S}$8

where

$\mathcal{S}$9

For each constraint, the cost loss is

$\mathcal{A}$0

where

$\mathcal{A}$1

This asymmetric clipping is one of the notable implementation choices in the method: $\mathcal{A}$2 is used for reward handling, whereas $\mathcal{A}$3 is used for cost handling.

The incremental penalty mechanism is built on ELU and then refined via CELU. The paper gives the ELU function as

$\mathcal{A}$4

but notes that ELU is not continuously differentiable at $\mathcal{A}$5 when $\mathcal{A}$6. It therefore uses the Continuously Differentiable ELU (CELU), written in the paper for $\mathcal{A}$7 as

$\mathcal{A}$8

The final objective is

$\mathcal{A}$9

where $\mathcal{R}$0 is the penalty factor. This construction gives the method its name. Inside the safe region, the transformed cost contributes a negative-valued smooth incentive; near the boundary, the transition remains smooth; outside the boundary, the penalty grows progressively rather than switching on abruptly (Hazra et al., 11 Sep 2025).

The paper explicitly contrasts this with Leaky ReLU, which can over-incentivize staying deep inside the safe region and thereby produce overly conservative policies. It also mentions a practical safeguard,

$\mathcal{R}$1

with $\mathcal{R}$2, to eliminate residual gradients if desired, while noting that this was not needed in practice.

4. Theoretical guarantees

Two theoretical results are emphasized. The first is an equivalence theorem. Under Slater’s condition and strong duality, if the penalty factor satisfies

$\mathcal{R}$3

where $\mathcal{R}$4 is the optimal Lagrange multiplier of the constrained problem, then the limit policy $\mathcal{R}$5 obtained by minimizing the IP3O objective is also a solution of the original constrained optimization problem (Hazra et al., 11 Sep 2025). The significance of this statement is that the transformed objective does not alter the optimal solution set provided the penalty scale is sufficiently large.

The second result is a worst-case error bound for the approximation induced by the current-policy surrogate, the CELU-based penalty approximation, and the residual gradient truncation term. The paper states that the bound depends on the KL divergence between current and updated policies,

$\mathcal{R}$6

the reward and cost advantage approximation magnitudes,

$\mathcal{R}$7

$\mathcal{R}$8

the discount factor $\mathcal{R}$9, the penalty scale $\mathcal{C}$0, and the residual truncation term $\mathcal{C}$1 (Hazra et al., 11 Sep 2025).

A plausible implication is that IP3O’s guarantees are coupled to the same trust-region intuition that motivates PPO: smaller policy shifts yield smaller approximation error. The paper makes this connection explicit by combining PPO-style clipping with a cost-shaping mechanism whose smoothness is controlled by $\mathcal{C}$2 and whose strength is controlled by $\mathcal{C}$3.

5. Experimental evaluation and empirical profile

The empirical study spans multiple constrained continuous-control and safety-oriented benchmarks (Hazra et al., 11 Sep 2025).

Benchmark family	Environments or tasks
MuJoCo Safety Velocity	Ant, Half-Cheetah, Humanoid, Swimmer
Safety Gymnasium	Goal tasks, Button tasks, with Point and Car agents
Bullet Safety Gymnasium	Ball and Car tasks
MetaDrive multi-agent scenarios	Intersection, Parking Lot

The comparison set includes first-order methods (CUP, FOCOPS), Lagrangian methods (CPPOPID, PPO + Lagrangian multiplier), second-order methods (CPO, PCPO), penalty methods (IPO, P3O), and vanilla PPO. For multi-agent experiments, the paper compares against MAPPO and MAP3O. All implementations were adapted from OmniSafe (Hazra et al., 11 Sep 2025).

The reported metrics are episodic return or cumulative reward, constraint violation or cost, and adherence to the cost limit $\mathcal{C}$4. The main findings are qualitative but consistent across suites. IP3O generally achieves better safety compliance than the baselines, usually attains competitive reward, and often achieves better reward when safety is respected. The paper also notes that methods such as FOCOPS and PCPO can achieve higher return in some cases but often fail to remain feasible. In Bullet Safety Gymnasium, IP3O achieves the best constraint compliance overall. In Safety Gymnasium, IP3O is often very safe, though with $\mathcal{C}$5 it can allow slightly more violations in exchange for higher reward. In MetaDrive, the multi-agent variant MAIP3O is competitive with MAPPO and MAP3O in collective reward while embedding safety awareness (Hazra et al., 11 Sep 2025).

These results are presented as evidence that the incentive-to-penalty transition is useful in practice, particularly in regimes where reactive penalties exhibit sharp gradient changes near the constraint boundary. The paper’s interpretation is therefore not simply that stronger regularization improves safety, but that smooth pre-violation shaping improves stability.

6. Ablations, implementation details, and limitations

The ablation studies focus on the CELU parameter $\mathcal{C}$6 and the cost budget $\mathcal{C}$7. On Half-Cheetah and Humanoid, $\mathcal{C}$8 gives the best balance of safety and reward, while $\mathcal{C}$9 yields higher reward but more constraint violations. Larger $P$0 increases safety pressure, though beyond a point the improvement saturates. On PointGoal1 and CarGoal1, IP3O adapts to varying cost budgets, learns feasible policies across different $P$1 values, and maintains reasonable reward while respecting the specified budget (Hazra et al., 11 Sep 2025). This suggests that $P$2 functions as a practical safety-performance knob rather than a purely numerical smoothing constant.

The implementation is on-policy. The pseudocode uses trajectory sampling, computation of reward and cost value functions, advantage estimation for both reward and cost, and PPO-style gradient updates with a trust-region criterion. Although the pseudocode includes a KL-based stopping criterion, the paper states that in practice PPO gradient clipping is used (Hazra et al., 11 Sep 2025).

Reported hyperparameters include a policy network of $P$3, a value network of $P$4, policy learning rate $P$5, value learning rate $P$6, batch size $P$7, Tanh activation, $P$8 epochs, $P$9 steps per epoch, discount $\rho$0, PPO clip factor $\rho$1, penalty factor $\rho$2, and GAE for advantage estimation with $\rho$3 and $\rho$4. Environment-specific $\rho$5 values are $\rho$6 for MuJoCo Safety Velocity, $\rho$7 for Safety Gymnasium, and $\rho$8 for Bullet Safety Gymnasium. Experiments were run on an Intel i7 CPU (12th gen), 16 GB RAM, and an NVIDIA GeForce RTX 3060 GPU (6 GB) (Hazra et al., 11 Sep 2025).

The limitations stated or implied by the paper are also specific. The method remains sensitive to $\rho$9 and the cost limit $\gamma \in (0,1)$0; excessively strong safety pressure can reduce reward and induce conservative behavior; the equivalence theorem assumes Slater’s condition and strong duality; and the worst-case bound contains approximation terms and KL deviations, so practical performance still depends on stable policy updates and accurate advantage estimates. The multi-agent extension is demonstrated, but scalability to larger systems is left as future work (Hazra et al., 11 Sep 2025).

In the context of PPO-based methods more broadly, IP3O is best characterized as a constrained, safety-aware surrogate optimization method rather than merely PPO with an added penalty. PPO clipping continues to limit large policy-ratio changes, but the defining novelty is the CELU-based transformation that changes the optimization landscape before, at, and after the constraint boundary. That distinction is the basis for both the paper’s stability argument and its empirical positioning relative to IPO-, P3O-, and Lagrangian-style baselines (Hazra et al., 11 Sep 2025).