MCPPO: Multi-Constraint PPO Methods

• MCPPO is a reinforcement learning framework that extends PPO by incorporating multiple constraints, ensuring safe and stable policy updates in complex settings.
• It employs exact penalty reformulations and surrogate decomposition—via component-wise or hierarchical clipping—to manage safety costs, sub-action updates, and objective ordering.
• Empirical results indicate that MCPPO enhances both constraint satisfaction and reward optimization, outperforming traditional methods in CMDPs, compound-action scenarios, and multi-objective tasks.

Searching arXiv for the cited papers and closely related MCPPO-style work. Multi-Constraint Proximal Policy Optimization (MCPPO) denotes a class of PPO-like policy-optimization methods designed for settings in which policy improvement is conditioned by multiple constraints, multiple objective components, or multiple action components. In the literature summarized here, the term is not introduced as a single canonical algorithm; rather, it is a useful umbrella for several closely related constructions. In constrained Markov decision processes (CMDPs), this perspective appears in Penalized Proximal Policy Optimization (P3O), which converts constrained policy iteration into a single minimization of an equivalent unconstrained problem while retaining PPO-style proximal updates (Zhang et al., 2022). In compound-action settings, analogous “multi-constraint” structure appears as component-wise clipping or mixed clipping over sub-actions rather than a single joint ratio (Song et al., 2023). In ordered multi-objective reinforcement learning, PPO is extended to topological Markov decision processes (TMDPs), where constraints are organized by a directed acyclic graph (DAG) and enforced through ancestor-based penalties during PPO updates (Wray et al., 2022).

1. Conceptual scope and relation to PPO

PPO is a state-of-the-art policy gradient algorithm that uses clipping to restrict policy updates. MCPPO-style methods preserve this proximal-update mechanism while altering the surrogate objective so that multiple restrictions can be handled within the same first-order optimization loop. The central motivation, across the papers considered here, is that standard PPO or standard constrained RL formulations become inefficient or unstable when the optimization target is governed by more than one limiting condition.

In safe RL, the relevant multi-constraint setting is the CMDP with several cost constraints. The P3O paper explicitly positions its method between classical constrained policy optimization methods such as CPO and FOCOPS and PPO-style trust-region approximation: instead of solving a quadratic approximation with a hard trust-region constraint, it uses an exact penalty reformulation for safety constraints and then uses PPO-style clipping to keep updates proximal (Zhang et al., 2022). This suggests that, in one important sense, MCPPO is not a departure from PPO but a reparameterization of what is clipped, penalized, or decomposed.

A second interpretation arises in compound-action spaces. Standard PPO implementations often treat the whole compound action as one joint action and compute a single likelihood ratio over the product of sub-action probabilities. The “joint action loss” paper argues that this can cause a sample to be discarded as soon as the joint ratio falls outside the clipping interval, even if some sub-actions are still informative and could be updated safely (Song et al., 2023). In MCPPO terminology, this corresponds not to multiple safety constraints but to multiple component-wise surrogate constraints.

A third interpretation appears in ordered multi-objective RL. The TMDP formulation generalizes CMDPs and lexicographic MDPs by encoding dependencies among objectives with a DAG and per-edge slack budgets. Topological Policy Optimization (TPO) then applies PPO objective-by-objective, using generalized Lagrangian advantages that penalize violations of higher-priority objectives (Wray et al., 2022). Here the “multi-constraint” structure is hierarchical rather than flat.

2. CMDP formulation and exact-penalty PPO

For CMDPs, the constrained policy improvement problem is written as

$$\pi_{k+1} = \arg\max_{\pi}\ \mathbb{E}_{s\sim d^{\pi},\, a\sim \pi}\!\left[ A^{\pi_k}_R(s,a)\right] \quad \text{s.t.}\quad J_{C_i}(\pi_{k})+ \frac{1}{1-\gamma}\mathbb{E}_{s\sim d^{\pi},\, a\sim \pi}\!\left[A_{C_i}^{\pi_k}(s,a)\right] \leq d_i,\ \forall i. \tag{1}$$

After introducing the parameterized policy $\pi(\theta)$ and importance ratio

$r(\theta)=\frac{\pi(\theta)}{\pi(\theta_k)},$

the optimization is rewritten as

$$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[-r(\theta)A_R^{\pi_k}(s,a)\right] \quad \text{s.t.}\quad \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i)\le 0,\ \forall i. \tag{2}$$

The exact-penalty unconstrained form is then

$$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[ -r(\theta)A_R^{\pi_k}(s,a) + \kappa \sum_i \max\left\{0,\, \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i) \right\} \right]. \tag{3}$$

Replacing the unknown future-state distribution $d^\pi$ by the on-policy distribution $d^{\pi_k}$ obtained from collected trajectories gives

$$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^{\pi_k},\ a\sim \pi_k} \left[ -r(\theta)A_R^{\pi_k}(s,a) + \kappa \sum_i \max\left\{0,\, \mathbb{E}_{s\sim d^{\pi_k},\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i) \right\} \right]. \tag{4}$$

Finally, PPO-style clipping is introduced for both reward and cost terms: $\mathcal{L}^{\mathrm{P3O}(\theta) = \mathcal{L}^{\mathrm{CLIP}_{R}(\theta)} + \kappa \sum_i \max\{0,\mathcal{L}^{\mathrm{CLIP}_{C_i}(\theta)}\}. \tag{5}$

The clipped reward surrogate is

$$\mathcal{L}^{\mathrm{CLIP}_{R}(\theta)} = \mathbb{E}_{s\sim d^{\pi_k},\, a\sim \pi_k} \left[ -\min\left\{ r(\theta)A_R^{\pi_k}(s,a),\, \mathrm{clip}(r(\theta),1-\epsilon,1+\epsilon)\,A_R^{\pi_k}(s,a) \right\} \right], \tag{6}$$

and each clipped constraint surrogate is

$\pi(\theta)$0

The method can therefore be summarized as constrained policy optimization $\pi(\theta)$1 exact penalty unconstrained objective $\pi(\theta)$2 PPO-clipped surrogate on both reward and cost terms. In MCPPO terms, this is the clearest instance of a multi-constraint PPO update based on explicit safety constraints rather than scalarized reward shaping (Zhang et al., 2022).

3. Multiple safety constraints and decentralized extensions

The multi-constraint mechanism in P3O is additive and direct: each cost constraint $\pi(\theta)$3 gets its own penalized clipped term, and the total penalty is

$\pi(\theta)$4

The paper explicitly states that extending to multiple constraints requires only “adding exact penalty functions with minor modification on the loss function.” There is no need for an inner-loop solution or solving a bigger quadratic program. The $\pi(\theta)$5 term acts as an exact penalty that activates only when a constraint surrogate is violated (Zhang et al., 2022).

This formulation is presented as a response to several difficulties in prior constrained RL. Existing primal methods like CPO or projection-based methods use local quadratic approximations and can have non-negligible approximation error; dual methods like PPO-Lagrangian are sensitive to multiplier tuning; and IPO-style barrier methods require feasible initialization and can be numerically unstable. The stated motivation is therefore not merely to add more penalties, but to obtain a single first-order objective that remains tractable when the number of constraints increases.

The same design is carried into the multi-agent extension. In a decentralized partially observed constrained MDP (DEC-POCMDP), the paper defines a joint reward and a shared global constraint, and then applies the same P3O loss to the multi-agent policy. It presents both a fully decentralized version and a CTDE-style centralized-critic version, with the same reward-surrogate-plus-penalized-cost-surrogate structure, now computed over multi-agent trajectories and recurrent policies or critics (Zhang et al., 2022).

A common misconception is that “multi-constraint PPO” necessarily implies a Lagrangian multiplier for each constraint. The P3O construction shows that this is not required. Its key distinction is the claim of an exact penalty reformulation with a finite penalty coefficient, so the unconstrained objective is theoretically equivalent to the constrained one when the penalty is large enough. This is a more specific statement than generic multi-penalty PPO formulations.

4. Theoretical guarantees and approximation structure

The principal theoretical guarantee in P3O is the exact penalty theorem. If $\pi(\theta)$6 is the Lagrange multiplier vector for the constrained optimum, then for any penalty coefficient satisfying

$\pi(\theta)$7

the constrained problem and the penalized problem share the same optimal solution set (Zhang et al., 2022). The paper emphasizes that this finite exactness is stronger than barrier methods or square penalties, because the penalty factor does not need to go to $\pi(\theta)$8.

The same work also analyzes the error introduced by replacing the true distribution $\pi(\theta)$9 with $r(\theta)=\frac{\pi(\theta)}{\pi(\theta_k)},$0 and by using the clipped surrogate approximation. The stated worst-case approximation error is

$r(\theta)=\frac{\pi(\theta)}{\pi(\theta_k)},$1

where $r(\theta)=\frac{\pi(\theta)}{\pi(\theta_k)},$2 is the number of constraints and

$r(\theta)=\frac{\pi(\theta)}{\pi(\theta_k)},$3

This links multi-constraint PPO directly to PPO’s proximality logic: if the KL divergence is small, the approximation error stays controlled. The paper also notes that too large a $r(\theta)=\frac{\pi(\theta)}{\pi(\theta_k)},$4 can hurt performance, since the approximation error scales with $r(\theta)=\frac{\pi(\theta)}{\pi(\theta_k)},$5.

An ordered analogue appears in TPO for TMDPs. There, the local constrained Bellman optimization uses a transformed violation function

$r(\theta)=\frac{\pi(\theta)}{\pi(\theta_k)},$6

which is zero when the constraint is satisfied and positive only when violated. The corresponding Lagrangian Bellman equation is

$r(\theta)=\frac{\pi(\theta)}{\pi(\theta_k)},$7

The paper then proves a policy gradient theorem of the form

$r(\theta)=\frac{\pi(\theta)}{\pi(\theta_k)},$8

where $r(\theta)=\frac{\pi(\theta)}{\pi(\theta_k)},$9 is the Lagrangian Q-function (Wray et al., 2022).

This suggests two distinct theoretical regimes for MCPPO-style methods. One regime seeks exact equivalence between constrained and penalized problems with finite $$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[-r(\theta)A_R^{\pi_k}(s,a)\right] \quad \text{s.t.}\quad \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i)\le 0,\ \forall i. \tag{2}$$0. The other uses structured Lagrangian penalties whose validity follows from a policy gradient theorem under fixed ancestor-based constraint terms.

5. Surrogate decomposition in compound-action spaces

A different MCPPO-style interpretation arises when the multiplicity is in the action itself rather than in externally specified safety costs. In compound-action environments, a single environment action is composed of several sub-actions. Standard PPO usually computes one joint likelihood ratio over the product of sub-action probabilities. Assuming sub-actions are independent, the compound action probability is

$$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[-r(\theta)A_R^{\pi_k}(s,a)\right] \quad \text{s.t.}\quad \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i)\le 0,\ \forall i. \tag{2}$$1

so the likelihood ratio becomes

$$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[-r(\theta)A_R^{\pi_k}(s,a)\right] \quad \text{s.t.}\quad \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i)\le 0,\ \forall i. \tag{2}$$2

The paper argues that this joint clipping is too strict in compound-action spaces: if the ratio of a sample exceeds the clipping range, the gradient the sample produces is zero, even when only one component is problematic (Song et al., 2023). It therefore proposes a per-component alternative with a ratio for each sub-action,

$$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[-r(\theta)A_R^{\pi_k}(s,a)\right] \quad \text{s.t.}\quad \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i)\le 0,\ \forall i. \tag{2}$$3

and a surrogate built by aggregating sub-action ratios or losses across components rather than taking one product ratio over the whole action.

It then introduces two mixed variants. The first is a mixed ratio,

$$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[-r(\theta)A_R^{\pi_k}(s,a)\right] \quad \text{s.t.}\quad \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i)\le 0,\ \forall i. \tag{2}$$4

followed by standard PPO clipping. The second combines the already-clipped losses: $$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[-r(\theta)A_R^{\pi_k}(s,a)\right] \quad \text{s.t.}\quad \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i)\le 0,\ \forall i. \tag{2}$$5

$$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[-r(\theta)A_R^{\pi_k}(s,a)\right] \quad \text{s.t.}\quad \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i)\le 0,\ \forall i. \tag{2}$$6

$$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[-r(\theta)A_R^{\pi_k}(s,a)\right] \quad \text{s.t.}\quad \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i)\le 0,\ \forall i. \tag{2}$$7

In the language of MCPPO, standard joint PPO clipping corresponds to a single constraint on the product-form joint policy ratio $$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[-r(\theta)A_R^{\pi_k}(s,a)\right] \quad \text{s.t.}\quad \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i)\le 0,\ \forall i. \tag{2}$$8, whereas sub-action loss corresponds to decomposing that single joint constraint into multiple component-wise surrogate terms $$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[-r(\theta)A_R^{\pi_k}(s,a)\right] \quad \text{s.t.}\quad \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i)\le 0,\ \forall i. \tag{2}$$9. The paper does not define formal separate KL constraints, but conceptually it addresses the same issue: how to safely update a policy with multiple action components without over-constraining the whole action when only one component is problematic (Song et al., 2023).

A common misunderstanding would be to equate this with CMDP-style safe RL. The paper itself does not introduce explicit constraint terms like separate KL constraints per component; instead, it changes the PPO surrogate itself. Accordingly, this line of work is better viewed as ratio-based surrogate decomposition for compound actions rather than a full constrained optimization framework.

6. Ordered multi-objective variants and empirical behavior

In TMDPs, the multi-constraint structure is encoded by a DAG over objectives. A TMDP is defined as

$$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[ -r(\theta)A_R^{\pi_k}(s,a) + \kappa \sum_i \max\left\{0,\, \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i) \right\} \right]. \tag{3}$$0

where $$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[ -r(\theta)A_R^{\pi_k}(s,a) + \kappa \sum_i \max\left\{0,\, \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i) \right\} \right]. \tag{3}$$1 is vector-valued and $$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[ -r(\theta)A_R^{\pi_k}(s,a) + \kappa \sum_i \max\left\{0,\, \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i) \right\} \right]. \tag{3}$$2 is a DAG over objectives $$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[ -r(\theta)A_R^{\pi_k}(s,a) + \kappa \sum_i \max\left\{0,\, \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i) \right\} \right]. \tag{3}$$3. For each objective $$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[ -r(\theta)A_R^{\pi_k}(s,a) + \kappa \sum_i \max\left\{0,\, \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i) \right\} \right]. \tag{3}$$4, the global constrained formulation is

$$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[ -r(\theta)A_R^{\pi_k}(s,a) + \kappa \sum_i \max\left\{0,\, \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i) \right\} \right]. \tag{3}$$5

The local action restriction form becomes

$$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[ -r(\theta)A_R^{\pi_k}(s,a) + \kappa \sum_i \max\left\{0,\, \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i) \right\} \right]. \tag{3}$$6

TPO then processes objectives in topological order, computes generalized Lagrangian advantages, and runs PPO using those advantages, with ancestor critics already learned and used inside the constraint penalty term (Wray et al., 2022).

Across the three papers, empirical results support different benefits of multi-constraint PPO-style design. On constrained locomotive tasks, P3O is reported to outperform CPO, PPO-Lagrangian, and FOCOPS with respect to both reward improvement and constraint satisfaction, including on multi-constraint Navigation and multi-agent Simple Spread (Zhang et al., 2022). On Navigation, when only one constraint is included in the loss, the corresponding cost is satisfied but the other is not; when both constraints are included, P3O satisfies both simultaneously at convergence. This directly illustrates the additive multi-constraint mechanism.

In compound-action experiments, the ranking in number of unclipped samples is reported as: compound action loss fewest unclipped samples, sub-action loss more unclipped samples, mix ratio loss in between, and mix loss most unclipped samples. In MuJoCo, mix ratio loss performs best overall in most tasks, while in Gym-$$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[ -r(\theta)A_R^{\pi_k}(s,a) + \kappa \sum_i \max\left\{0,\, \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i) \right\} \right]. \tag{3}$$7RTS sub-action loss works well, especially on the 10×10 and 16×16 maps (Song et al., 2023). The paper attributes this contrast to differences in sub-action correlation and to the fact that multiplying sub-action probabilities as if independent is not appropriate in Gym-$$\theta_{k+1} = \arg\min_{\theta} \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[ -r(\theta)A_R^{\pi_k}(s,a) + \kappa \sum_i \max\left\{0,\, \mathbb{E}_{s\sim d^\pi,\ a\sim \pi_k} \left[r(\theta)A_{C_i}^{\pi_k}(s,a)\right] +(1-\gamma)(J_{C_i}(\pi_k)-d_i) \right\} \right]. \tag{3}$$8RTS.

In the TMDP experiments, training for 300,000 iterations in a multi-objective robot navigation problem shows that increasing slack on constrained objectives allows the goal objective to improve, that the ordering in the DAG matters, and that fan and chain structures induce different tradeoff behavior. The paper also demonstrates the method on a real robot in a household, where changing slack values alters whether monitoring or avoidance dominates the resulting path (Wray et al., 2022).

These results do not establish a single universally optimal MCPPO formulation. Rather, they indicate that the term covers at least three technically distinct patterns: exact-penalty PPO for flat safety constraints, surrogate decomposition for compound-action clipping, and topologically ordered PPO for structured multi-objective constraints. The unifying principle is proximal policy optimization under multiple simultaneously active restrictions, but the semantics of those restrictions differ substantially across formulations.