LC-MAPPO: Constrained Multi-Agent PPO

• LC-MAPPO is a framework that augments Multi-Agent PPO with adaptive Lagrange multipliers to enforce explicit cost and safety constraints in cooperative settings.
• It integrates dual ascent with PPO’s clipped surrogate loss to manage various constraints such as safety, resource allocation, and terminal feasibility across diverse applications.
• Empirical studies reveal that LC-MAPPO variants achieve faster convergence, lower violation rates, and improved reward performance compared to unconstrained methods.

Searching arXiv for the cited LC-MAPPO and related constrained multi-agent PPO papers to ground the article. Lagrangian-Constrained Multi-Agent Proximal Policy Optimization (LC-MAPPO) denotes a class of constrained multi-agent reinforcement learning methods that augment Multi-Agent Proximal Policy Optimization with Lagrange multipliers so that policy optimization is carried out subject to explicit cost, safety, welfare, or terminal-feasibility constraints. In the literature covered here, the formulation appears in safe cooperative Markov games as MAPPO-Lagrangian (Gu et al., 2021), in adaptive traffic signal control as MAPPO-LCE, described as LC-MAPPO by Satheesh and Powell (2025) (Satheesh et al., 30 Mar 2025), in multi-agent data harvesting as LC-MAPPO with terminal penalties and optional adversarial smoothing (Wu et al., 22 Aug 2025), and in privacy-aware edge-device collaborative inference as HC-MAPPO-L (Wang et al., 23 Feb 2026). A related construction, TUC-PPO, applies the same primal-dual PPO pattern to a global team-utility constraint in spatial public goods games (Yang et al., 3 Jul 2025). Across these variants, the shared idea is to preserve PPO’s clipped surrogate core while introducing adaptive dual variables that penalize constraint violation during learning.

1. Conceptual scope and problem class

LC-MAPPO is defined on constrained multi-agent decision problems in which maximizing expected return alone is insufficient. The motivating settings in the cited works include safety-constrained Markov games (Gu et al., 2021), adaptive traffic signal control with GreenTime, GreenSkip, and PhaseSkip constraints (Satheesh et al., 30 Mar 2025), data harvesting with terminal requirements that all data must be collected and all agents must reach their destinations (Wu et al., 22 Aug 2025), and collaborative DNN inference under a long-term average delay constraint (Wang et al., 23 Feb 2026). TUC-PPO places the same template in spatial public goods games, where the constraint is that expected average team utility must exceed a threshold (Yang et al., 3 Jul 2025).

In these formulations, the joint policy is factorized across agents, but the optimization target is modified by one or more Lagrange multipliers. The constrained objective is converted into a saddle-point problem of the form

$$\max_{\theta}\min_{\lambda \ge 0}\; \mathcal{L}(\theta,\lambda),$$

or, equivalently in some expositions, a primal maximization with dual ascent on nonnegative multipliers. The specific constraint semantics differ by application: per-agent discounted costs in Safe Multi-Agent MuJoCo (Gu et al., 2021), network-level operational constraints in traffic control (Satheesh et al., 30 Mar 2025), a terminal vector of feasibility residuals in data harvesting (Wu et al., 22 Aug 2025), a single delay budget in collaborative inference (Wang et al., 23 Feb 2026), and a collective payoff threshold in public goods games (Yang et al., 3 Jul 2025).

A common misconception is that LC-MAPPO names a single standardized algorithm. The papers instead present a family of closely related primal-dual PPO constructions with distinct constraint structures, critic designs, and update rules. This suggests that “LC-MAPPO” functions primarily as a design pattern: MAPPO-style policy optimization plus adaptive Lagrangian enforcement of shared constraints.

2. Constrained optimization and Lagrangian relaxation

The canonical constrained multi-agent formulation in the safe MARL setting is a fully cooperative $N$-agent Markov game with shared reward $R$ and agent-specific instantaneous costs $C^i_j$, subject to discounted cost thresholds $c^i_j$ (Gu et al., 2021). The optimization problem is

$$\text{maximize}_{\theta}\;J(\pi_\theta) \qquad \text{subject to}\quad J^i_j(\pi_\theta)\le c^i_j,\ \forall i,j.$$

Its Lagrangian relaxation introduces multipliers $\lambda^i_j\ge 0$:

$$L(\theta,\lambda) = J(\pi_\theta) - \sum_{i=1}^N \sum_{j=1}^{m^i} \lambda^i_j \bigl(J^i_j(\pi_\theta)-c^i_j\bigr).$$

The same structure is specialized differently in later works. In adaptive traffic signal control, the constrained objective maximizes discounted reward subject to $K$ cost constraints,

$$C_i(\theta) = E_{\pi_\theta}\Bigl[\sum_{t=0}^\infty \gamma^t\, c_i(s_t,a_t)\Bigr]\le d_i,\quad i=1,\dots,K,$$

with three instantiated costs: GreenTime, PhaseSkip, and GreenSkip (Satheesh et al., 30 Mar 2025). In privacy-aware collaborative inference, the constrained Markov Decision Process uses a single long-term average delay constraint,

$N$0

and a scalar multiplier $N$1 (Wang et al., 23 Feb 2026). In data harvesting, the constraint is terminal rather than cumulative: the final state $N$2 induces

$N$3

where $N$4 is the maximum time for any agent to reach its destination and $N$5 is the remaining uncollected data; the objective is

$N$6

(Wu et al., 22 Aug 2025). TUC-PPO replaces safety or feasibility costs with a team-welfare inequality,

$N$7

and introduces a single dual variable $N$8 (Yang et al., 3 Jul 2025).

These formulations differ in whether costs are per-agent or global, discounted or average, stepwise or terminal, and one-sided or vector-valued. A plausible implication is that LC-MAPPO is particularly flexible when the operational requirement can be written as an expectation constraint compatible with dual ascent.

3. PPO integration and primal-dual updates

The defining algorithmic move in LC-MAPPO is to retain PPO’s clipped surrogate while modifying the advantage or reward signal by a Lagrangian penalty. Standard PPO uses

$N$9

with a critic loss and an entropy bonus (Wu et al., 22 Aug 2025). LC-MAPPO variants incorporate constraint information in one of three closely related ways.

In MAPPO-Lagrangian for safe MARL, each agent forms a Lagrangian advantage

$R$0

then optimizes the clipped objective

$R$1

(Gu et al., 2021). In MAPPO-LCE for traffic signal control, the weighted sum of cost advantages is subtracted inside the clipped surrogate,

$R$2

(Satheesh et al., 30 Mar 2025). In HC-MAPPO-L, only the user policy layer is Lagrangian-enhanced:

$R$3

(Wang et al., 23 Feb 2026).

The data-harvesting variant embeds the penalty directly in the terminal reward,

$R$4

so that the return and advantages automatically propagate the terminal feasibility penalty backward through time (Wu et al., 22 Aug 2025). TUC-PPO instead uses a one-sided mini-batch penalty,

$R$5

and adds it to the PPO surrogate with value and entropy regularization (Yang et al., 3 Jul 2025).

Dual updates are projected onto the nonnegative orthant. Representative update rules include

$R$6

for data harvesting (Wu et al., 22 Aug 2025),

$R$7

for traffic control (Satheesh et al., 30 Mar 2025), and

$R$8

for TUC-PPO (Yang et al., 3 Jul 2025). The common interpretation is that the multiplier rises when the observed batch or trajectory violates the target constraint and stays at zero otherwise.

4. Critics, decentralization, and architectural variants

Most LC-MAPPO implementations use centralized-training, decentralized-execution (CTDE), but the manner of centralization differs. In the data-harvesting formulation, the critic observes the full joint state

$R$9

making the method explicitly CTDE: the centralized critic computes value estimates using global state, whereas each actor acts from its local observation at execution time (Wu et al., 22 Aug 2025). The actor can be parameter-shared for homogeneous agents or can use separate actor heads with a common backbone.

HC-MAPPO-L also uses CTDE, but factorizes the policy into three hierarchically structured layers: an auto-regressive GRU deployment policy, a Lagrangian-enhanced association and partitioning policy, and an attention-based resource allocation policy (Wang et al., 23 Feb 2026). Only the user layer incorporates the Lagrangian penalty; deployment and allocation use unconstrained PPO losses. This selective insertion of the dual term is a distinctive architectural choice.

MAPPO-LCE in traffic control includes a learned Lagrange Cost Estimator, $C^i_j$0, trained by regression,

$C^i_j$1

to predict instantaneous costs and provide more stable cost estimates for both cost advantages and multiplier updates (Satheesh et al., 30 Mar 2025). The paper explicitly attributes this design to variance reduction in cost estimation.

TUC-PPO departs from generic LC-MAPPO by fully factorizing the policy and dispensing with a mixing network, because “agents’ objectives share the same team penalty” (Yang et al., 3 Jul 2025). The authors contrast this with a generic LC-MAPPO that “typically requires a centralized critic or mixing network” and “may maintain multiple dual variables for per-agent or per-constraint.” This comparison makes clear that the degree of centralization in constrained multi-agent PPO is a design variable rather than a fixed property.

5. Representative instantiations and empirical behavior

The cited papers evaluate LC-MAPPO-type methods in substantively different environments, but several empirical regularities recur: faster convergence to feasible policies than unconstrained PPO, lower violation rates than unconstrained baselines, and sensitivity to multiplier adaptation.

Setting	LC-MAPPO-style method	Reported empirical outcome
Safe Multi-Agent MuJoCo	MAPPO-Lagrangian	Reduced cost almost to zero within a few thousand timesteps and kept it there; matched or exceeded IPPO/MAPPO reward in many tasks (Gu et al., 2021)
Adaptive traffic signal control	MAPPO-LCE / LC-MAPPO	Outperformed MAPPO by 12.60%, IPPO by 10.29%, and QTRAN by 13.10% across all environments and traffic constraints (Satheesh et al., 30 Mar 2025)
Data harvesting	LC-MAPPO	Converged in mean $C^i_j$2 time units versus $C^i_j$3 with no penalty and $C^i_j$4 with a large fixed penalty (Wu et al., 22 Aug 2025)
Spatial public goods games	TUC-PPO	Achieved full cooperation at $C^i_j$5, versus $C^i_j$6 for PPO, and reached 100% cooperation at $C^i_j$7 in $C^i_j$8 iterations (Yang et al., 3 Jul 2025)
Privacy-aware collaborative inference	HC-MAPPO-L	Maintained average delay $C^i_j$9 while unconstrained baselines were reported at $c^i_j$0–$c^i_j$1; user cost and delay converged within $c^i_j$2 iterations (Wang et al., 23 Feb 2026)

In the data-harvesting study, two simulation scenarios were tested, each with 3 agents and continuous control. Baselines included A* search on a discretized joint-action graph, DDQN with a 5-way discrete action for each agent, PPO with no terminal penalty, PPO with a fixed large penalty, and LC-MAPPO. The paper reports that continuous-action PPO found more efficient trajectories than DDQN’s coarse grid, and that under random or adversarial state noise with $c^i_j$3, the smoothed LC-MAPPO policy sustained approximately $c^i_j$4 time units versus approximately $c^i_j$5 for the non-smoothed version (Wu et al., 22 Aug 2025).

In traffic control, the constraints are explicitly operational rather than abstract. GreenTime penalizes lights that remain green longer than $c^i_j$6, PhaseSkip penalizes phases skipped more than $c^i_j$7 times in a row, and GreenSkip penalizes lights denied green more than $c^i_j$8 consecutive phase-changes. Satheesh and Powell report experiments on three real-world CityFlow datasets—Hangzhou, Jinan, and New York—and describe lower constraint-violation rates together with improved reward, average delay, and throughput relative to IPPO, MAPPO, and QTRAN (Satheesh et al., 30 Mar 2025).

In TUC-PPO, the relevant outcome is collective cooperation rather than safety or feasibility. The method is reported as robust to half/half, Bernoulli 0.5, and all-defector initializations under the stated enhancement factors, while sensitivity to the entropy coefficient $c^i_j$9 is described as pronounced: values greater than $$\text{maximize}_{\theta}\;J(\pi_\theta) \qquad \text{subject to}\quad J^i_j(\pi_\theta)\le c^i_j,\ \forall i,j.$$0 inject excessive noise and require higher $$\text{maximize}_{\theta}\;J(\pi_\theta) \qquad \text{subject to}\quad J^i_j(\pi_\theta)\le c^i_j,\ \forall i,j.$$1 for cooperation (Yang et al., 3 Jul 2025).

6. Theoretical properties, robustness mechanisms, and recurring limitations

The strongest formal guarantees among the cited works appear in the safe MARL treatment of constrained policy optimization. For MACPO, the paper states monotonic improvement,

$$\text{maximize}_{\theta}\;J(\pi_\theta) \qquad \text{subject to}\quad J^i_j(\pi_\theta)\le c^i_j,\ \forall i,j.$$2

and safety at every iteration,

$$\text{maximize}_{\theta}\;J(\pi_\theta) \qquad \text{subject to}\quad J^i_j(\pi_\theta)\le c^i_j,\ \forall i,j.$$3

under the exact trust-region constrained policy optimization procedure (Gu et al., 2021). The same paper describes MAPPO-Lagrangian as a PPO-based practical approximation that, in experiments, “strictly enforces near-zero cost” and “recovers the guarantees asymptotically.” This is not a finite-sample proof for PPO clipping; it is an empirical and asymptotic statement.

Other works make more limited theoretical remarks. TUC-PPO states that PPO’s clipping ensures policy-update stability, that dual ascent follows standard convex-concave saddle-point theory, and that the one-sided ReLU penalty prevents over-enforcing the constraint, keeping updates well-conditioned (Yang et al., 3 Jul 2025). MAPPO-LCE states that under standard assumptions—bounded advantages, small clips, and Lipschitz continuity—the primal-dual PPO updates converge to a local saddle point, and that the learned cost estimator accelerates multiplier adaptation by reducing variance in cost estimates (Satheesh et al., 30 Mar 2025). HC-MAPPO-L describes the dual update as ensuring that $$\text{maximize}_{\theta}\;J(\pi_\theta) \qquad \text{subject to}\quad J^i_j(\pi_\theta)\le c^i_j,\ \forall i,j.$$4 grows when average delay exceeds the budget, thereby penalizing delay in the policy loss (Wang et al., 23 Feb 2026).

Robustness mechanisms also vary. The data-harvesting paper introduces an adversarial state-smoothing regularizer, denoted SR$$\text{maximize}_{\theta}\;J(\pi_\theta) \qquad \text{subject to}\quad J^i_j(\pi_\theta)\le c^i_j,\ \forall i,j.$$5L, in which a small adversary network $$\text{maximize}_{\theta}\;J(\pi_\theta) \qquad \text{subject to}\quad J^i_j(\pi_\theta)\le c^i_j,\ \forall i,j.$$6 perturbs the state within an $$\text{maximize}_{\theta}\;J(\pi_\theta) \qquad \text{subject to}\quad J^i_j(\pi_\theta)\le c^i_j,\ \forall i,j.$$7-ball and maximizes the KL divergence between $$\text{maximize}_{\theta}\;J(\pi_\theta) \qquad \text{subject to}\quad J^i_j(\pi_\theta)\le c^i_j,\ \forall i,j.$$8 and $$\text{maximize}_{\theta}\;J(\pi_\theta) \qquad \text{subject to}\quad J^i_j(\pi_\theta)\le c^i_j,\ \forall i,j.$$9. The regularizer

$\lambda^i_j\ge 0$0

is then added to the actor loss; in practice the paper uses $\lambda^i_j\ge 0$1 in $\lambda^i_j\ge 0$2-norm (Wu et al., 22 Aug 2025). This is a robustness device rather than a constraint-enforcement mechanism in the dual sense.

A recurring limitation is the need to tune dual learning rates and penalty scales. The cited works explicitly specify such hyperparameters—$\lambda^i_j\ge 0$3 in data harvesting, $\lambda^i_j\ge 0$4 in TUC-PPO, $\lambda^i_j\ge 0$5 in MAPPO-Lagrangian, and small $\lambda^i_j\ge 0$6 with optional clipping of $\lambda^i_j\ge 0$7 in HC-MAPPO-L (Wu et al., 22 Aug 2025, Yang et al., 3 Jul 2025, Gu et al., 2021, Wang et al., 23 Feb 2026). This suggests that feasibility, stability, and reward quality depend materially on multiplier dynamics. Another misconception is that fixed large penalties are equivalent to adaptive Lagrangian updates; the data-harvesting results directly contradict this by reporting convergence above $\lambda^i_j\ge 0$8 time units for PPO with a large fixed penalty, versus $\lambda^i_j\ge 0$9 for adaptive LC-MAPPO (Wu et al., 22 Aug 2025).

7. Relation to adjacent methods and research directions

LC-MAPPO sits at the intersection of safe reinforcement learning, constrained optimization, and cooperative MARL. The safe MARL paper positions MAPPO-Lagrangian alongside MACPO, explicitly combining ideas from constrained policy optimization and multi-agent trust region learning (Gu et al., 2021). MAPPO-LCE adds an auxiliary learned cost estimator to stabilize dual updates in a real-world control domain (Satheesh et al., 30 Mar 2025). HC-MAPPO-L shows that the same primal-dual idea can be embedded selectively within a hierarchical controller rather than uniformly across all policy components (Wang et al., 23 Feb 2026). TUC-PPO demonstrates that the “constraint” need not be a safety cost; it can also be a collective welfare threshold in a social-dilemma game (Yang et al., 3 Jul 2025).

From these examples, a coherent taxonomy emerges. One branch uses per-agent or per-constraint multipliers for safety costs in continuous-control Markov games (Gu et al., 2021). A second uses multiple operational constraints with cost critics or estimators in networked control (Satheesh et al., 30 Mar 2025). A third imposes terminal feasibility conditions by modifying only the terminal reward and then propagating that penalty through GAE and return computation (Wu et al., 22 Aug 2025). A fourth places a single global multiplier on a long-term average resource or latency budget in hierarchical systems (Wang et al., 23 Feb 2026). A fifth uses one-sided welfare penalties in fully decentralized population games (Yang et al., 3 Jul 2025).

This suggests that the central unresolved issue is not whether PPO can be Lagrangian-constrained, but how the constraint signal should be represented: as terminal penalties, cumulative costs, estimated costs, cost advantages, or team-utility thresholds; with centralized critics, factorized actors, or hierarchical layers; and with single or multiple dual variables. The literature summarized here indicates that these choices are application-defining rather than incidental, and that LC-MAPPO is best understood as a configurable constrained PPO framework for multi-agent settings rather than a single canonical algorithm.