Multi-Agent Guided Policy Optimization

• Multi-Agent Guided Policy Optimization (MAGPO) is a framework that couples a centralized auto-regressive guider with decentralized learner policies to fully utilize centralized information in CTDE settings.
• It employs mechanisms like KL regularization, synchronized guidance, and localized value decompositions to align policy updates and enhance coordination across agents.
• Empirical evaluations show MAGPO outperforms conventional CTDE baselines on numerous benchmarks, highlighting its ability to improve scalability and overall performance in multi-agent tasks.

Searching arXiv for papers on Multi-Agent Guided Policy Optimization and closely related multi-agent PPO theory. Multi-Agent Guided Policy Optimization (MAGPO) denotes a family of cooperative multi-agent reinforcement learning methods in which decentralized policies are improved using explicit guidance derived from centralized structure, localized multi-agent value information, or other auxiliary signals that shape exploration and policy updates. In the strict sense, the term is used by the 2025 framework “Multi-Agent Guided Policy Optimization” (Li et al., 24 Jul 2025), which couples a centralized auto-regressive guider policy with decentralized learner policies under Centralized Training with Decentralized Execution (CTDE). In a broader methodological sense, the term also describes earlier and adjacent lines of work in which per-agent updates are guided by localized multi-agent $Q$-functions, synchronized marginal advantages, coordinated trust-region surrogates, or critic-free group-relative signals (Zhao et al., 2023, Wan et al., 2020, Wu et al., 2021, Chen et al., 3 Jun 2025). Across these formulations, the recurring objective is to use centralized information more fully than standard CTDE baselines while preserving deployable decentralized execution, improving coordination, and, in some cases, obtaining monotonic improvement or global-optimality guarantees (Li et al., 24 Jul 2025, Zhao et al., 2023).

1. Conceptual scope and problem setting

MAGPO arises in cooperative multi-agent reinforcement learning under partial observability, limited communication, and shared team rewards. The 2025 MAGPO framework formalizes this setting as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) with tuple

$$\langle \mathcal{N}, \mathcal{S}, \mathcal{A}, r, \mathcal{P}, \mathcal{O}, \mathcal{Z}, \gamma \rangle,$$

where agents act from local observations or histories, while training may exploit centralized state information (Li et al., 24 Jul 2025). The target object is a decentralized joint policy $\boldsymbol{\pi}=\{\pi_i\}_{i=1}^n$ maximizing

$$V_\rho(\boldsymbol{\pi}) \triangleq \mathbb{E}_{s_0\sim \rho}\Big[\, \mathbb{E}\big[\sum_{t=0}^\infty \gamma^t r_t \,\big|\, s_0 \big] \Big].$$

The central motivation is that prevailing CTDE methods often use centralized information only in critics or value functions, leaving the policy itself factorized and thereby underusing centralized training signal (Li et al., 24 Jul 2025). This critique also appears in different form in related work. “Local Optimization Achieves Global Optimality in Multi-Agent Reinforcement Learning” studies fully cooperative Markov games and shows that localized multi-agent $Q$-functions can furnish per-agent descent directions aligned with global improvement (Zhao et al., 2023). “Coordinated Proximal Policy Optimization” derives a joint multi-agent PPO-style objective in which cross-agent ratio products coordinate step sizes and induce dynamic credit assignment (Wu et al., 2021). “Multi-agent Policy Optimization with Approximatively Synchronous Advantage Estimation” addresses asynchronous bias by marginalizing over partners’ policies and approximately synchronizing advantage estimation across agents (Wan et al., 2020). “Heterogeneous Group-Based Reinforcement Learning for LLM-based Multi-Agent Systems” replaces critic-based guidance with group-relative reward normalization over structured multi-agent rollouts (Chen et al., 3 Jun 2025).

Within this landscape, MAGPO may therefore refer either to a named framework (Li et al., 24 Jul 2025) or to a broader design pattern: decentralized policies are not optimized independently, but are guided by centralized policies, centralized critics, localized value decompositions, synchronized counterfactuals, or structured multi-agent rollout comparisons. This suggests that the term sits at the intersection of CTDE, trust-region policy optimization, sequential joint-policy factorization, and multi-agent credit assignment.

2. Canonical MAGPO framework: centralized guider and decentralized learners

The named MAGPO framework introduces two coupled policy classes: a centralized guider policy and decentralized learner policies (Li et al., 24 Jul 2025). The guider is an auto-regressive joint policy

$$\boldsymbol{\mu}(\mathbf{a}\mid s) = \mu^{i_1}(a^{i_1}\mid s)\, \mu^{i_2}(a^{i_2}\mid s,a^{i_1})\dots \mu^{i_n}(a^{i_n}\mid s,\mathbf{a}^{i_{1:n-1}}),$$

defined over an ordering of agents. The learner is factorized and decentralized,

$$\boldsymbol{\pi}(\mathbf{a}\mid s) = \prod_{j=1}^n \pi^{i_j}(a^{i_j}\mid s)$$

in the theory section, and $\pi^{i_j}(a^{i_j}\mid o^{i_j})$ in the partial-observability setting used in practice (Li et al., 24 Jul 2025).

This architecture is designed to resolve two asymmetries identified in teacher–student CTDE systems. The first is observation asymmetry: a teacher may rely on privileged centralized information unavailable to decentralized students. The second is policy asymmetry: a centralized teacher’s joint policy may exploit correlated strategies that cannot be represented by any product of local policies, even absent observation asymmetry (Li et al., 24 Jul 2025). MAGPO addresses both by never allowing the guider to drift far from the learner: guider updates are regularized toward the learner, the learner is aligned to the guider through KL projection, and then the guider is backtracked to the learner after every iteration (Li et al., 24 Jul 2025).

The high-level iteration has four stages (Li et al., 24 Jul 2025):

1. Data collection by rolling out the current guider.
2. Guider training by RL, using a PPO-style or Policy Mirror Descent (PMD) update subject to alignment with the learner.
3. Learner training by KL imitation of the updated guider, plus an optional auxiliary RL objective.
4. Guider backtracking, setting the next guider equal to the updated learner.

This coupling is the defining structural feature of the named MAGPO framework. Centralized exploration is delegated to a policy class capable of sequential coordination; decentralized deployability is preserved because every centralized improvement is projected back into the factorized learner class before the next iteration (Li et al., 24 Jul 2025).

3. Optimization objectives and training dynamics

In the tabular full-observability theory, the guider update is written as a PMD step: $$\hat{\boldsymbol{\mu}}_k = \arg\max_{\boldsymbol{\mu}} \Big\{ \eta_k \langle Q^{\boldsymbol{\mu}_k}(s,\cdot), \boldsymbol{\mu}(\cdot\mid s) \rangle - \mathrm{D}_{\mathrm{KL}}\big( \boldsymbol{\mu}(\cdot\mid s), \boldsymbol{\mu}_k(\cdot\mid s) \big) \Big\},$$ with closed form

$$\hat{\boldsymbol{\mu}}_k(\mathbf{a}\mid s) = \boldsymbol{\mu}_k(\mathbf{a}\mid s)\, \frac{\exp\left(\eta_k Q^{\boldsymbol{\mu}_k}(s,\mathbf{a})\right)}{z_k(s)}.$$

Because of guider backtracking, $$\langle \mathcal{N}, \mathcal{S}, \mathcal{A}, r, \mathcal{P}, \mathcal{O}, \mathcal{Z}, \gamma \rangle,$$0 at the start of each iteration (Li et al., 24 Jul 2025).

The learner projection is then

$$\langle \mathcal{N}, \mathcal{S}, \mathcal{A}, r, \mathcal{P}, \mathcal{O}, \mathcal{Z}, \gamma \rangle,$$1

which enforces that the decentralized learner tracks the centralized update as closely as possible within the decentralized policy class (Li et al., 24 Jul 2025).

In practice, MAGPO replaces exact PMD with a PPO-style guider objective incorporating a KL alignment term to the learner. The guider loss is

$$\langle \mathcal{N}, \mathcal{S}, \mathcal{A}, r, \mathcal{P}, \mathcal{O}, \mathcal{Z}, \gamma \rangle,$$2

where the “double clipping” mechanism bounds drift relative to both the previous guider and the learner, and $$\langle \mathcal{N}, \mathcal{S}, \mathcal{A}, r, \mathcal{P}, \mathcal{O}, \mathcal{Z}, \gamma \rangle,$$3 activates KL alignment when guider–learner ratios exit $$\langle \mathcal{N}, \mathcal{S}, \mathcal{A}, r, \mathcal{P}, \mathcal{O}, \mathcal{Z}, \gamma \rangle,$$4 (Li et al., 24 Jul 2025). The learner loss is

$$\langle \mathcal{N}, \mathcal{S}, \mathcal{A}, r, \mathcal{P}, \mathcal{O}, \mathcal{Z}, \gamma \rangle,$$5

The first term is KL projection from learner to guider; the second is an auxiliary RL term (Li et al., 24 Jul 2025).

The ratio bound $$\langle \mathcal{N}, \mathcal{S}, \mathcal{A}, r, \mathcal{P}, \mathcal{O}, \mathcal{Z}, \gamma \rangle,$$6 is the principal control parameter mediating centralized expressiveness versus decentralizability (Li et al., 24 Jul 2025). Smaller $$\langle \mathcal{N}, \mathcal{S}, \mathcal{A}, r, \mathcal{P}, \mathcal{O}, \mathcal{Z}, \gamma \rangle,$$7 forces tighter guider–learner coupling and is advantageous when centralized policies tend to exploit non-decentralizable correlations; larger $$\langle \mathcal{N}, \mathcal{S}, \mathcal{A}, r, \mathcal{P}, \mathcal{O}, \mathcal{Z}, \gamma \rangle,$$8 grants the guider greater freedom when the task admits a decentralized realization of coordinated behavior (Li et al., 24 Jul 2025). This suggests that MAGPO is not merely a distillation procedure but a constrained bilevel optimization scheme over two policy classes.

4. Theoretical foundations and related guarantee regimes

The named MAGPO framework proves monotonic policy improvement in cooperative Markov games with full observability and tabular policies (Li et al., 24 Jul 2025). The main theorem states that for the idealized MAGPO procedure with exact PMD update and exact KL projection,

$$\langle \mathcal{N}, \mathcal{S}, \mathcal{A}, r, \mathcal{P}, \mathcal{O}, \mathcal{Z}, \gamma \rangle,$$9

for all $\boldsymbol{\pi}=\{\pi_i\}_{i=1}^n$0 (Li et al., 24 Jul 2025). The proof combines the closed-form PMD guider update, KL projection optimality, and the performance-difference lemma. In expanded form, the argument yields

$\boldsymbol{\pi}=\{\pi_i\}_{i=1}^n$1

(Li et al., 24 Jul 2025).

A second theoretical pillar comes from the localized-value line of work. In “Local Optimization Achieves Global Optimality in Multi-Agent Reinforcement Learning,” joint policy is factorized sequentially,

$\boldsymbol{\pi}=\{\pi_i\}_{i=1}^n$2

and the multi-agent performance difference lemma decomposes global suboptimality into agent-wise inner products involving localized $\boldsymbol{\pi}=\{\pi_i\}_{i=1}^n$3-functions $\boldsymbol{\pi}=\{\pi_i\}_{i=1}^n$4 (Zhao et al., 2023). For any two joint policies $\boldsymbol{\pi}=\{\pi_i\}_{i=1}^n$5 and $\boldsymbol{\pi}=\{\pi_i\}_{i=1}^n$6,

$\boldsymbol{\pi}=\{\pi_i\}_{i=1}^n$7

This result underwrites a multi-agent PPO algorithm whose local policies are updated similarly to vanilla PPO and which converges to the globally optimal policy at a sublinear rate under standard regularity conditions (Zhao et al., 2023). In the idealized setting, the convergence rate simplifies to

$\boldsymbol{\pi}=\{\pi_i\}_{i=1}^n$8

(Zhao et al., 2023). This provides a distinct guarantee regime from MAGPO proper: not monotonic improvement under centralized guider projection, but global convergence via sequential localized mirror descent (Zhao et al., 2023).

A third guarantee regime appears in “Coordinated Proximal Policy Optimization,” which derives a joint surrogate objective over the product of agent importance ratios and proves monotonicity of policy improvement when optimizing a theoretically grounded joint objective (Wu et al., 2021). The practical CoPPO objective for agent $\boldsymbol{\pi}=\{\pi_i\}_{i=1}^n$9 is

$$V_\rho(\boldsymbol{\pi}) \triangleq \mathbb{E}_{s_0\sim \rho}\Big[\, \mathbb{E}\big[\sum_{t=0}^\infty \gamma^t r_t \,\big|\, s_0 \big] \Big].$$0

with inner clipping

$$V_\rho(\boldsymbol{\pi}) \triangleq \mathbb{E}_{s_0\sim \rho}\Big[\, \mathbb{E}\big[\sum_{t=0}^\infty \gamma^t r_t \,\big|\, s_0 \big] \Big].$$1

(Wu et al., 2021). This objective dynamically rescales each agent’s effective advantage by the evolving policies of the others, thereby achieving coordinated step-size adaptation and dynamic credit assignment (Wu et al., 2021).

These lines of theory are not identical, but they collectively establish that MAGPO-style methods are not merely heuristic variants of MAPPO. Depending on formulation, they can admit monotonic improvement guarantees (Li et al., 24 Jul 2025), global optimality results for localized sequential updates (Zhao et al., 2023), or trust-region lower bounds for coordinated PPO objectives (Wu et al., 2021).

5. Guidance mechanisms beyond the canonical framework

Although the named MAGPO framework centers on guider–learner coupling (Li et al., 24 Jul 2025), the broader literature exhibits several distinct guidance mechanisms.

One mechanism is localized value guidance. The localized $$V_\rho(\boldsymbol{\pi}) \triangleq \mathbb{E}_{s_0\sim \rho}\Big[\, \mathbb{E}\big[\sum_{t=0}^\infty \gamma^t r_t \,\big|\, s_0 \big] \Big].$$2-functions $$V_\rho(\boldsymbol{\pi}) \triangleq \mathbb{E}_{s_0\sim \rho}\Big[\, \mathbb{E}\big[\sum_{t=0}^\infty \gamma^t r_t \,\big|\, s_0 \big] \Big].$$3 of (Zhao et al., 2023) supply descent directions that are aligned with global improvement under sequential conditional factorization. The associated advantage decomposition,

$$V_\rho(\boldsymbol{\pi}) \triangleq \mathbb{E}_{s_0\sim \rho}\Big[\, \mathbb{E}\big[\sum_{t=0}^\infty \gamma^t r_t \,\big|\, s_0 \big] \Big].$$4

enables per-agent PPO-style updates that are theoretically tied to joint optimality (Zhao et al., 2023).

A second mechanism is synchronized marginal-advantage guidance. “Multi-agent Policy Optimization with Approximatively Synchronous Advantage Estimation” defines a marginal $$V_\rho(\boldsymbol{\pi}) \triangleq \mathbb{E}_{s_0\sim \rho}\Big[\, \mathbb{E}\big[\sum_{t=0}^\infty \gamma^t r_t \,\big|\, s_0 \big] \Big].$$5-value

$$V_\rho(\boldsymbol{\pi}) \triangleq \mathbb{E}_{s_0\sim \rho}\Big[\, \mathbb{E}\big[\sum_{t=0}^\infty \gamma^t r_t \,\big|\, s_0 \big] \Big].$$6

and a marginal advantage

$$V_\rho(\boldsymbol{\pi}) \triangleq \mathbb{E}_{s_0\sim \rho}\Big[\, \mathbb{E}\big[\sum_{t=0}^\infty \gamma^t r_t \,\big|\, s_0 \big] \Big].$$7

approximated by Monte Carlo reorganization of joint-action samples (Wan et al., 2020). The resulting PPO-style subproblem

$$V_\rho(\boldsymbol{\pi}) \triangleq \mathbb{E}_{s_0\sim \rho}\Big[\, \mathbb{E}\big[\sum_{t=0}^\infty \gamma^t r_t \,\big|\, s_0 \big] \Big].$$8

decomposes multi-agent policy optimization into single-agent trust-region updates under an approximate synchrony hypothesis (Wan et al., 2020). This suggests that “guidance” need not be a separate policy; it may be an advantage estimator structured to correct asynchronous bias.

A third mechanism is group-relative rollout guidance. In the critic-free MHGPO algorithm, group identifiers induce heterogeneous rollout groups, and agent-level advantages are z-scores within those groups: $$V_\rho(\boldsymbol{\pi}) \triangleq \mathbb{E}_{s_0\sim \rho}\Big[\, \mathbb{E}\big[\sum_{t=0}^\infty \gamma^t r_t \,\big|\, s_0 \big] \Big].$$9 (Chen et al., 3 Jun 2025). The update then uses a PPO-style clipped objective with a KL penalty to a reference model (Chen et al., 3 Jun 2025). Here guidance is relative rather than critic-based: trajectories are optimized by comparison against structured alternatives sampled from the same branching process.

A fourth mechanism is saliency- and communication-guided masking. “MAGIC-MASK: Multi-Agent Guided Inter-Agent Collaboration with Mask-Based Explainability for Reinforcement Learning” does not use the MAGPO name for its primary contribution, but explicitly interprets its method as a MAGPO-style system in which PPO policies are guided by perturbation-based saliency masks, shared critical-state information $Q$0, and KL-regularized reward-preservation objectives (Maliha et al., 30 Sep 2025). Its core policy optimizer is the clipped PPO loss

$Q$1

(Maliha et al., 30 Sep 2025), but the exploratory distribution is shaped by masks and cross-agent saliency sharing.

These formulations indicate that MAGPO is best understood as a family resemblance concept rather than a single algorithmic template. What is common is not a specific objective, but an explicit guidance channel that couples local optimization to centralized structure, partner behavior, or coordinated exploration.

6. Empirical performance, benchmarks, and comparative position

The named MAGPO framework reports evaluation on 43 tasks across 6 suites: CoordSum, Robotic Warehouse (RWARE), Level-Based Foraging (LBF), Connector, SMAX, and Multi-Agent Particle Environment (MPE) (Li et al., 24 Jul 2025). It uses 10 random seeds per task, 20M environment steps, and 122 evaluation checkpoints with 32 evaluation episodes each (Li et al., 24 Jul 2025). The headline findings are that MAGPO surpasses all CTDE baselines on 33/43 tasks and outperforms all baselines, including centralized-execution baselines, on 19/43 tasks (Li et al., 24 Jul 2025). It is reported as consistently the best CTDE method and often competitive with Sable, the strongest CTCE baseline in the benchmark set (Li et al., 24 Jul 2025).

The ablations in (Li et al., 24 Jul 2025) emphasize three points. First, guider quality matters: improvements in the underlying CTCE backbone, such as MAT or Sable, propagate into improved CTDE performance when used as MAGPO guiders. Second, the ratio bound $Q$2 is decisive: smaller $Q$3 is better on tasks that expose policy asymmetry, whereas larger $Q$4 can help when centralized coordination is naturally decentralizable. Third, the learner-side RL auxiliary loss weighted by $Q$5 can substantially improve performance in MAGPO, whereas the same addition yields only modest gains in unconstrained CTDS baselines (Li et al., 24 Jul 2025).

Related methods support the view that explicit guidance improves coordination. In the ratio game experiment of (Zhao et al., 2023), independent policy gradient can get stuck near a stationary point, whereas the sequential localized $Q$6-guided algorithm reaches global optimum more reliably and faster. In SMAC and particle environments, ASAE/MAPO shows the best performance on most tasks among the compared baselines and exhibits larger gains as the number of agents increases, supporting the claim that asynchronous estimation harms scalability (Wan et al., 2020). CoPPO outperforms several strong baselines and is competitive with MAPPO in cooperative matrix games and SMAC, with lower running policy gradient variance in the penalty game and strong performance on hard maps such as 3s5z, 10m_vs_11m, and MMM2 (Wu et al., 2021). In LLM-based multi-agent search, MHGPO-FoF and MHGPO-RR outperform MAPPO on HotpotQA, with MHGPO-RR achieving 40.86 accuracy, 37.04 EM, and 49.72 F1 versus MAPPO’s 38.22, 34.45, and 46.40, while also reducing memory and time per training step by eliminating the critic (Chen et al., 3 Jun 2025).

The available evidence therefore supports a broad empirical claim: methods that introduce structured guidance into multi-agent policy optimization tend to outperform purely decentralized PPO-style baselines when coordination demands are high, partner non-stationarity is severe, or centralized training signal would otherwise be weakly exploited. A plausible implication is that MAGPO-style designs are especially valuable in the regime where CTDE’s usual separation between centralized critics and decentralized actors becomes the limiting factor.

7. Interpretations, misconceptions, and extensions

A common misconception is that MAGPO is synonymous with MAPPO plus a centralized critic. The named MAGPO framework explicitly rejects this equivalence: its central innovation is not critic centralization but the introduction of a centralized auto-regressive guider whose policy-level exploration and improvement are continuously aligned with the decentralized learner class (Li et al., 24 Jul 2025). By contrast, standard MAPPO retains decentralized actors and uses centralized information primarily in the critic (Li et al., 24 Jul 2025).

A second misconception is that MAGPO requires a teacher policy that is later distilled into decentralized actors. The 2025 MAGPO framework instead constrains the teacher-like guider throughout training, using KL alignment and backtracking so that the centralized policy does not accumulate irreproducible behaviors (Li et al., 24 Jul 2025). This distinguishes it from vanilla CTDS approaches, where the teacher and student can diverge substantially (Li et al., 24 Jul 2025).

A third misconception is that all MAGPO-style methods rely on critics. This is contradicted by MHGPO, which is explicitly critic-free and uses group-relative reward normalization for advantage estimation (Chen et al., 3 Jun 2025). Conversely, another misconception is that critic-free methods exhaust the notion of guidance. In fact, localized $Q$7-guidance (Zhao et al., 2023), marginal synchronized advantages (Wan et al., 2020), coordinated ratio products (Wu et al., 2021), and saliency-guided masking (Maliha et al., 30 Sep 2025) all instantiate different guidance channels.

The literature also suggests extensions beyond classical cooperative control. “PRISMA: Reinforcement Learning Guided Two-Stage Policy Optimization in Multi-Agent Architecture for Open-Domain Multi-Hop Question Answering” interprets a Plan–Retrieve–Inspect–Solve–Memoize architecture as a MAGPO-style system in which Planner, Solver, and Inspector are separately optimized via Two-Stage Group Relative Policy Optimization, with the Inspector acting as a trajectory-conditioned guiding agent over the others (Liu et al., 9 Jan 2026). This suggests that MAGPO has become a more general pattern for multi-agent optimization in LLM systems as well as in Markov games.

Several limitations remain consistent across the literature. The theoretical guarantees of the named MAGPO framework assume full observability, tabular policies, exact PMD, and exact KL projection (Li et al., 24 Jul 2025). The global-optimality results of (Zhao et al., 2023) depend on sequential conditional factorization, log-linear parameterization, bounded estimation error, and concentrability conditions. ASAE’s synchrony is only approximate and may weaken under rapid policy change (Wan et al., 2020). CoPPO’s joint-ratio objectives introduce variance–coordination trade-offs controlled by inner and outer clipping (Wu et al., 2021). MHGPO has been validated primarily on a three-agent LLM search system (Chen et al., 3 Jun 2025). These constraints indicate that MAGPO remains a fertile but not yet unified theory class.

Taken together, the research record positions Multi-Agent Guided Policy Optimization as a principled response to a central weakness of conventional CTDE: centralized information is often available during training but insufficiently translated into policy-level coordination. Whether through centralized guider projection (Li et al., 24 Jul 2025), localized multi-agent value guidance (Zhao et al., 2023), synchronized marginal advantages (Wan et al., 2020), coordinated trust-region scaling (Wu et al., 2021), or structured group-relative rollouts (Chen et al., 3 Jun 2025), MAGPO methods seek to make that translation explicit.