Multi-Agent Reflective Policy Optimization (MARPO)

• Multi-Agent Reflective Policy Optimization (MARPO) is a framework that enhances sample efficiency by leveraging inter-temporal dependencies in agent trajectories.
• It employs a dynamic, KL-derived asymmetric clipping strategy to adaptively control policy updates and maintain stable learning in complex environments.
• Empirical results on benchmarks like SMAC and Google Research Football demonstrate MARPO’s superior convergence rates, win rates, and reduced performance variance.

Multi-Agent Reflective Policy Optimization (MARPO) is a policy gradient framework for multi-agent reinforcement learning (MARL) that targets the problem of sample inefficiency. The approach introduces two principal innovations: a reflection mechanism that leverages information from successive timesteps within trajectories to improve sample utilization, and a dynamic, KL-guided asymmetric clipping strategy that stabilizes policy updates during training. Empirical evaluations demonstrate its superior performance across challenging MARL environments, including StarCraft II Multi-Agent Challenge (SMAC), its SMAC-Hard variant, SMACv2, and Google Research Football (GRF), comparing favorably against state-of-the-art policy- and value-based methods (Wu et al., 28 Dec 2025).

1. Motivation and Problem Context

Conventional on-policy MARL methods, such as MAPPO, perform gradient updates using only single state–action pairs from recently collected trajectories. Each fresh rollout is costly, particularly in high-dimensional or partially observable domains. PPO-type algorithms mitigate instability in policy optimization via fixed, symmetric objective clipping but do not fully exploit the rich structure of sampled trajectories, often neglecting the influence of actions at time $k$ on returns at subsequent steps. This under-utilization limits sample efficiency and slows convergence.

MARPO addresses these limitations by integrating two key techniques:

• Reflection Mechanism: Harnesses step-to-step correlations by incorporating joint information from pairs of adjacent timesteps.
• KL-Derived Asymmetric Clipping: Replaces fixed symmetric clipping with a clipping scheme determined by the current KL divergence between policies, allowing for adaptive trust region bounds.

2. Reflection Mechanism

The reflection mechanism extends the surrogate policy objective from standard clipped PPO to encode inter-temporal dependencies. For MARPO, the multi-agent surrogate objective (Eqs. 8–10) incorporates two terms:

1. Single-step clipped surrogate ($L_0^{\mathrm{clip}}$): The per-agent analogue of PPO's clipped loss using importance ratio $\rho_i^k$ at time $k$.
2. Two-step reflective term ($L_1^{\mathrm{clip}}$): Couples the product of importance ratios at consecutive timesteps, $\rho_i^k\rho_i^{k+1}$, with the advantage evaluated at step $k+1$. The clipping function composes two clipped ratios:

$$c(\rho_i^k,\rho_i^{k+1}) = \mathrm{clip}(\rho_i^k,x_1,x_2) \cdot \mathrm{clip}(\rho_i^{k+1},x_1',x_2')$$

The full objective is

$$L(\pi,\pi_\mathrm{old}) = L_0^{\mathrm{clip}}(\pi,\pi_\mathrm{old}) + \alpha L_1^{\mathrm{clip}}(\pi,\pi_\mathrm{old}),$$

where $\alpha$ tunes the weight of the reflection term.

By leveraging two-step dependencies, MARPO increases the information extracted per trajectory, resulting in improved stability and accelerated learning. In the single-agent case, monotonic policy improvement is achieved as shown in related RPO work, which conceptually carries over to MARPO’s on-policy, multi-agent extension.

3. Asymmetric Clipping via KL Divergence

MARPO replaces PPO’s fixed symmetric clipping interval $L_0^{\mathrm{clip}}$0 with an interval dynamically derived from the forward KL divergence between current and previous policies.

For each iteration, the clipping roots $L_0^{\mathrm{clip}}$1 satisfy

$L_0^{\mathrm{clip}}$2

where $L_0^{\mathrm{clip}}$3 is a target KL divergence, maintained as an exponential moving average:

$L_0^{\mathrm{clip}}$4

Numerical inversion of $L_0^{\mathrm{clip}}$5 yields the dynamic, generally asymmetric, clipping bounds used for both steps in $L_0^{\mathrm{clip}}$6 and $L_0^{\mathrm{clip}}$7. This mechanism allows larger updates when the KL between policies is small, and restricts changes when drift is significant, yielding a theoretically sound trust region based on the convexity and non-negativity of $L_0^{\mathrm{clip}}$8.

4. Training Algorithm

The MARPO training procedure proceeds as follows:

• Inputs: Policy parameters $L_0^{\mathrm{clip}}$9, reflection weight $\rho_i^k$0, number of iterations $\rho_i^k$1, epochs per iteration $\rho_i^k$2, EMA rate $\rho_i^k$3, initial KL target $\rho_i^k$4.
• Per Iteration:
  • Sample minibatch $k$0.
  • Compute advantages $k$1 using GAE.
  • Compute $k$2 and $k$3 using current clipping bounds.
  • Update policy parameters via gradient descent on $k$4.

Algorithmic details and objective functions correspond directly to equations and the formal procedure in (Wu et al., 28 Dec 2025).

5. Empirical Evaluation

MARPO is evaluated on established MARL benchmarks:

• Environments: SMAC, SMAC-Hard, SMACv2, Google Research Football (GRF).
• Baselines: Policy-based (MAPPO, HAPPO, MAT); Value-based (QMIX, QPLEX, LDSA).
• Protocol: 10 million environment steps; evaluation on final 2 million steps. Network architectures, learning rates, and seeds are controlled.

Key experimental findings are summarized in the following table (Table 1, SMAC-Hard final win rates):

Env	MAPPO	HAPPO	LDSA	QMIX	QPLEX	MAT	MARPO
3m	99.2 ± 4.7	37.3 ± 8.7	99.6 ± 1.5	99.8 ± 0.8	5.7 ± 19.3	99.9 ± 0.6	100.0 ± 0.0
3s5z	71.0 ± 1.7	68.1 ± 1.1	12.3 ± 0.7	34.9 ± 0.8	26.4 ± 3.0	79.7 ± 0.5	87.2 ± 0.4
2s_vs_1sc	85.2 ± 13.5	0.0 ± 0.0	93.9 ± 9.5	61.4 ± 36.7	81.5 ± 15.5	10.3 ± 15.0	94.8 ± 4.2
3s_vs_4z	97.8 ± 7.5	14.4 ± 18.6	95.0 ± 6.5	75.8 ± 20.2	78.8 ± 22.8	94.4 ± 6.3	97.0 ± 3.2
10m_vs_11m	53.0 ± 4.7	57.6 ± 1.0	45.7 ± 2.6	65.2 ± 1.1	0.4 ± 0.0	43.2 ± 1.7	74.3 ± 1.2
2c_vs_64zg	97.3 ± 3.3	73.3 ± 16.9	87.0 ± 9.5	69.7 ± 32.3	35.1 ± 32.5	79.7 ± 12.3	97.4 ± 3.3

Across all environments, MARPO achieves faster convergence, higher win rates, and lower performance variance than competing methods. Figures 2–6 in (Wu et al., 28 Dec 2025) provide comprehensive supporting visualizations across tasks.

6. Theoretical Properties and Ablations

MARPO’s design is underpinned by theoretical guarantees:

• The function $k$5 is convex and non-negative, ensuring the validity of the clipping trust region for any achievable target KL divergence.
• The two-step reflection term aligns with theoretically justified monotonic improvement results from the single-agent setting.
• Ablation studies indicate performance drops when either the reflection or KL-derived clipping component is removed. Hyperparameter sweeps show that the approach is relatively insensitive to broad choices of EMA rate and bias settings, provided KL-guided clipping is used.

A plausible implication is that MARPO’s sample efficiency and stability gains largely derive from the synergy between exploiting trajectory-level structure and principled, adaptive trust region control.

7. Distinctive Advantages and Significance

The empirical and theoretical analysis highlights several advantages:

• Enhanced Sample Efficiency: By utilizing the reflection mechanism, MARPO extracts additional supervision from each sampled trajectory, particularly benefiting scenarios with rare but high-value transitions.
• Stable Policy Updates: KL-adaptive asymmetric clipping permits aggressive updates when risk is low (small KL), and restrains changes when diverging (large KL), avoiding the fixed hyperparameter tuning challenges of standard PPO.
• Robust, General Applicability: Consistent performance improvements across varied, competitive multi-agent environments indicate strong generalization capabilities.

MARPO thus represents a computationally lightweight yet theoretically principled extension of MAPPO and PPO that improves sample efficiency and convergence in complex MARL settings (Wu et al., 28 Dec 2025).