MAPGRPO: Multi-Agent Progressive Policy Optimization

• MAPGRPO is a distributed MARL framework that employs shared policy parameterization to efficiently aggregate local agent updates.
• It utilizes progressive group-relative policy updates with role-specific rewards and group normalization to reduce variance and stabilize credit assignment.
• Empirical results show MAPGRPO's superior scalability and performance in environments like cooperative navigation, multi-hop QA, and robotic control.

Multi-Agents Progressive Group Relative Policy Optimization (MAPGRPO) is a distributed optimization framework for scalable multi-agent reinforcement learning (MARL). It targets efficient policy optimization in environments with numerous agents, where joint action and observation spaces are high-dimensional and present tractability challenges for conventional centralized or decentralized approaches. MAPGRPO is characterized by progressive and group-relative policy updates that leverage shared parameterization, distributed gradient-based learning, and group normalization strategies to ensure both scalability and stability in collaborative, competitive, and reasoning-oriented tasks.

1. Distributed Optimization and Shared Policy Parameterization

MAPGRPO formulates the MARL problem as a distributed optimization. Instead of each agent learning an individual policy, all agents share a central policy parameter $\theta$ and perform local updates based on their individual interactions. Each agent receives a local copy $\theta_n$ and computes a gradient update relative to its experience, followed by a central aggregation step: $$\theta_n \leftarrow \theta_0 + \alpha_1 \nabla_{\theta_n} L_n(\theta, \theta_n)$$

$$\theta \leftarrow \theta + \epsilon \nabla_{\theta} \sum_n L_n(\theta, \theta_n)$$

This approach efficiently aggregates local improvements while circumventing the exponential scaling in parameter space typical of separate agent policies. The parameter sharing assumption, supported by empirical observations, shows that agents’ policies naturally cluster in parameter space, allowing the central policy to approximate all agents well (Khan et al., 2018).

2. Progressive Group Relative Policy Optimization

MAPGRPO extends standard group relative policy optimization by progressively and sequentially optimizing specialized agents (modules), each with a role-specific reward function. For example, in multi-hop reasoning settings, the optimization proceeds as follows:

• Agent $k$ is trained given fixed policies $\theta^*_{<k}$ for previous agents, maximizing an objective tailored to its role: $$\theta_k^* = \arg\max_{\theta_k} J_k(\theta_k | \theta^*_{<k})$$ Reward functions, such as those in OPERA, are designed for planning ($f_{logic}$, $f_{struct}$, $f_{exec}$), answer extraction (exact match, sufficiency adequacy), and query rewriting (retrieval effectiveness, format compliance). Group-normalized advantages are computed: $$A_i(x, y_i) = r(x, y_i) - \frac{1}{G} \sum_{j=1}^G r(x, y_j)$$ Progressive sequencing of the optimization ensures that each agent is optimized on realistic distributions induced by previously trained modules, reducing distribution mismatch and facilitating stable credit assignment (Liu et al., 22 Aug 2025).

3. Group Relative Normalization and Variance Reduction

MAPGRPO implements group-relative reward normalization to reduce policy gradient variance and stabilize learning. For a group of $G$ candidate outputs, each agent’s advantage is normalized against the group mean reward, and the policy gradient update is taken over these normalized quantities: $$\nabla_\theta J_{GRPO} = \mathbb{E}\left[ \sum_{i=1}^G A_i(x, y_i) \nabla_\theta \log \pi_\theta(y_i|x) \right]$$ A KL-divergence regularization term further constrains policy divergence from reference distributions: $$\mathcal{L}_{GRPO}(\theta) = - J_{GRPO}(\theta) + \beta D_{KL}[\pi_\theta || \pi_{ref}]$$ In continuous control, group normalization is extended via trajectory-based clustering and state-aware advantage estimation, where policies are grouped by trajectory features and states by DBSCAN clustering. This enables group-specific advantage clipping for highly dimensional action/state spaces (Khanda et al., 25 Jul 2025).

4. Empirical Performance and Scalability

MAPGRPO achieves superior scalability and competitive or better performance compared to traditional approaches:

• In cooperative navigation and predator–prey benchmarks, distributed MAPGRPO (DiMAPG) quickly converges and yields higher minimum rewards across agents compared to joint action "Kitchensink" and independent methods (Khan et al., 2018).
• Survival tasks with hundreds of agents are only tractable via MAPGRPO, as centralized and fully decentralized approaches fail due to the curse of dimensionality.
• In multi-hop retrieval (OPERA), MAPGRPO yields statistically significant improvements in exact-match accuracy on complex benchmarks (HotpotQA, 2WikiMultiHopQA, Musique). Sequential agent optimization produces robust, high-quality multi-step reasoning plans and answers (Liu et al., 22 Aug 2025).
• In curriculum learning with counterfactual group relative policy advantage (CGRPA), MAPGRPO stabilizes credit assignment under dynamically shifting task difficulty, leading to faster policy exploration and improved final win rates in challenging SMAC benchmarks (Jin et al., 9 Jun 2025).

5. Theoretical Foundations and Optimization Guarantees

MAPGRPO draws on distributed optimization principles, multi-agent performance difference lemmas, and policy mirror descent:

• The joint performance gap can be telescopically decomposed: $$J(\pi^*) - J(\pi) = \frac{1}{1-\gamma} \sum_{m=1}^N \mathbb{E}_{s \sim \nu^*, a^{1:m-1} \sim \pi^{*1:m-1}} \left[ \langle Q_\pi^{1:m}(s, a^{1:m}), \pi^{*m}(\cdot) - \pi^m(\cdot) \rangle \right]$$ with local policy updates along maximally improving directions, enabling sequential (progressive) agent optimization and provable global optimality at sublinear convergence rates (Zhao et al., 2023).
• In multi-agent PPO extensions, coordinated update mechanisms (CoPPO) achieve monotonic joint policy improvement and dynamic credit assignment by modulating each agent's update step size according to joint behavior of peers, verified theoretically and via experiments (Wu et al., 2021).

6. Credit Assignment, Curriculum, and Adaptivity

Fine-grained credit assignment is achieved through counterfactual advantage functions and group-based KL regularization: $$A_i^{CF}(s, u) = Q_{tot}(s, u) - \mathbb{E}_{\tilde{u}_i \sim \pi_i}[ Q_{tot}(s, (u^{-i}, \tilde{u}_i)) ] - \alpha D_{KL}(\pi_i \| \bar{\pi}_g)$$ FlexDiff curriculum scheduler integrates dynamic difficulty adjustment with agent performance statistics (sliding window mean, variance, momentum buffer). This adaptivity ensures agents learn generalizable strategies across shifting environment meta-stationarity (Jin et al., 9 Jun 2025).

7. Extensions, Limitations, and Applications

MAPGRPO principles extend naturally to continuous control domains:

• Trajectory-based clustering and state-aware advantage estimation facilitate robust optimization in high-dimensional robotic tasks, with theoretical guarantees on convergence and computational complexity.
• Applications include multi-robot systems for locomotion/manipulation, multi-hop QA, legal reasoning/citation generation, and multi-agent planning under partial observability.
• Open challenges involve hyperparameter selection for curriculum schedules, computational overhead in multi-agent/judge evaluations, and careful aggregation of agent outputs for optimal ensemble behavior.

MAPGRPO thus provides a rigorous, scalable foundation for progressive, group-relative multi-agent policy optimization, combining distributed learning, progressive specialization, and robust credit assignment to solve large-scale cooperative and reasoning-intensive problems.