---
title: 'MHGPO: Multi-Agent Policy Optimization'
url: https://www.emergentmind.com/topics/multi-agent-heterogeneous-group-policy-optimization-mhgpo
type: topic
---

# MHGPO: Multi-Agent Policy Optimization

Multi-Agent Heterogeneous Group Policy Optimization (MHGPO) is a reinforcement learning framework designed for optimizing large language model (LLM)-based multi-agent systems (MAS) in a cooperative, parameter-shared, decentralized-execution setting. Unlike conventional multi-agent reinforcement learning (MARL) algorithms such as Multi-Agent Proximal Policy Optimization (MAPPO), which rely on critic networks to estimate value functions and facilitate policy updates, MHGPO eschews value-based critics altogether. Instead, it employs a group-based relative advantage estimation within batches of heterogeneous rollout trajectories, realizing a stable, memory-efficient, and scalable optimization regime. Empirical evidence demonstrates superior task performance and computational efficiency compared to MAPPO on LLM-based multi-agent search systems [2506.02718].

## 1. Formal Problem Definition

MHGPO formalizes the MAS setting as a parameter-shared, cooperative-training, decentralized-execution MARL problem, where $n$ heterogeneous agents $A_1, \dots, A_n$ are all instantiated from a single LLM backbone. System states $s \in S$ represent the current prompt delivered to the next pipeline agent, while each agent $A_k$ emits a variable-length token sequence $o_k$, with actions $a^t_k = o^t_k \in \mathcal{V}$ composing the output. The deterministic transition function $P(s' \mid s, a_k)$ is determined by prompt concatenation and agent routing rules. Reward assignment consists of a global shared reward $R^{\rm shared}$ (e.g., F1 score against gold answers) retroactively distributed along the agent chain, plus step-level agent-specific penalties $R_k^{\rm spe}(q_k, o_k)$ for malformed outputs. All agents are parameterized by the joint policy $\pi_\theta(o \mid q)$. The optimization objective is to maximize the cumulative expected (undiscounted) return:

$$
J(\theta)=\mathbb{E}_{q\sim D,\,o\sim\pi_\theta}\left[\sum_{k=1}^n\sum_{t=1}^{T_k}R^t_{k}\right]
$$

## 2. Critic-Free Algorithmic Framework

MHGPO eliminates the need for value-network or critic function approximation by employing a group-based advantage estimation (Group-Relative Advantage, GRA). Each mini-batch training iteration follows:

- **Group Rollout Sampling:** For each input question $q$, sample $G$ full trajectories $\{\mathcal{T}_i\}_{i=1}^G$ under a reference policy $\theta_{\rm ref}$.
- **Backward Reward Propagation:** Assign shared reward $R^{\rm shared}_i$ to each final answer and propagate rewards backward via averaging. Agent-specific penalties $R^{\rm spe}_{k,i}$ are added to obtain total $R_{k,i}$ per agent and trajectory.
- **Group-Relative Advantage Calculation:**
  
  $$
  \Delta_{k,i} = \hat{A}_{k,i} = \frac{R_{k,i} - \mu_g}{\sigma_g}
  $$
  with
  $$
  \mu_g = \frac{1}{|g|} \sum_{(l,j)\in g} R_{l,j}\,, \quad \sigma_g = \sqrt{\frac{1}{|g|} \sum_{(l,j)\in g}(R_{l,j} - \mu_g)^2}\,,
  $$
  where group $g = m_{k,i}$ indexes the collection of rollouts that share a sampling context.
- **Critic-Free Policy Gradient Update:** Using PPO-style clipping and reference policy KL-penalization, the loss is

  $$
  \mathcal{L}(\theta) = -\mathbb{E}_{q, \{o_{k,i}\}}\left[\frac{1}{n}\sum_{k=1}^n \frac{1}{G_k}\sum_{i=1}^{G_k} \sum_{t=1}^{T_{k,i}} \min\left(r^t_{k,i} \Delta^t_{k,i},\,\mathrm{clip}\left(r^t_{k,i},1-\epsilon,1+\epsilon\right)\Delta^t_{k,i}\right) \right] + \beta D_{KL}\left(\pi_\theta\|\pi_{\rm ref}\right)
  $$
  where $r^t_{k,i} = \frac{\pi_\theta(o^t_{k,i}\mid q,o^{<t}_{k,i})}{\pi_{\rm ref}(o^t_{k,i}\mid q,o^{<t}_{k,i})}$.

## 3. Group Rollout Sampling Methodologies

MHGPO’s scalability and stability are shaped by the choice of group rollout sampling strategy. For each, a *base fork* utility is deployed to handle pipeline branching:

- **Independent Sampling (IS):** For each agent $A_i$ and each $q$, fork only at $A_i$, producing $G$ continuations; repeated for $i = 1,\dots, n$, resulting in homogeneous groups. Low variance per agent, no cross-agent coupling, $nG$ rollouts per sample.
- **Fork-on-First (FoF):** Forks only at the first pipeline agent, so downstream agents receive $G$ distinct inputs, but groups are always size $G$. Captures full pipeline dependencies, $nG$ agent calls per batch, higher ultimate accuracy.
- **Round-Robin (RR):** Randomly selects the fork point per $q$ using categorical probabilities $p_1,\dots,p_n$, with post-hoc re-grouping to guarantee equal group sizes. Allows tuning between compute cost and variance, with typical overhead of $(n-1)+G$ agent calls per sample.

Sampling strategy selection affects agent variance, inter-agent dependency modeling, and computational overhead.

## 4. Policy Update Mechanism

MHGPO combines the above mechanisms into a PPO-inspired gradient update step, eschewing the learning and maintenance of a value function:

$$
\theta \leftarrow \theta - \alpha \nabla_\theta \left[ -\frac{1}{n}\sum_{k,i,t} \min(r^t_{k,i}\Delta^t_{k,i},\,\mathrm{clip}(r^t_{k,i},1-\epsilon,1+\epsilon) \Delta^t_{k,i}) + \beta D_{KL} (\pi_\theta \| \pi_{\rm ref}) \right]
$$

The KL penalty $\beta$ and PPO clipping parameter $\epsilon$ serve the same stabilizing function as in single-agent PPO. The absence of a value-function network eliminates the need for memory-intensive critic computations.

## 5. Empirical and Theoretical Comparisons with MAPPO

Comparison with MAPPO—the primary baseline—highlights key theoretical and practical distinctions:

| Algorithm         | Critic Needed | Memory Usage | Training Stability | Compute Cost | Final F1 (HotpotQA) |
|-------------------|--------------|--------------|-------------------|--------------|---------------------|
| MAPPO             | Yes          | High         | Less stable       | High         | 46.40%              |
| MHGPO-IS          | No           | Low          | Most stable, lower accuracy ceiling | Moderate    | 45.58%              |
| MHGPO-FoF         | No           | Low          | High, slower convergence | Moderate    | 49.43%              |
| MHGPO-RR          | No           | Low          | Balanced          | Lowest       | 49.72%              |

- MAPPO requires a full-size critic network for each agent, incurring up to 30–40% higher GPU memory usage and greater training instability, particularly in heterogeneous output regimes.
- MHGPO’s critic-free paradigm not only reduces hardware demands but also delivers smoother convergence curves and less collapse across agents [2506.02718].

## 6. Experimental Protocol and Quantitative Outcomes

Evaluations were conducted in a multi-agent search system (MASS) with a three-agent pipeline: Rewriter (query generation), Reranker (snippet selection), and Answerer (final synthesis). The system used Contriever-based retrieval from Wikipedia, and was benchmarked on HotpotQA (in-domain), 2WikiMultiHopQA, and MuSiQue (out-of-domain) using metrics such as Accuracy, Exact Match (EM), and F1.

Key experimental hyperparameters included batch size 512, one RL epoch per step, PPO epochs = 1, group size $G=4$, and RR rollout probabilities $(0.7, 0.1, 0.2)$. Results demonstrated that all MHGPO variants outperformed MAPPO on in-domain and out-of-domain tasks, with MHGPO-FoF and MHGPO-RR achieving the highest F1 scores (49.43% and 49.72% respectively) on HotpotQA, compared to MAPPO’s 46.40%. The IS strategy converged fastest but to a lower final score, FoF reached the highest final accuracy with slower convergence, and RR offered near-FoF accuracy at 15–20% lower rollout overhead [2506.02718].

## 7. Ablation, Sensitivity, and Design Trade-Offs

- **Group Size ($G$):** Increasing $G$ reduces estimator variance at the expense of linear growth in rollout cost. $G=4$ was found to offer a balanced trade-off.
- **Sampling Strategy Effects:** IS minimizes agent interference but fails to capture full pipeline dependencies. FoF maximizes global coupling for highest accuracy at moderate cost. RR strikes a balance between accuracy (near FoF) and rollout efficiency.
- **RR Probability Tuning ($\{p_i\}$):** Biasing RR toward early forks (e.g., $p_1=0.7$) stabilizes downstream agents by enforcing early coupling.

## 8. Scope, Limitations, and Prospects

Experiments to date focus on a fixed three-agent QA pipeline. Adaptation to larger agent pools, cyclic topologies, or dynamic agent assignment is not yet demonstrated. Training does not incorporate warm-up phases such as supervised fine-tuning (SFT); integration with SFT or direct preference optimization (DPO) is posited as a possible enhancement. Other prospective extensions include dynamic or hierarchical group sizing, handling branching MAS graph structures, and incorporating off-policy correction to leverage historical rollouts and improve sample efficiency.

A plausible implication is that the group advantage approach in MHGPO could generalize to MAS with more intricate coordination flows, variable agent roles, or open-ended dialogue architectures, contingent on future empirical validation.

MHGPO thus constitutes an efficient, scalable, and empirically validated alternative to critic-dependent MARL approaches in LLM-driven MAS, leveraging heterogeneity-aware group-based optimization principles for end-to-end policy improvement [2506.02718].

Source: https://www.emergentmind.com/topics/multi-agent-heterogeneous-group-policy-optimization-mhgpo