---
title: Multi-Group Policy Optimization (MUPO)
url: https://www.emergentmind.com/topics/multi-group-policy-optimization-mupo
type: topic
---

# Multi-Group Policy Optimization (MUPO)

Multi-Group Policy Optimization (MUPO) denotes a family of reinforcement-learning formulations in which policy updates are organized around multiple groups—of sampled actions, completions, objectives, stakeholder preferences, trajectories, turns, or tasks—rather than a single undifferentiated batch. Across recent work, the term is used both as a general template for group-based policy optimization and as the name of specific algorithms; accordingly, the literature is unified more by a shared structural motif than by a single canonical objective [2502.01652][2502.15145][2604.00479]. One important exception is "Policy Bifurcation in Safe Reinforcement Learning" [2403.12847], where MUPO denotes **multimodal policy optimization**, a different acronym attached to a Gaussian-mixture policy for bifurcated safe control.

## 1. Terminology and scope

Recent usage shows that MUPO is not yet a fully standardized label. In one line of work, Hybrid GRPO is explicitly re-read as an instance of a more general MUPO template in which multiple actions are sampled per state and group-derived statistics are inserted into a PPO-style update [2502.01652]. In another, Projection Optimization extends multi-objective RLHF to **multi-group** settings in which each group has its own aggregation weights, fairness exponent, and target threshold, leading to consensus and malfare objectives over shared policies [2502.15145]. In the VLM literature, MUPO becomes the name of a concrete algorithm that partitions sampled reasoning traces into several groups and adds an inter-group diversity reward [2604.00479]. In multi-task RL, PolicyGradEx instantiates a related idea by partitioning many objectives into a small number of policy groups and training one shared policy per group [2511.12779].

| Formulation | Grouping unit | Central mechanism |
|---|---|---|
| Hybrid GRPO as MUPO template | Multiple actions per state | Group statistic plus value function and PPO clip |
| Projection Optimization / MOPO | Stakeholder or objective groups | Target-set projection, consensus, malfare |
| MUPO for VLM reasoning | Clusters of reasoning traces per prompt | Local GRPO within each cluster plus diversity reward |
| PolicyGradEx | Groups of tasks/objectives | Affinity estimation and clustering into shared policies |

This suggests MUPO is best understood as a design pattern for structured policy optimization over multiple groups, with the precise meaning of “group” determined by the domain.

## 2. Shared mathematical pattern

A recurring abstraction appears in several papers. First, the algorithm forms groups: for example, multiple actions sampled from the same state, multiple responses sampled for the same prompt, or multiple tasks assigned to the same policy. Second, it computes intra-group statistics such as means, variances, relative rewards, ranks, or target-set distances. Third, it uses those quantities to define an advantage-like signal or a higher-level optimization target. Fourth, it applies a stabilizing update mechanism such as PPO clipping, KL regularization, or projection onto feasible sets.

The Hybrid GRPO reinterpretation states this generic structure explicitly. It describes MUPO through: group-based sampling, intra-group statistics, a hybrid advantage, and trust-region or clipping, with a generic form
\[
A_T^{\text{MUPO}}
=
g\Big(\{\tilde{R}^{(t)}\}_{t=1}^N, V(S_T), V(S_{T+1}), \dots\Big),
\]
where \(g\) parameterizes the hybrid scheme [2502.01652]. In that reading, MUPO is not restricted to critic-free updates; it may combine empirical group estimates with bootstrapped value terms.

The multi-agent heterogeneous rollout formulation makes the same point in a critic-free setting. There, each rollout pair \((k,i)\) is assigned a group identifier \(m_{k,i}\), and the advantage is standardized within its group:
\[
\hat{A}_{k,i} =
\frac{R_{k,i} - \text{mean}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})}
{\text{std}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})}.
\]
The policy objective is then PPO-like but without a value function [2506.02718]. The commonality with Hybrid GRPO is structural rather than architectural: both replace a global baseline with group-relative estimation.

A plausible implication is that MUPO is less a single estimator than a hierarchy of design choices: how groups are formed, which statistics are computed within them, how those statistics are converted into learning signals, and how the resulting objective is regularized.

## 3. Stakeholder-, objective-, and task-group optimization

In RLHF, the most explicit multi-group formalization appears in "Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF" [2502.15145]. There are \(m\) objectives with rewards \(r_i(x,y)\), and \(N\) groups, each with its own aggregation weights \(\alpha^{(n)}\), exponent \(p^{(n)} \le 1\), and threshold \(c^{(n)}\). The framework defines an expected reward vector \(S(\pi)\in\mathbb{R}^m\) and target sets
\[
W^{(n)} =
\Big\{
z\in\mathbb{R}_{\ge 0}^m :
\Big(\sum_{i=1}^m \alpha_i^{(n)} z_i^{p^{(n)}}\Big)^{1/p^{(n)}} \ge c^{(n)}
\Big\}.
\]
Two MUPO objectives follow. The **consensus** objective seeks a single policy close to the intersection \(W^{\mathrm{cons}}=\cap_n W^{(n)}\):
\[
\pi^* = \arg\min_\pi d(S(\pi), W^{\mathrm{cons}}).
\]
The **malfare** objective aggregates squared distances to each group’s target set:
\[
\pi^* =
\arg\min_\pi
\left(
\sum_{n=1}^N \zeta_n \big(d^2(S(\pi),W^{(n)})\big)^q
\right)^{1/q}.
\]
The algorithmic core reduces nonlinear multi-group aggregation to a sequence of linear direction subproblems, solved by reward-based or reward-free primitives, and the paper reports sublinear regret for both offline and online variants [2502.15145].

A related but distinct use of group structure appears in "Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimation" [2511.12779]. There the goal is to partition \(n\) tasks into \(k \ll n\) groups, train one shared policy per group, and maximize
\[
\frac{1}{n} \sum_{g=1}^k \sum_{i\in G_g} R_i(\theta^{(g)}).
\]
PolicyGradEx estimates a task-affinity matrix from first-order surrogate adaptation and clusters tasks by maximizing intra-cluster affinity scores [2511.12779]. This suggests that “group” in MUPO can reside not only in rollout space but also at the level of objectives, stakeholders, or tasks.

## 4. Group-relative advantage construction and efficiency mechanisms

The Hybrid GRPO reinterpretation provides a canonical example of group-derived advantage construction. For each macro-step \(T\), it samples \(N\) actions
\[
a_T^{(t)} \sim \pi_\theta(a\mid S_T), \quad t=1,\dots,N,
\]
computes transformed rewards \(\tilde R^{(t)} = f(r(S_T,a_T^{(t)}))\), and defines
\[
A_T^{\text{Hybrid}} =
\left[
\frac{1}{N}\sum_{t=1}^N \tilde R^{(t)} + \gamma V(S_{T+1})
\right] - V(S_T).
\]
The clipped objective remains PPO-like, but the immediate reward term is replaced by a group average [2502.01652]. The stated purpose is a bias–variance compromise: critic-based stabilization from PPO combined with multi-sample averaging.

MO-GRPO addresses a different failure mode: reward hacking in multi-objective GRPO. Standard GRPO sums reward components and normalizes once within the sampled group, which biases the gradient toward high-variance objectives. MO-GRPO instead normalizes each component separately within the group,
\[
\tilde R_i(q,o_g)=\frac{R_i(q,o_g)-\mu_i(q)}{\sigma_i(q)},
\]
and sums the normalized components in the advantage
\[
A_g^{\mathrm{MO}}=\sum_{i=1}^K \frac{R_i(q,o_g)-\mu_i(q)}{\sigma_i(q)}.
\]
The paper shows that this equalizes contributions in the uncorrelated case and preserves preference ordering under positive affine transformations [2509.22047]. In MUPO terms, this is a within-group normalization rule that prevents one objective group from dominating due to scale or variance.

Two later works target the computational cost of large-group optimization. DPPO introduces prompt-level and completion-level pruning while preserving an unbiased estimate of the full GRPO gradient through importance-sampling rescaling factors \(\gamma(q)\) and \(\gamma(o,q)\) [2603.04135]. Mu-GRPO, by contrast, asks how stale GRPO can become, organizes training into a small number of large rollout–optimization stages, and stabilizes high-staleness learning with relaxed clipping and negative-advantage veto [2605.17570]. The reported result is that GRPO-style learning can tolerate much larger rollout staleness than standard low-\(\mu\) practice, provided the objective is modified appropriately [2605.17570].

## 5. Multi-agent, turn-level, and structural decision variants

In multi-agent systems, group structure often moves from response space to structural or heterogeneous rollout space. Graph-GRPO samples \(K\) communication graphs per query and computes an edge-level conditional success rate
\[
S_{ij}
=
\frac{\sum_{k=1}^K \mathbb{I}((i,j)\in \mathcal G_k)\, r_k}
{\sum_{k=1}^K \mathbb{I}((i,j)\in \mathcal G_k)+\epsilon},
\]
followed by a z-scored edge advantage
\[
A_{ij} = \frac{S_{ij}-\mu_S}{\sigma_S+\epsilon}.
\]
The policy gradient is then applied at the level of individual edges rather than whole graphs, enabling fine-grained credit assignment in topology optimization [2603.02701].

"Heterogeneous Group-Based Reinforcement Learning for LLM-based Multi-Agent Systems" proposes Multi-Agent Heterogeneous Group Policy Optimization (MHGPO), a critic-free algorithm in which group identifiers can bind together rollouts from different agents and different intermediate prompts [2506.02718]. Its importance lies in relaxing the homogeneous-group assumption inherited from single-agent GRPO.

Agent-GSPO operates in sequence space rather than action space. For each query \(x\), GSPO samples a group of \(G\) sequences, normalizes rewards within that group, and uses a sequence-level ratio
\[
s_i(\theta)=
\left(
\frac{\pi_\theta(y_i\mid x)}
{\pi_{\theta_{\text{old}}}(y_i\mid x)}
\right)^{1/|y_i|}.
\]
Agent-GSPO then uses this objective to optimize communication-aware rewards in multi-agent systems, including token, turn, and repetition penalties, together with an optional dual update for a communication budget [2510.22477].

Turn- and segment-level extensions further refine the grouping granularity. GTPO redefines multi-turn tool-integrated reasoning as an MDP over turns, with turn-level discounted return
\[
R_{i,j}=\sum_{m=j}^{\mathcal T}\gamma^{m-j} r_{i,m},
\]
and normalizes those returns across trajectories at each turn index [2511.14846]. Workflow-R1 introduces GSsPO, where each Think–Action cycle is a sub-sequence \(s\), and the sub-sequence ratio is the geometric mean of token-level ratios within that segment [2602.01202]. These works show that the “group” in MUPO may be a turn, a sub-sequence, a graph edge, or a heterogeneous multi-agent rollout, provided the optimization unit matches the semantics of the decision process.

## 6. Divergent-thinking MUPO in vision-language models, and terminological ambiguity

The most explicit use of MUPO as a named algorithm appears in "All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models" [2604.00479]. That paper argues that GRPO induces **diversity collapse**: reasoning diversity drops sharply within the first approximately 20 steps of GRPO training, and RL-trained VLMs become deep but narrow thinkers, whereas base models remain broader and achieve stronger acc@\(k\) scaling when multiple samples are drawn [2604.00479]. MUPO addresses this by sampling \(N=15\) responses per example, embedding each reasoning segment with Qwen3-Embedding-0.6B, clustering the embeddings into \(K\) groups by constrained \(k\)-means, and optimizing a multi-group objective
\[
\mathcal J_{\text{MUPO}}(\theta)
=
\mathbb E\Big[
\sum_{k=1}^K
w_k |G_k|
\sum_{i=1}^{|G_k|}
\min\big(
r_i(\theta)\hat A_i^k,\,
\mathrm{clip}(r_i(\theta),1-\epsilon,1+\epsilon)\hat A_i^k
\big)
\Big],
\]
with load-balancing weight
\[
w_k=\left(\frac{N}{K|G_k|}\right)^\beta.
\]
Each sample reward is
\[
R_i^k = R_{\text{acc}} + R_{\text{fmt}} + \lambda \cdot \mathbf 1[R_{\text{acc}}=1]\cdot R_{\text{div}},
\]
and the diversity reward averages cosine distance to reasoning traces in other groups [2604.00479]. The paper reports that MUPO-Thinker-7B reaches average Acc@1 of \(51.6\%\) and Acc@4 of \(58.8\%\) on mathematical benchmarks, and average Acc@1 of \(65.6\%\) and Acc@4 of \(72.4\%\) on MMStar, HallusionBench, and MMVet; the 3B model reaches \(48.3\%\) Acc@1 and \(56.0\%\) Acc@4 [2604.00479]. An ablation over the number of groups reports \(K=1\) at \(51.9\%\), \(K=2\) at \(54.3\%\), \(K=3\) at \(55.4\%\), \(K=4\) at \(55.3\%\), and \(K=5\) at \(55.0\%\), making \(K=3\) the best setting in that study [2604.00479].

Two misconceptions are therefore worth correcting. First, MUPO is not a single universally agreed-upon algorithm; current usage spans target-set RLHF, group-relative rollout methods, task clustering, and explicit diversity-aware VLM training. Second, MUPO is not uniformly critic-free: Hybrid GRPO-style readings retain \(V(s)\), whereas other variants remove critics entirely [2502.01652]. A further terminological complication is the safe-RL paper in which MUPO stands for **multimodal policy optimization**, not multi-group policy optimization. There the policy is a Gaussian mixture
\[
\pi_\theta(\cdot\mid x)=\sum_{i=1}^k \pi_{\mathrm{gate}}^i(x;\theta)\,\mathcal N(\mu_i(x;\theta),\sigma_i^2(x;\theta)),
\]
and bifurcated behavior is realized by selecting the Gaussian component with the highest mixing coefficient [2403.12847]. This is a distinct lineage centered on policy bifurcation in non-simply connected safe sets.

Taken together, the literature indicates that MUPO is best viewed as a broad organizing idea: optimize policies not against a single monolithic signal, but against multiple structured groups whose internal statistics, geometry, or diversity carry information that a flat objective would discard. This suggests that future consolidation of the term will depend on whether the field converges on a shared mathematical core—such as grouped advantages, grouped constraints, or grouped diversity incentives—or continues to use MUPO as a flexible label for several related, but distinct, policy-optimization strategies.

Source: https://www.emergentmind.com/topics/multi-group-policy-optimization-mupo