Multi-Group Policy Optimization (MUPO)

• MUPO is a reinforcement learning design pattern that organizes policy updates across multiple groups defined by actions, tasks, or objectives.
• It computes intra-group statistics like means and variances to construct advantage signals, balancing bias–variance trade-offs in updates.
• By partitioning samples into groups, MUPO enables tailored optimization strategies that enhance performance and stability in complex environments.

Multi-Group Policy Optimization (MUPO) denotes a family of reinforcement-learning formulations in which policy updates are organized around multiple groups—of sampled actions, completions, objectives, stakeholder preferences, trajectories, turns, or tasks—rather than a single undifferentiated batch. Across recent work, the term is used both as a general template for group-based policy optimization and as the name of specific algorithms; accordingly, the literature is unified more by a shared structural motif than by a single canonical objective (Sane, 30 Jan 2025, Xiong et al., 21 Feb 2025, Tian et al., 1 Apr 2026). One important exception is "Policy Bifurcation in Safe Reinforcement Learning" (Zou et al., 2024), where MUPO denotes multimodal policy optimization, a different acronym attached to a Gaussian-mixture policy for bifurcated safe control.

1. Terminology and scope

Recent usage shows that MUPO is not yet a fully standardized label. In one line of work, Hybrid GRPO is explicitly re-read as an instance of a more general MUPO template in which multiple actions are sampled per state and group-derived statistics are inserted into a PPO-style update (Sane, 30 Jan 2025). In another, Projection Optimization extends multi-objective RLHF to multi-group settings in which each group has its own aggregation weights, fairness exponent, and target threshold, leading to consensus and malfare objectives over shared policies (Xiong et al., 21 Feb 2025). In the VLM literature, MUPO becomes the name of a concrete algorithm that partitions sampled reasoning traces into several groups and adds an inter-group diversity reward (Tian et al., 1 Apr 2026). In multi-task RL, PolicyGradEx instantiates a related idea by partitioning many objectives into a small number of policy groups and training one shared policy per group (Zhang et al., 16 Nov 2025).

Formulation	Grouping unit	Central mechanism
Hybrid GRPO as MUPO template	Multiple actions per state	Group statistic plus value function and PPO clip
Projection Optimization / MOPO	Stakeholder or objective groups	Target-set projection, consensus, malfare
MUPO for VLM reasoning	Clusters of reasoning traces per prompt	Local GRPO within each cluster plus diversity reward
PolicyGradEx	Groups of tasks/objectives	Affinity estimation and clustering into shared policies

This suggests MUPO is best understood as a design pattern for structured policy optimization over multiple groups, with the precise meaning of “group” determined by the domain.

2. Shared mathematical pattern

A recurring abstraction appears in several papers. First, the algorithm forms groups: for example, multiple actions sampled from the same state, multiple responses sampled for the same prompt, or multiple tasks assigned to the same policy. Second, it computes intra-group statistics such as means, variances, relative rewards, ranks, or target-set distances. Third, it uses those quantities to define an advantage-like signal or a higher-level optimization target. Fourth, it applies a stabilizing update mechanism such as PPO clipping, KL regularization, or projection onto feasible sets.

The Hybrid GRPO reinterpretation states this generic structure explicitly. It describes MUPO through: group-based sampling, intra-group statistics, a hybrid advantage, and trust-region or clipping, with a generic form

$$A_T^{\text{MUPO}} = g\Big(\{\tilde{R}^{(t)}\}_{t=1}^N, V(S_T), V(S_{T+1}), \dots\Big),$$

where $g$ parameterizes the hybrid scheme (Sane, 30 Jan 2025). In that reading, MUPO is not restricted to critic-free updates; it may combine empirical group estimates with bootstrapped value terms.

The multi-agent heterogeneous rollout formulation makes the same point in a critic-free setting. There, each rollout pair $(k,i)$ is assigned a group identifier $m_{k,i}$, and the advantage is standardized within its group: $$\hat{A}_{k,i} = \frac{R_{k,i} - \text{mean}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})} {\text{std}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})}.$$ The policy objective is then PPO-like but without a value function (Chen et al., 3 Jun 2025). The commonality with Hybrid GRPO is structural rather than architectural: both replace a global baseline with group-relative estimation.

A plausible implication is that MUPO is less a single estimator than a hierarchy of design choices: how groups are formed, which statistics are computed within them, how those statistics are converted into learning signals, and how the resulting objective is regularized.

3. Stakeholder-, objective-, and task-group optimization

In RLHF, the most explicit multi-group formalization appears in "Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF" (Xiong et al., 21 Feb 2025). There are $m$ objectives with rewards $r_i(x,y)$, and $N$ groups, each with its own aggregation weights $\alpha^{(n)}$, exponent $p^{(n)} \le 1$, and threshold $g$0. The framework defines an expected reward vector $g$1 and target sets

$g$2

Two MUPO objectives follow. The consensus objective seeks a single policy close to the intersection $g$3: $g$4 The malfare objective aggregates squared distances to each group’s target set: $g$5 The algorithmic core reduces nonlinear multi-group aggregation to a sequence of linear direction subproblems, solved by reward-based or reward-free primitives, and the paper reports sublinear regret for both offline and online variants (Xiong et al., 21 Feb 2025).

A related but distinct use of group structure appears in "Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimation" (Zhang et al., 16 Nov 2025). There the goal is to partition $g$6 tasks into $g$7 groups, train one shared policy per group, and maximize

$g$8

PolicyGradEx estimates a task-affinity matrix from first-order surrogate adaptation and clusters tasks by maximizing intra-cluster affinity scores (Zhang et al., 16 Nov 2025). This suggests that “group” in MUPO can reside not only in rollout space but also at the level of objectives, stakeholders, or tasks.

4. Group-relative advantage construction and efficiency mechanisms

The Hybrid GRPO reinterpretation provides a canonical example of group-derived advantage construction. For each macro-step $g$9, it samples $(k,i)$0 actions

$(k,i)$1

computes transformed rewards $(k,i)$2, and defines

$(k,i)$3

The clipped objective remains PPO-like, but the immediate reward term is replaced by a group average (Sane, 30 Jan 2025). The stated purpose is a bias–variance compromise: critic-based stabilization from PPO combined with multi-sample averaging.

MO-GRPO addresses a different failure mode: reward hacking in multi-objective GRPO. Standard GRPO sums reward components and normalizes once within the sampled group, which biases the gradient toward high-variance objectives. MO-GRPO instead normalizes each component separately within the group,

$(k,i)$4

and sums the normalized components in the advantage

$(k,i)$5

The paper shows that this equalizes contributions in the uncorrelated case and preserves preference ordering under positive affine transformations (Ichihara et al., 26 Sep 2025). In MUPO terms, this is a within-group normalization rule that prevents one objective group from dominating due to scale or variance.

Two later works target the computational cost of large-group optimization. DPPO introduces prompt-level and completion-level pruning while preserving an unbiased estimate of the full GRPO gradient through importance-sampling rescaling factors $(k,i)$6 and $(k,i)$7 (Zhu et al., 4 Mar 2026). Mu-GRPO, by contrast, asks how stale GRPO can become, organizes training into a small number of large rollout–optimization stages, and stabilizes high-staleness learning with relaxed clipping and negative-advantage veto (Tian et al., 17 May 2026). The reported result is that GRPO-style learning can tolerate much larger rollout staleness than standard low-$(k,i)$8 practice, provided the objective is modified appropriately (Tian et al., 17 May 2026).

5. Multi-agent, turn-level, and structural decision variants

In multi-agent systems, group structure often moves from response space to structural or heterogeneous rollout space. Graph-GRPO samples $(k,i)$9 communication graphs per query and computes an edge-level conditional success rate

$m_{k,i}$0

followed by a z-scored edge advantage

$m_{k,i}$1

The policy gradient is then applied at the level of individual edges rather than whole graphs, enabling fine-grained credit assignment in topology optimization (Cang et al., 3 Mar 2026).

"Heterogeneous Group-Based Reinforcement Learning for LLM-based Multi-Agent Systems" proposes Multi-Agent Heterogeneous Group Policy Optimization (MHGPO), a critic-free algorithm in which group identifiers can bind together rollouts from different agents and different intermediate prompts (Chen et al., 3 Jun 2025). Its importance lies in relaxing the homogeneous-group assumption inherited from single-agent GRPO.

Agent-GSPO operates in sequence space rather than action space. For each query $m_{k,i}$2, GSPO samples a group of $m_{k,i}$3 sequences, normalizes rewards within that group, and uses a sequence-level ratio

$m_{k,i}$4

Agent-GSPO then uses this objective to optimize communication-aware rewards in multi-agent systems, including token, turn, and repetition penalties, together with an optional dual update for a communication budget (Fan et al., 26 Oct 2025).

Turn- and segment-level extensions further refine the grouping granularity. GTPO redefines multi-turn tool-integrated reasoning as an MDP over turns, with turn-level discounted return

$m_{k,i}$5

and normalizes those returns across trajectories at each turn index (Ding et al., 18 Nov 2025). Workflow-R1 introduces GSsPO, where each Think–Action cycle is a sub-sequence $m_{k,i}$6, and the sub-sequence ratio is the geometric mean of token-level ratios within that segment (Kong et al., 1 Feb 2026). These works show that the “group” in MUPO may be a turn, a sub-sequence, a graph edge, or a heterogeneous multi-agent rollout, provided the optimization unit matches the semantics of the decision process.

6. Divergent-thinking MUPO in vision-LLMs, and terminological ambiguity

The most explicit use of MUPO as a named algorithm appears in "All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-LLMs" (Tian et al., 1 Apr 2026). That paper argues that GRPO induces diversity collapse: reasoning diversity drops sharply within the first approximately 20 steps of GRPO training, and RL-trained VLMs become deep but narrow thinkers, whereas base models remain broader and achieve stronger acc@$m_{k,i}$7 scaling when multiple samples are drawn (Tian et al., 1 Apr 2026). MUPO addresses this by sampling $m_{k,i}$8 responses per example, embedding each reasoning segment with Qwen3-Embedding-0.6B, clustering the embeddings into $m_{k,i}$9 groups by constrained $$\hat{A}_{k,i} = \frac{R_{k,i} - \text{mean}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})} {\text{std}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})}.$$0-means, and optimizing a multi-group objective

$$\hat{A}_{k,i} = \frac{R_{k,i} - \text{mean}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})} {\text{std}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})}.$$1

with load-balancing weight

$$\hat{A}_{k,i} = \frac{R_{k,i} - \text{mean}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})} {\text{std}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})}.$$2

Each sample reward is

$$\hat{A}_{k,i} = \frac{R_{k,i} - \text{mean}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})} {\text{std}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})}.$$3

and the diversity reward averages cosine distance to reasoning traces in other groups (Tian et al., 1 Apr 2026). The paper reports that MUPO-Thinker-7B reaches average Acc@1 of $$\hat{A}_{k,i} = \frac{R_{k,i} - \text{mean}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})} {\text{std}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})}.$$4 and Acc@4 of $$\hat{A}_{k,i} = \frac{R_{k,i} - \text{mean}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})} {\text{std}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})}.$$5 on mathematical benchmarks, and average Acc@1 of $$\hat{A}_{k,i} = \frac{R_{k,i} - \text{mean}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})} {\text{std}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})}.$$6 and Acc@4 of $$\hat{A}_{k,i} = \frac{R_{k,i} - \text{mean}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})} {\text{std}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})}.$$7 on MMStar, HallusionBench, and MMVet; the 3B model reaches $$\hat{A}_{k,i} = \frac{R_{k,i} - \text{mean}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})} {\text{std}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})}.$$8 Acc@1 and $$\hat{A}_{k,i} = \frac{R_{k,i} - \text{mean}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})} {\text{std}(\{R_{l,j}\mid m_{l,j}=m_{k,i}\})}.$$9 Acc@4 (Tian et al., 1 Apr 2026). An ablation over the number of groups reports $m$0 at $m$1, $m$2 at $m$3, $m$4 at $m$5, $m$6 at $m$7, and $m$8 at $m$9, making $r_i(x,y)$0 the best setting in that study (Tian et al., 1 Apr 2026).

Two misconceptions are therefore worth correcting. First, MUPO is not a single universally agreed-upon algorithm; current usage spans target-set RLHF, group-relative rollout methods, task clustering, and explicit diversity-aware VLM training. Second, MUPO is not uniformly critic-free: Hybrid GRPO-style readings retain $r_i(x,y)$1, whereas other variants remove critics entirely (Sane, 30 Jan 2025). A further terminological complication is the safe-RL paper in which MUPO stands for multimodal policy optimization, not multi-group policy optimization. There the policy is a Gaussian mixture

$r_i(x,y)$2

and bifurcated behavior is realized by selecting the Gaussian component with the highest mixing coefficient (Zou et al., 2024). This is a distinct lineage centered on policy bifurcation in non-simply connected safe sets.

Taken together, the literature indicates that MUPO is best viewed as a broad organizing idea: optimize policies not against a single monolithic signal, but against multiple structured groups whose internal statistics, geometry, or diversity carry information that a flat objective would discard. This suggests that future consolidation of the term will depend on whether the field converges on a shared mathematical core—such as grouped advantages, grouped constraints, or grouped diversity incentives—or continues to use MUPO as a flexible label for several related, but distinct, policy-optimization strategies.