Multi-Scale Group Relative Policy Optimization (GRPO)

• Multi-Scale GRPO is a framework that extends conventional GRPO by distributing group-relative normalization across multiple scales such as time, hierarchy, and reward channels.
• It introduces novel mechanisms like tree-based temporal grouping, per-objective normalization, and hierarchical agent separation to address credit assignment challenges in complex reinforcement learning settings.
• Empirical results show significant performance improvements and faster training across diverse applications including text-to-image generation, robotics, and multi-agent control.

Searching arXiv for the cited GRPO variants to ground the article and confirm relevant sources. arxiv_search(query="GRPO multi-scale multi-group trajectory-wise hierarchical momentum federated off-policy diffusion graph", max_results=10) arxiv_search(query="(Lyu et al., 30 Nov 2025, Chen et al., 10 Jun 2025, Hong et al., 17 Nov 2025, Ding et al., 5 Jun 2025, Cang et al., 3 Mar 2026, Bai et al., 15 Dec 2025, Sane, 30 Jan 2025, Tian et al., 17 May 2026, Salmani-Zarchi et al., 4 Jun 2026, Ichihara et al., 26 Sep 2025)", max_results=20) Multi-Scale Group Relative Policy Optimization (GRPO) denotes a family of extensions of Group Relative Policy Optimization in which the group-relative signal is no longer confined to a single prompt-level, rollout-level normalization step, but is instead distributed across multiple granularities such as denoising stages, step-versus-trajectory structure, hierarchical planner–executor agents, reward channels, model sizes, or federated client levels. The term is not standardized across the literature. Some methods are explicitly hierarchical or multi-group, notably Multi-GRPO, TGRPO, and Multi-Agent Deep Research’s M-GRPO, whereas others are better characterized as scale-aware or multi-granularity adaptations rather than formal multi-scale algorithms (Lyu et al., 30 Nov 2025, Chen et al., 10 Jun 2025, Hong et al., 17 Nov 2025, Kwon et al., 9 May 2026).

1. GRPO and the rationale for moving beyond a single scale

Standard GRPO is a critic-free policy optimization method in which, for a fixed prompt or query, a group of sampled outputs is scored and each sample receives a relative advantage obtained by centering and typically standardizing its reward within that group. In a canonical formulation, the within-group advantage is written as

$$A_k = \frac{r_k - \mathrm{mean}(r_1,\dots,r_K)}{\mathrm{std}(r_1,\dots,r_K)}.$$

This replaces a learned value baseline with a group-relative baseline and is typically embedded in a PPO-style clipped objective with optional KL regularization (Chen et al., 2 Jun 2026, Salmani-Zarchi et al., 4 Jun 2026).

The motivation for multi-scale extensions follows from a recurring limitation of single-level group normalization. When one scalar group-relative signal is broadcast uniformly across all tokens, denoising steps, graph edges, or sub-agent trajectories, the method can obscure where credit should actually be assigned. This difficulty appears in several distinct forms: early denoising steps in text-to-image generation have much higher uncertainty than late ones; robotic control requires both step-local and trajectory-global credit; vertical multi-agent systems contain nested trajectories with different action frequencies; multi-objective rewards have heterogeneous variances; and federated clients exhibit incomparable reward scales and task difficulties. Multi-scale GRPO, in its broadest sense, is the attempt to preserve GRPO’s group-relative normalization while relocating or duplicating that normalization across these additional structural levels (Lyu et al., 30 Nov 2025, Chen et al., 10 Jun 2025, Hong et al., 17 Nov 2025, Chen et al., 2 Jun 2026).

A theoretical backdrop is provided by the observation that the GRPO policy gradient is a U-statistic. That analysis yields finite-sample mean squared error characterizations, asymptotic equivalence to an oracle policy gradient algorithm with access to a value function, and a universal scaling law for choosing group size (Zhou et al., 1 Mar 2026). A plausible implication is that multi-scale GRPO is not only a matter of inventing new grouping axes, but also of deciding which group size is appropriate at each axis.

2. Temporal multi-scale GRPO: from whole trajectories to segments and steps

The most explicit temporal multi-scale formulation in the cited literature is Multi-GRPO for text-to-image generation. Standard GRPO in diffusion-style alignment assigns one trajectory-level advantage

$$\hat A^i = \frac{R_i-\operatorname{mean}(\{R_k\}_{k=1}^G)}{\operatorname{std}(\{R_k\}_{k=1}^G)}$$

to every denoising step of trajectory $i$. Multi-GRPO replaces this with two orthogonal grouping mechanisms. First, it introduces tree-based trajectories with branching schedule

$B=\{b_1:B_1,\; b_2:B_2,\; \dots,\; b_K:B_K\},$

and evaluates early denoising states by averaging descendant leaf rewards,

$R_j^n = \frac{1}{|D_j^n|} \sum_{l\in D_j^n} R_0^l.$

Advantages are then normalized within temporal groups at branching boundaries,

$$A_{b_{k+1}^n} = \frac{ R_{b_{k+1}^n}-\mu_{b_{k+1}} }{ \sigma_{b_{k+1}} }.$$

Second, it performs reward-based grouping for multi-objective alignment by computing per-reward normalized advantages

$$\hat A_i^m = \frac{R_i^m-\mu_m}{\sigma_m}, \qquad \hat A_i = \frac{1}{M}\sum_{m=1}^M w_m \hat A_i^m.$$

This is a genuine multi-group, and in the temporal component a natural multi-scale, extension of GRPO because it redistributes credit across coarse and fine phases of denoising instead of collapsing the trajectory to one scalar (Lyu et al., 30 Nov 2025).

TGRPO brings the same intuition into online reinforcement learning for Vision-Language-Action models. Its central claim is that robotics requires both local and global temporal credit. It therefore defines a step-level relative advantage

$$S_{i,t} = \frac{ R_{i,t} - \frac{1}{n}\sum_{i=1}^{n} R_{i,t} }{ \sqrt{ \frac{1}{n-1}\sum_{i=1}^{n}\left( R_{i,t} - \frac{1}{n}\sum_{i=1}^{n} R_{i,t} \right)^2 } },$$

a trajectory-level relative advantage

$$T_i = \frac{ R_i - \frac{1}{n}\sum_{i=1}^{n} R_i }{ \sqrt{ \frac{1}{n-1}\sum_{i=1}^{n}\left( R_i - \frac{1}{n}\sum_{i=1}^{n} R_i \right)^2 } },$$

and a fused signal

$Adv_{i,t} = \alpha_1 S_{i,t} + \alpha_2 T_i.$

This two-level construction is explicitly intended to overcome cases in which many trajectories share the same instantaneous reward but differ in long-horizon task potential (Chen et al., 10 Jun 2025).

Multi-Layer GRPO is staged rather than explicitly multi-scale, but it moves in the same direction. It applies standard GRPO in a first layer to generate an initial response, then feeds the original query together with that first-layer response into a second-layer GRPO process trained to identify and correct errors. The method thereby creates a self-correction loop and what the paper calls implicit process-level supervision without an explicit dense process reward model (Ding et al., 5 Jun 2025). This suggests a staged, coarse-to-refine reading of multi-scale GRPO even though the paper does not formalize multiple temporal resolutions.

3. Hierarchical and structural scales: agents, graphs, and models

A second major interpretation of multi-scale GRPO is hierarchical control. In “Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO,” the system is explicitly vertical: a main agent $$\hat A^i = \frac{R_i-\operatorname{mean}(\{R_k\}_{k=1}^G)}{\operatorname{std}(\{R_k\}_{k=1}^G)}$$0 plans and delegates, while sub-agents $$\hat A^i = \frac{R_i-\operatorname{mean}(\{R_k\}_{k=1}^G)}{\operatorname{std}(\{R_k\}_{k=1}^G)}$$1 execute tool-using subtasks under a call–return protocol. Main-agent and sub-agent trajectories have different lengths and different invocation frequencies, so the method computes separate group-relative statistics at each level: $$\hat A^i = \frac{R_i-\operatorname{mean}(\{R_k\}_{k=1}^G)}{\operatorname{std}(\{R_k\}_{k=1}^G)}$$2 Because a planner rollout can contain a variable number $$\hat A^i = \frac{R_i-\operatorname{mean}(\{R_k\}_{k=1}^G)}{\operatorname{std}(\{R_k\}_{k=1}^G)}$$3 of sub-agent trajectories, the paper also introduces a trajectory-alignment scheme that duplicates or drops sub-trajectories to enforce a fixed target count $$\hat A^i = \frac{R_i-\operatorname{mean}(\{R_k\}_{k=1}^G)}{\operatorname{std}(\{R_k\}_{k=1}^G)}$$4. This is a clear hierarchical extension of GRPO across nested rollout scales (Hong et al., 17 Nov 2025).

Graph-GRPO occupies a different position. It is explicitly described as a graph-structured, edge-credit version of GRPO for multi-agent communication topology learning, not as a formal multi-scale or hierarchical method. The policy emits a DAG-masked adjacency-probability matrix,

$$\hat A^i = \frac{R_i-\operatorname{mean}(\{R_k\}_{k=1}^G)}{\operatorname{std}(\{R_k\}_{k=1}^G)}$$5

samples full communication graphs edgewise, and then computes an edge-level conditional success rate

$$\hat A^i = \frac{R_i-\operatorname{mean}(\{R_k\}_{k=1}^G)}{\operatorname{std}(\{R_k\}_{k=1}^G)}$$6

followed by normalized edge utility

$$\hat A^i = \frac{R_i-\operatorname{mean}(\{R_k\}_{k=1}^G)}{\operatorname{std}(\{R_k\}_{k=1}^G)}$$7

The method therefore projects graph-level reward down to edge-level relative advantages. This is better described as multi-granularity credit assignment than formal multi-scale optimization (Cang et al., 3 Mar 2026).

CoDistill-GRPO provides yet another structural reading: model-scale coupling. It jointly trains a large model $$\hat A^i = \frac{R_i-\operatorname{mean}(\{R_k\}_{k=1}^G)}{\operatorname{std}(\{R_k\}_{k=1}^G)}$$8 and a small model $$\hat A^i = \frac{R_i-\operatorname{mean}(\{R_k\}_{k=1}^G)}{\operatorname{std}(\{R_k\}_{k=1}^G)}$$9. The small model’s effective reward augments sparse task reward with an on-policy knowledge-distillation term,

$i$0

while the large model trains on small-model rollouts using token-level importance ratios

$i$1

This is a concrete two-scale GRPO recipe in which scale refers to model capacity rather than time or hierarchy (Kwon et al., 9 May 2026).

4. Reward-scale, temperature-scale, and optimization-timescale extensions

Several GRPO variants address scale mismatch in the reward or training dynamics rather than in explicit hierarchy.

MO-GRPO targets multi-objective settings where one reward component has much larger variance than others. Standard GRPO aggregates rewards first and normalizes once,

$i$2

which the paper shows biases updates toward high-variance objectives. MO-GRPO instead normalizes each reward separately and then sums,

$i$3

The method therefore addresses heterogeneous reward scales directly, and can reasonably be read as a scale-normalized multi-objective extension of GRPO (Ichihara et al., 26 Sep 2025).

MDP-GRPO studies a different reward pathology: discrete, low-dispersion verifier rewards in multi-constraint instruction following. It identifies low-variance amplification, mean-centering blindness, and zero-variance collapse, then proposes a four-part stabilization recipe: multi-temperature sampling, dual-anchor advantages, prospect-theoretic shaping, and asymmetric KL regularization. Its dual-anchor advantage mixes group-relative and goal-aware references,

$i$4

where

$i$5

This is not a hierarchical multi-scale algorithm, but it is a multi-reference, multi-temperature GRPO variant operating across reward and sampling scales (Salmani-Zarchi et al., 4 Jun 2026).

M-GRPO, the momentum-anchored self-supervised RL method, is explicitly not a multi-scale method. Its contribution is a slowly evolving EMA momentum policy,

$i$6

plus IQR-based entropy filtering before majority-vote pseudo-labeling. Its relevance to multi-scale GRPO is therefore contrastive: it extends GRPO along the axis of training stability, not along formal structural scales (Bai et al., 15 Dec 2025).

Mu-GRPO can be interpreted more plausibly as a multi-timescale reformulation. It separates training into a small number of large sequential generation–optimization stages, with

$i$7

and successfully operates in a high-staleness regime such as $i$8. The policy is refreshed only at stage boundaries, while many optimizer updates occur within each stage. This produces a fast optimization timescale, a slower rollout-refresh timescale, and a token-level negative-advantage veto mechanism for stale off-support suffixes. The paper does not market this as multi-scale GRPO, but a multi-timescale reading is technically well motivated (Tian et al., 17 May 2026).

5. Representative variants and empirical behavior

The literature contains both explicit multi-scale variants and closely related scale-aware adaptations. The following summary organizes the main variants by the scale axis they modify.

Variant	Scale axis	Key mechanism
Multi-GRPO	Denoising time and reward channels	Tree-based temporal groups and per-reward grouping
TGRPO	Step vs trajectory	$i$9
M-GRPO (multi-agent deep research)	Planner vs executor hierarchy	Separate GRPO objectives with trajectory alignment
CoDistill-GRPO	Small vs large model	KD-shaped reward for small model; importance-weighted large update
MO-GRPO	Reward variance scale	Per-objective normalization before aggregation
MDP-GRPO	Reward dispersion and temperature	Multi-temperature sampling and dual-anchor advantages
Mu-GRPO	Rollout-refresh vs optimization timescale	Staged high-staleness training

Empirically, the benefits of multi-scale or scale-aware GRPO are reported across diverse domains. Multi-GRPO improves FLUX.1-Dev on PickScore-25k from PickScore $B=\{b_1:B_1,\; b_2:B_2,\; \dots,\; b_K:B_K\},$0 to $B=\{b_1:B_1,\; b_2:B_2,\; \dots,\; b_K:B_K\},$1, and on OCR-Color-10 it raises $B=\{b_1:B_1,\; b_2:B_2,\; \dots,\; b_K:B_K\},$2 from $B=\{b_1:B_1,\; b_2:B_2,\; \dots,\; b_K:B_K\},$3 under naive-sum Flow-GRPO to $B=\{b_1:B_1,\; b_2:B_2,\; \dots,\; b_K:B_K\},$4 while keeping strong OCR and color rewards (Lyu et al., 30 Nov 2025). TGRPO raises average success on ten LIBERO-Object tasks to $B=\{b_1:B_1,\; b_2:B_2,\; \dots,\; b_K:B_K\},$5, compared with $B=\{b_1:B_1,\; b_2:B_2,\; \dots,\; b_K:B_K\},$6 for SFT and $B=\{b_1:B_1,\; b_2:B_2,\; \dots,\; b_K:B_K\},$7 for PPO; the ablation $B=\{b_1:B_1,\; b_2:B_2,\; \dots,\; b_K:B_K\},$8 drops to $B=\{b_1:B_1,\; b_2:B_2,\; \dots,\; b_K:B_K\},$9, while $R_j^n = \frac{1}{|D_j^n|} \sum_{l\in D_j^n} R_0^l.$0 achieves $R_j^n = \frac{1}{|D_j^n|} \sum_{l\in D_j^n} R_0^l.$1, showing that fused multi-scale credit outperforms either scale alone (Chen et al., 10 Jun 2025).

Graph-GRPO reports $R_j^n = \frac{1}{|D_j^n|} \sum_{l\in D_j^n} R_0^l.$2 average across MMLU, GSM8K, AQuA, MultiArith, SVAMP, and HumanEval, compared with $R_j^n = \frac{1}{|D_j^n|} \sum_{l\in D_j^n} R_0^l.$3 for EIB-LEARNER, and its edge-level GRPO variant exceeds a graph-level GRPO baseline by $R_j^n = \frac{1}{|D_j^n|} \sum_{l\in D_j^n} R_0^l.$4 average points on selected tasks (Cang et al., 3 Mar 2026). MDP-GRPO improves strict constraint satisfaction by up to $R_j^n = \frac{1}{|D_j^n|} \sum_{l\in D_j^n} R_0^l.$5 on Llama-3.2-3B and remains effective even at small group size $R_j^n = \frac{1}{|D_j^n|} \sum_{l\in D_j^n} R_0^l.$6 through multi-temperature sampling (Salmani-Zarchi et al., 4 Jun 2026). CoDistill-GRPO raises Qwen2.5-Math-1.5B on Minerva by over $R_j^n = \frac{1}{|D_j^n|} \sum_{l\in D_j^n} R_0^l.$7 percentage points over the base model and by $R_j^n = \frac{1}{|D_j^n|} \sum_{l\in D_j^n} R_0^l.$8 points over GRPO, while its large-model training nearly matches standard GRPO with an approximate $R_j^n = \frac{1}{|D_j^n|} \sum_{l\in D_j^n} R_0^l.$9 speedup (Kwon et al., 9 May 2026). Mu-GRPO matches or exceeds standard GRPO on four of five LLMs while reducing rollout-generation time by $$A_{b_{k+1}^n} = \frac{ R_{b_{k+1}^n}-\mu_{b_{k+1}} }{ \sigma_{b_{k+1}} }.$$0 on average and total wall-clock time by $$A_{b_{k+1}^n} = \frac{ R_{b_{k+1}^n}-\mu_{b_{k+1}} }{ \sigma_{b_{k+1}} }.$$1 on average, with a headline result of around $$A_{b_{k+1}^n} = \frac{ R_{b_{k+1}^n}-\mu_{b_{k+1}} }{ \sigma_{b_{k+1}} }.$$2 wall-clock speedup on DeepSeek-7B (Tian et al., 17 May 2026). Finally, the two-rollout analysis in “It Takes Two: Your GRPO Is Secretly DPO” shows that 2-GRPO achieves performance on par with 16-GRPO despite using only $$A_{b_{k+1}^n} = \frac{ R_{b_{k+1}^n}-\mu_{b_{k+1}} }{ \sigma_{b_{k+1}} }.$$3 of the rollouts and reducing training time by over $$A_{b_{k+1}^n} = \frac{ R_{b_{k+1}^n}-\mu_{b_{k+1}} }{ \sigma_{b_{k+1}} }.$$4, demonstrating that scale can refer not only to hierarchy but also to group size itself (Wu et al., 1 Oct 2025).

6. Misconceptions, boundaries, and research directions

A common misconception is that any GRPO method with more than one sampled object is automatically multi-scale. The literature is more precise. Graph-GRPO is best classified as graph-structured GRPO with edge-level credit assignment, not as formal multi-scale or hierarchical GRPO (Cang et al., 3 Mar 2026). Momentum-Anchored GRPO is a stability-oriented extension based on EMA anchoring and entropy filtering rather than a multi-scale reformulation (Bai et al., 15 Dec 2025). Hybrid GRPO is a multi-sample bridge between PPO and the paper’s description of DeepSeek GRPO; it is relevant as a foundation because it combines empirical multi-sample evaluation with value bootstrapping, but the paper itself states that it does not yet define a complete multi-scale optimization framework (Sane, 30 Jan 2025). Likewise, M$$A_{b_{k+1}^n} = \frac{ R_{b_{k+1}^n}-\mu_{b_{k+1}} }{ \sigma_{b_{k+1}} }.$$5GRPO stands for Mamba-based Multi-Agent Group Relative Policy Optimization, not multi-scale GRPO, and its core contribution lies in critic-free multi-agent PPO/GRPO with a Mamba-based temporal-relational policy backbone rather than explicit multi-scale hierarchy (Feng et al., 21 Apr 2026).

A second misconception is that larger group size is always necessary for GRPO. The contrastive reinterpretation of GRPO as DPO-like pairwise optimization shows that group size changes the effective estimator and compute profile, but not always the attainable performance frontier. This does not imply that group size is irrelevant; rather, it implies that group size itself is a scale variable whose optimal value depends on the variance–throughput tradeoff and, according to the U-statistic theory, can be analyzed systematically (Wu et al., 1 Oct 2025, Zhou et al., 1 Mar 2026).

A third boundary concerns what is meant by “scale.” In the present literature, at least five distinct meanings coexist: temporal scale, as in denoising stages or step-versus-trajectory fusion; hierarchical control scale, as in planner–executor systems; reward-scale heterogeneity, as in MO-GRPO and MDP-GRPO; model-capacity scale, as in CoDistill-GRPO; and systems or aggregation scale, as in federated client weighting or staged high-staleness training (Lyu et al., 30 Nov 2025, Chen et al., 10 Jun 2025, Kwon et al., 9 May 2026, Chen et al., 2 Jun 2026, Tian et al., 17 May 2026). This suggests that “Multi-Scale GRPO” is presently better understood as an organizing concept for a research direction than as the name of a single canonical algorithm.

The main open direction is to unify these axes into explicitly multi-level objectives. Several papers already point toward such unification. Multi-GRPO shows how a group sampled at the whole-trajectory level can produce temporally localized advantages (Lyu et al., 30 Nov 2025). TGRPO shows that step-level and trajectory-level normalization can be fused in one update rule (Chen et al., 10 Jun 2025). M-GRPO shows how distinct GRPO objectives can coexist at planner and executor levels (Hong et al., 17 Nov 2025). CoDistill-GRPO demonstrates cross-scale coupling between policies of different capacities (Kwon et al., 9 May 2026). U-statistic theory and group-size scaling laws suggest that a principled future multi-scale GRPO framework could vary group size across these levels rather than treating $$A_{b_{k+1}^n} = \frac{ R_{b_{k+1}^n}-\mu_{b_{k+1}} }{ \sigma_{b_{k+1}} }.$$6 as globally fixed (Zhou et al., 1 Mar 2026). Such a framework has not yet been formalized in the cited literature, but the components from which it would be built are already visible.