--- title: Multi-Scale Group Relative Policy Optimization (GRPO) url: https://www.emergentmind.com/topics/multi-scale-group-relative-policy-optimization-grpo type: topic --- # Multi-Scale Group Relative Policy Optimization (GRPO) Searching arXiv for the cited GRPO variants to ground the article and confirm relevant sources. arxiv_search(query="GRPO multi-scale multi-group trajectory-wise hierarchical momentum federated off-policy diffusion graph", max_results=10) arxiv_search(query="2512.00743 2506.08440 2511.13288 2506.04746 2603.02701 2512.13070 2502.01652 2605.17570 2606.06058 2509.22047", max_results=20) Multi-Scale Group Relative Policy Optimization (GRPO) denotes a family of extensions of Group Relative Policy Optimization in which the group-relative signal is no longer confined to a single prompt-level, rollout-level normalization step, but is instead distributed across multiple granularities such as denoising stages, step-versus-trajectory structure, hierarchical planner–executor agents, reward channels, model sizes, or federated client levels. The term is not standardized across the literature. Some methods are explicitly hierarchical or multi-group, notably Multi-GRPO, TGRPO, and Multi-Agent Deep Research’s M-GRPO, whereas others are better characterized as scale-aware or multi-granularity adaptations rather than formal multi-scale algorithms [2512.00743] [2506.08440] [2511.13288] [2605.08873]. ## 1. GRPO and the rationale for moving beyond a single scale Standard GRPO is a critic-free policy optimization method in which, for a fixed prompt or query, a group of sampled outputs is scored and each sample receives a relative advantage obtained by centering and typically standardizing its reward within that group. In a canonical formulation, the within-group advantage is written as \[ A_k = \frac{r_k - \mathrm{mean}(r_1,\dots,r_K)}{\mathrm{std}(r_1,\dots,r_K)}. \] This replaces a learned value baseline with a group-relative baseline and is typically embedded in a PPO-style clipped objective with optional KL regularization [2606.03094] [2606.06058]. The motivation for multi-scale extensions follows from a recurring limitation of single-level group normalization. When one scalar group-relative signal is broadcast uniformly across all tokens, denoising steps, graph edges, or sub-agent trajectories, the method can obscure where credit should actually be assigned. This difficulty appears in several distinct forms: early denoising steps in text-to-image generation have much higher uncertainty than late ones; robotic control requires both step-local and trajectory-global credit; vertical multi-agent systems contain nested trajectories with different action frequencies; multi-objective rewards have heterogeneous variances; and federated clients exhibit incomparable reward scales and task difficulties. Multi-scale GRPO, in its broadest sense, is the attempt to preserve GRPO’s group-relative normalization while relocating or duplicating that normalization across these additional structural levels [2512.00743] [2506.08440] [2511.13288] [2606.03094]. A theoretical backdrop is provided by the observation that the GRPO policy gradient is a U-statistic. That analysis yields finite-sample mean squared error characterizations, asymptotic equivalence to an oracle policy gradient algorithm with access to a value function, and a universal scaling law for choosing group size [2603.01162]. A plausible implication is that multi-scale GRPO is not only a matter of inventing new grouping axes, but also of deciding which group size is appropriate at each axis. ## 2. Temporal multi-scale GRPO: from whole trajectories to segments and steps The most explicit temporal multi-scale formulation in the cited literature is Multi-GRPO for text-to-image generation. Standard GRPO in diffusion-style alignment assigns one trajectory-level advantage \[ \hat A^i = \frac{R_i-\operatorname{mean}(\{R_k\}_{k=1}^G)}{\operatorname{std}(\{R_k\}_{k=1}^G)} \] to every denoising step of trajectory \(i\). Multi-GRPO replaces this with two orthogonal grouping mechanisms. First, it introduces tree-based trajectories with branching schedule \[ B=\{b_1:B_1,\; b_2:B_2,\; \dots,\; b_K:B_K\}, \] and evaluates early denoising states by averaging descendant leaf rewards, \[ R_j^n = \frac{1}{|D_j^n|} \sum_{l\in D_j^n} R_0^l. \] Advantages are then normalized within temporal groups at branching boundaries, \[ A_{b_{k+1}^n} = \frac{ R_{b_{k+1}^n}-\mu_{b_{k+1}} }{ \sigma_{b_{k+1}} }. \] Second, it performs reward-based grouping for multi-objective alignment by computing per-reward normalized advantages \[ \hat A_i^m = \frac{R_i^m-\mu_m}{\sigma_m}, \qquad \hat A_i = \frac{1}{M}\sum_{m=1}^M w_m \hat A_i^m. \] This is a genuine multi-group, and in the temporal component a natural multi-scale, extension of GRPO because it redistributes credit across coarse and fine phases of denoising instead of collapsing the trajectory to one scalar [2512.00743]. TGRPO brings the same intuition into online reinforcement learning for Vision-Language-Action models. Its central claim is that robotics requires both local and global temporal credit. It therefore defines a step-level relative advantage \[ S_{i,t} = \frac{ R_{i,t} - \frac{1}{n}\sum_{i=1}^{n} R_{i,t} }{ \sqrt{ \frac{1}{n-1}\sum_{i=1}^{n}\left( R_{i,t} - \frac{1}{n}\sum_{i=1}^{n} R_{i,t} \right)^2 } }, \] a trajectory-level relative advantage \[ T_i = \frac{ R_i - \frac{1}{n}\sum_{i=1}^{n} R_i }{ \sqrt{ \frac{1}{n-1}\sum_{i=1}^{n}\left( R_i - \frac{1}{n}\sum_{i=1}^{n} R_i \right)^2 } }, \] and a fused signal \[ Adv_{i,t} = \alpha_1 S_{i,t} + \alpha_2 T_i. \] This two-level construction is explicitly intended to overcome cases in which many trajectories share the same instantaneous reward but differ in long-horizon task potential [2506.08440]. Multi-Layer GRPO is staged rather than explicitly multi-scale, but it moves in the same direction. It applies standard GRPO in a first layer to generate an initial response, then feeds the original query together with that first-layer response into a second-layer GRPO process trained to identify and correct errors. The method thereby creates a self-correction loop and what the paper calls implicit process-level supervision without an explicit dense process reward model [2506.04746]. This suggests a staged, coarse-to-refine reading of multi-scale GRPO even though the paper does not formalize multiple temporal resolutions. ## 3. Hierarchical and structural scales: agents, graphs, and models A second major interpretation of multi-scale GRPO is hierarchical control. In “Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO,” the system is explicitly vertical: a main agent \(\mathcal M\) plans and delegates, while sub-agents \(\mathcal S_i\) execute tool-using subtasks under a call–return protocol. Main-agent and sub-agent trajectories have different lengths and different invocation frequencies, so the method computes separate group-relative statistics at each level: \[ \hat{A}_{q,\mathcal{M}}^{(k)}= \frac{\mathcal{R}_{\mathcal{M}}(o^{(k)}_{\mathcal{M}})-\mu_{q,\mathcal{M}}}{\sigma_{q,\mathcal{M}}}, \qquad \hat{A}^{(k),i}_{q,\mathcal{S}}= \frac{\mathcal{R}_{\mathcal{S}}(o^{(k)}_{\mathcal{S}_i})-\mu_{q,\mathcal{S}}}{\sigma_{q,\mathcal{S}}}. \] Because a planner rollout can contain a variable number \(d_k\) of sub-agent trajectories, the paper also introduces a trajectory-alignment scheme that duplicates or drops sub-trajectories to enforce a fixed target count \(d\). This is a clear hierarchical extension of GRPO across nested rollout scales [2511.13288]. Graph-GRPO occupies a different position. It is explicitly described as a graph-structured, edge-credit version of GRPO for multi-agent communication topology learning, not as a formal multi-scale or hierarchical method. The policy emits a DAG-masked adjacency-probability matrix, \[ (P_\theta)_{ij} = \begin{cases} \sigma(h_i W h_j^T) & \text{if } j