Multi-Step GRPO: Group-Relative Policy Optimization

• MS-GRPO is a framework that extends group-relative policy optimization to multi-step decision-making scenarios across trajectories and episodes.
• It leverages intra-group normalization to compute a single trajectory-level advantage, reducing variance and eliminating the need for external baselines.
• The approach is applied in domains like neural combinatorial optimization and sequential language agents, offering improved training stability and efficiency.

Multi-Step Group-Relative Policy Optimization (MS-GRPO) denotes a class of GRPO-based training schemes that extend group-relative policy optimization from single-turn outputs to trajectories, episodes, or repeated optimization steps. The term is not standardized across the literature. In neural combinatorial optimization, it refers to applying one group-normalized trajectory advantage across all autoregressive decisions in a sampled solution (Sepúlveda et al., 9 Jun 2026). In sequential decision-making with language agents, it denotes assigning the full cumulative episode reward to every token emitted across all steps of an episode (Dilkes et al., 14 Aug 2025). Other works use “multi-step” to describe iterative application of GRPO across training iterations, while several domain-specific GRPO variants explicitly state that they do not introduce an algorithm called MS-GRPO (Zhou et al., 1 Mar 2026, Cang et al., 3 Mar 2026).

1. Terminology and definitional scope

The expression “MS-GRPO” is best understood as a naming family rather than a single canonical algorithm. Different papers attach the label to different loci of temporality: trajectory generation, episode-level reinforcement, outer-loop optimization, or multi-stage correction. Several adjacent methods are technically close to MS-GRPO while explicitly using different names.

Usage	Meaning of “multi-step”	Representative source
Explicit MS-GRPO	One trajectory-level advantage is applied across all autoregressive steps	(Sepúlveda et al., 9 Jun 2026)
Explicit MS-GRPO	Full cumulative episode reward is attributed to every episode step and token	(Dilkes et al., 14 Aug 2025)
Iterative GRPO	GRPO is viewed as a multi-step meta-algorithm over training iterations	(Zhou et al., 1 Mar 2026)
Not MS-GRPO by name	Multiple communication rounds occur under a fixed topology; no n-step returns or GAE	(Cang et al., 3 Mar 2026)
Related variants	Two-layer self-correction or tree-based temporal grouping rather than a single MS-GRPO definition	(Ding et al., 5 Jun 2025, Lyu et al., 30 Nov 2025)

This suggests that the term functions as an umbrella label for “GRPO under temporally extended decision structure,” not as a universally fixed surrogate objective.

2. Core GRPO mechanism in multi-step settings

Across the explicit MS-GRPO formulations, the central mechanism is group-relative normalization. For one instance or prompt, a policy samples a small group of trajectories, computes returns within that group, and replaces an external rollout baseline or critic with an intra-group normalized advantage. In the neural combinatorial optimization formulation, the group statistics are

$$\mu_G = \frac{1}{G}\sum_{i=1}^G R(\tau_i), \qquad \sigma_G^2 = \frac{1}{G}\sum_{i=1}^G (R(\tau_i)-\mu_G)^2,$$

and the trajectory advantage is

$$A_i = \frac{R(\tau_i)-\mu_G}{\max(\sigma_G,\epsilon)}.$$

The corresponding unclipped group-relative policy gradient is

$$g(\theta)=\mathbb{E}_{x}\mathbb{E}_{\tau_1,\ldots,\tau_G\sim\pi_\theta(\cdot|x)} \left[ \frac{1}{G}\sum_{i=1}^{G} A_i \sum_{t=1}^{T}\nabla_\theta \log \pi_\theta(a_t^{(i)}|s_t^{(i)}) \right].$$

Its defining property is that a single trajectory-level scalar $A_i$ multiplies the sum of per-step log-probability gradients, so the algorithm is multi-step in sequence structure but not in the sense of per-step temporal-difference credit assignment (Sepúlveda et al., 9 Jun 2026).

The language-agent formulation preserves the same pattern at episode scale. There, each episode $j$ receives a composite reward

$C_j = \sum_t (r_{j,t} + \Phi_{j,t}),$

combining environment reward and task-specific shaping reward, and the group-relative episode advantage is

$$A_j = \frac{C_j - \text{mean}(\{C_1,\dots,C_G\})}{\text{std}(\{C_1,\dots,C_G\})}.$$

That scalar then weights every token emitted in every action-generation step of the episode through a PPO-style clipped objective with a KL penalty to a reference model (Dilkes et al., 14 Aug 2025).

Relative to PPO, these formulations remove an explicit learned value baseline. Relative to plain REINFORCE, they add within-group centering, variance normalization, clipping, and, in several implementations, KL regularization. Relative to DPO-style objectives, they optimize outcome rewards over sequential interactions rather than static single-turn preferences (Sepúlveda et al., 9 Jun 2026, Dilkes et al., 14 Aug 2025).

3. What “multi-step” means for credit assignment

A recurring source of confusion is that “multi-step” does not necessarily mean n-step returns, generalized advantage estimation, or temporal-difference bootstrapping. In the neural combinatorial optimization formulation, the advantage is computed once per trajectory and then applied uniformly across all autoregressive decoding steps. The paper states explicitly that there is “no per-step advantage or temporal credit decomposition” (Sepúlveda et al., 9 Jun 2026).

The same non-temporal-broadcast structure appears in several neighboring GRPO applications. In Graph-GRPO, a sampled communication graph is executed for up to $T=3$ communication rounds, but training uses a single final binary reward per graph and computes edge-level advantages from group-relative conditional success rates. The paper explicitly states that Graph-GRPO “does not use multi-step returns (e.g., n-step returns $G_t^{(n)}$) or generalized advantage estimation (GAE) across rounds”; the multi-step aspect refers only to multi-round agent communication under a fixed sampled topology (Cang et al., 3 Mar 2026). In GRPO for speech recognition, rewards are also sequence-level, and the loss accumulates across decoding tokens, but there is “no discounted return,” “no value function,” and “no temporal-difference bootstrapping” (Shivakumar et al., 2 Sep 2025).

A different answer to the credit-assignment problem appears in Multi-GRPO for text-to-image generation. Instead of broadcasting one terminal advantage to all denoising steps, it introduces tree-based trajectories. Early denoising states are evaluated by the average reward of their descendant leaves, and each segment between branching points receives a segment-specific normalized advantage. This replaces shared trajectory-wide credit with temporal groups anchored at branching nodes, while a separate reward-based grouping normalizes each reward stream independently before aggregation (Lyu et al., 30 Nov 2025).

The literature therefore contains at least three distinct temporal semantics: broadcasted trajectory advantage, episode-wide return attribution, and temporally localized advantages derived from structured branching. The shared element is group-relative normalization; the object being normalized varies.

4. Representative formulations and reported results

The empirical record for MS-GRPO-style methods is broad, spanning routing, language agents, molecular generation, reasoning, and text-to-image alignment.

Domain	Formulation	Selected reported result
Neural combinatorial optimization (Sepúlveda et al., 9 Jun 2026)	Baseline-free MS-GRPO with $G=4$ and PPO-clip	At 100 epochs: TSP-50 5.81, TSP-100 8.20, CVRP-50 10.98, CVRP-100 16.78
Sequential LLM agents (Dilkes et al., 14 Aug 2025)	Episode-level MS-GRPO with cumulative reward attribution	FrozenLake-NotSlippery: $$A_i = \frac{R(\tau_i)-\mu_G}{\max(\sigma_G,\epsilon)}.$$0 for post-trained 3B vs $$A_i = \frac{R(\tau_i)-\mu_G}{\max(\sigma_G,\epsilon)}.$$1 for 72B baseline
Molecular optimization (Javaid et al., 12 Feb 2026)	Scaffold-conditioned GRPO with trajectory-level centered rewards	Obj. Score $$A_i = \frac{R(\tau_i)-\mu_G}{\max(\sigma_G,\epsilon)}.$$2 and Success Rate $$A_i = \frac{R(\tau_i)-\mu_G}{\max(\sigma_G,\epsilon)}.$$3
Two-layer self-correction (Ding et al., 5 Jun 2025)	MGRPO as a two-step GRPO pipeline	MATH: 80.9 $$A_i = \frac{R(\tau_i)-\mu_G}{\max(\sigma_G,\epsilon)}.$$4 87.5 $$A_i = \frac{R(\tau_i)-\mu_G}{\max(\sigma_G,\epsilon)}.$$5 90.4; GSM8K: 83.4 $$A_i = \frac{R(\tau_i)-\mu_G}{\max(\sigma_G,\epsilon)}.$$6 93.4 $$A_i = \frac{R(\tau_i)-\mu_G}{\max(\sigma_G,\epsilon)}.$$7 95.6
Text-to-image alignment (Lyu et al., 30 Nov 2025)	Tree-based Multi-GRPO with temporal and reward grouping	PickScore-25k: 24.24 vs 23.65 for Flow-GRPO

The neural combinatorial optimization study is the clearest explicit MS-GRPO benchmark in an autoregressive multi-step setting. It evaluates five RL algorithms on TSP and CVRP and reports that GRPO avoids the reproducible TSP-100 collapse seen with REINFORCE, where performance degrades from cost $$A_i = \frac{R(\tau_i)-\mu_G}{\max(\sigma_G,\epsilon)}.$$8 to $$A_i = \frac{R(\tau_i)-\mu_G}{\max(\sigma_G,\epsilon)}.$$9 after warmup. At matched gradient updates, it remains within about $$g(\theta)=\mathbb{E}_{x}\mathbb{E}_{\tau_1,\ldots,\tau_G\sim\pi_\theta(\cdot|x)} \left[ \frac{1}{G}\sum_{i=1}^{G} A_i \sum_{t=1}^{T}\nabla_\theta \log \pi_\theta(a_t^{(i)}|s_t^{(i)}) \right].$$0–$$g(\theta)=\mathbb{E}_{x}\mathbb{E}_{\tau_1,\ldots,\tau_G\sim\pi_\theta(\cdot|x)} \left[ \frac{1}{G}\sum_{i=1}^{G} A_i \sum_{t=1}^{T}\nabla_\theta \log \pi_\theta(a_t^{(i)}|s_t^{(i)}) \right].$$1 of POMO on TSP-100 and CVRP-100, while eliminating the external rollout baseline (Sepúlveda et al., 9 Jun 2026).

The sequential language-agent study is the clearest explicit MS-GRPO benchmark for episodic interaction. It reports that a post-trained 3B model outperforms a 72B baseline by $$g(\theta)=\mathbb{E}_{x}\mathbb{E}_{\tau_1,\ldots,\tau_G\sim\pi_\theta(\cdot|x)} \left[ \frac{1}{G}\sum_{i=1}^{G} A_i \sum_{t=1}^{T}\nabla_\theta \log \pi_\theta(a_t^{(i)}|s_t^{(i)}) \right].$$2 on Frozen Lake, and introduces absolute-advantage-weighted episode sampling. In one ablation, generating $$g(\theta)=\mathbb{E}_{x}\mathbb{E}_{\tau_1,\ldots,\tau_G\sim\pi_\theta(\cdot|x)} \left[ \frac{1}{G}\sum_{i=1}^{G} A_i \sum_{t=1}^{T}\nabla_\theta \log \pi_\theta(a_t^{(i)}|s_t^{(i)}) \right].$$3 episodes and sampling $$g(\theta)=\mathbb{E}_{x}\mathbb{E}_{\tau_1,\ldots,\tau_G\sim\pi_\theta(\cdot|x)} \left[ \frac{1}{G}\sum_{i=1}^{G} A_i \sum_{t=1}^{T}\nabla_\theta \log \pi_\theta(a_t^{(i)}|s_t^{(i)}) \right].$$4 yields about $$g(\theta)=\mathbb{E}_{x}\mathbb{E}_{\tau_1,\ldots,\tau_G\sim\pi_\theta(\cdot|x)} \left[ \frac{1}{G}\sum_{i=1}^{G} A_i \sum_{t=1}^{T}\nabla_\theta \log \pi_\theta(a_t^{(i)}|s_t^{(i)}) \right].$$5 time savings relative to using all 100 for optimization, with similar final reward on Snake-Standard (Dilkes et al., 14 Aug 2025).

The molecular optimization study instantiates the same core idea under a different name. GRXForm uses per-starting-structure grouping, centered rewards $$g(\theta)=\mathbb{E}_{x}\mathbb{E}_{\tau_1,\ldots,\tau_G\sim\pi_\theta(\cdot|x)} \left[ \frac{1}{G}\sum_{i=1}^{G} A_i \sum_{t=1}^{T}\nabla_\theta \log \pi_\theta(a_t^{(i)}|s_t^{(i)}) \right].$$6, and one trajectory-level advantage applied across all atom-and-bond decisions. On held-out Kinase MPO scaffolds, it reports a higher objective score and success rate than its global-baseline REINFORCE ablation, directly attributing the gain to variance reduction under heterogeneous starting-structure difficulty (Javaid et al., 12 Feb 2026).

The two-layer reasoning study is not named MS-GRPO, but it is a genuine multi-step extension of GRPO in the sense of sequential optimization stages. MGRPO first trains initial reasoning, then trains self-correction conditioned on the model’s own prior reasoning trace and answer. The large gains from Acc.@t1 to Acc.@t2 on MATH, GSM8K, Minerva Math, and OlympiadBench indicate that GRPO-style grouping can be stacked across refinement stages, not only across tokens within a single rollout (Ding et al., 5 Jun 2025).

5. Statistical theory, estimator bias, and optimization pathologies

The most developed statistical account of GRPO treats its gradient as a second-order U-statistic. For a fixed prompt, the GRPO gradient estimator can be written as a symmetric order-2 U-statistic over paired sampled outputs. The first-order Hoeffding projection coincides with an oracle policy-gradient estimator that uses the prompt-conditional value as a baseline. From this, the paper derives an MSE decomposition whose leading term matches the oracle estimator and a universal scaling law for optimal group size,

$$g(\theta)=\mathbb{E}_{x}\mathbb{E}_{\tau_1,\ldots,\tau_G\sim\pi_\theta(\cdot|x)} \left[ \frac{1}{G}\sum_{i=1}^{G} A_i \sum_{t=1}^{T}\nabla_\theta \log \pi_\theta(a_t^{(i)}|s_t^{(i)}) \right].$$7

It further establishes asymptotic equivalence between GRPO and an oracle policy-gradient algorithm, and shows that the optimal group size is independent of training length and sampling budget in the population-level bound (Zhou et al., 1 Mar 2026).

A separate theoretical line studies the bias induced by PPO-style importance weighting in GRPO. It shows that standard GRPO, with token-level importance ratios against a frozen old policy, estimates the gradient at the old policy rather than the current one. An ablation that removes importance sampling entirely and performs four inner updates using the fixed old-policy gradient still achieves performance comparable to standard GRPO. The same paper proposes Trajectory-level Importance Corrected GRPO (TIC-GRPO), which replaces token-level ratios with a single trajectory-level ratio and yields an asymptotically unbiased estimator of the current-policy gradient (Pang et al., 4 Aug 2025).

Multi-step reuse of a fixed sampled group also exposes objective-level biases. Under a unified surrogate analysis of MS-GRPO-style training with $$g(\theta)=\mathbb{E}_{x}\mathbb{E}_{\tau_1,\ldots,\tau_G\sim\pi_\theta(\cdot|x)} \left[ \frac{1}{G}\sum_{i=1}^{G} A_i \sum_{t=1}^{T}\nabla_\theta \log \pi_\theta(a_t^{(i)}|s_t^{(i)}) \right].$$8 inner optimization steps, non-uniform group weights induce systematic shared-prefix gradient bias: centered advantages cancel on shared tokens only when the effective token weights are uniform. The same analysis argues that AdamW makes training dynamics largely insensitive to global reward scaling when $$g(\theta)=\mathbb{E}_{x}\mathbb{E}_{\tau_1,\ldots,\tau_G\sim\pi_\theta(\cdot|x)} \left[ \frac{1}{G}\sum_{i=1}^{G} A_i \sum_{t=1}^{T}\nabla_\theta \log \pi_\theta(a_t^{(i)}|s_t^{(i)}) \right].$$9, because scaling the advantages rescales both first and second moments. It also shows that optimizer momentum can continue moving parameters after clipping suppresses the advantage-driven gradient, pushing token-level ratios beyond the intended clipping region under repeated inner steps (Fontana et al., 8 Jan 2026).

Taken together, these results separate three issues that are often conflated: variance reduction from group normalization, bias from stale-policy optimization, and objective distortion from weighting and optimizer dynamics. MS-GRPO can improve stability relative to baseline-dependent REINFORCE, yet still inherit nontrivial estimator and optimization biases.

6. Related variants and broader design space

Several neighboring methods broaden the meaning of multi-step GRPO beyond the explicit MS-GRPO label. Graph-GRPO adapts GRPO to discrete communication-topology learning in multi-agent systems. It samples $A_i$0 DAGs per query, computes edge-conditional success rates, performs z-score normalization over observed edges, and reports best average accuracy $A_i$1, improving over EIB-LEARNER by $A_i$2 on average. Its graph-level ablation drops by $A_i$3 on average, which the paper uses to argue for fine-grained edge-level credit assignment. At the same time, it explicitly states that the method is not a temporal multi-step RL algorithm (Cang et al., 3 Mar 2026).

GRPO for speech recognition similarly remains sequence-level rather than temporally bootstrapped. The method uses groups of $A_i$4 hypotheses, rule-based rewards such as WER, EM, or total errors, PPO-style clipping, and a KL coefficient $A_i$5 when used. It reports up to $A_i$6 relative WER improvement, reduced hallucinations, and strong out-of-domain gains, but does not introduce MS-GRPO as a named method (Shivakumar et al., 2 Sep 2025).

In generative modeling, the multi-step design space is even wider. Co-GRPO reformulates masked diffusion generation as a finite-horizon MDP in which both the denoising model and the inference schedule are actions. It then applies trajectory-level GRPO with alternating co-optimization of model parameters and schedule parameters across $A_i$7 steps, improving ImageReward from 0.942 to 1.122 and HPSv2 from 28.83 to 29.37 (Zhou et al., 25 Dec 2025). Pro-GRPO addresses the cost of large groups through an “Expand-and-Prune” strategy with multi-step Optimal Variance Filtering over latent previews, reporting about $A_i$8 speedup for Pro-GRPO Standard and about $A_i$9 for Pro-GRPO Flash (Ge et al., 17 Dec 2025).

Other papers move in the opposite direction by reintroducing structure that standard GRPO omits. Hybrid GRPO retains a value function and combines multi-sample empirical rewards with bootstrapped value estimation, including n-step returns and GAE as possible multi-step components (Sane, 30 Jan 2025). A continuous-control extension constructs state-aware, group-aware multi-step advantages via clustered baselines and $j$0-returns, positioning MS-GRPO as a critic-free alternative to critic-based continuous-control methods while adding trajectory clustering and regularized policy updates (Khanda et al., 25 Jul 2025).

This suggests that MS-GRPO is evolving less as a single fixed update rule than as a design space organized around one invariant principle: rewards are normalized relative to peer trajectories sampled for the same conditioning context, and that relative signal is then propagated through a temporally extended decision process. The open questions are correspondingly structural: how to assign credit within long trajectories, how to choose group size under fixed compute, how to prevent stale-policy and optimizer-induced bias, and when critic-free relative baselines remain preferable to hybrid or critic-based alternatives.