How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning

Abstract: Group Relative Policy Optimization (GRPO) has been a key driver of recent progress in reinforcement learning with verifiable rewards (RLVR) for LLMs, but it is typically trained in a low-staleness, near-on-policy regime that incurs substantial system overhead. We ask a simple question: How off-policy can GRPO be? We show that GRPO-style algorithms can tolerate substantially larger rollout staleness than previously assumed, and propose Mu-GRPO, an RL training framework that organizes training into a small number (e.g., four) of large sequential generation-optimization stages. This design induces high rollout staleness while greatly reducing rollout-optimization switching overhead. To stabilize learning under stale data, Mu-GRPO combines relaxed clipping, which preserves useful stale-rollout gradients, with negative-advantage veto, which removes destabilizing post-trigger suffix updates in negative-advantage responses. Across five LLMs and multiple math reasoning benchmarks, Mu-GRPO matches or exceeds the performance of standard GRPO while achieving around 2x speedup in wall-clock training time, establishing a substantially improved performance-efficiency trade-off for LLM reinforcement learning.

• The paper introduces Mu-GRPO, which combines relaxed importance-ratio clipping with a negative-advantage veto to stabilize GRPO at rollout staleness levels as high as μ=1024.
• The paper finds that Mu-GRPO matches or exceeds standard GRPO on 4 of 5 models, including accuracy gains from 41.8% to 48.0% on Qwen2.5-Math-7B and from 60.2% to 68.6% on DeepSeek-7B.
• The paper demonstrates practical efficiency gains, including average 1.53× lower total training time, a 2.2× speedup on DeepSeek-7B, and a 2.16× speedup for code generation without meaningful accuracy loss.

Motivation and problem statement

Reinforcement learning with verifiable rewards (RLVR) has become the dominant post-training paradigm for improving LLM reasoning, with Group Relative Policy Optimization (GRPO) as the workhorse algorithm. Although GRPO is commonly described as on-policy, practical implementations are never strictly so: rollouts are generated by a policy several updates old, and training–inference mismatch between high-throughput inference engines (vLLM, SGLang) and sharded trainers (FSDP, Megatron-LM) introduces further distribution shift. Standard GRPO operates at low rollout staleness — typically $\mu = 4$, meaning one rollout batch supports only four optimizer updates — which forces frequent weight synchronizations and GPU memory contention between inference and training engines. The paper asks how far this staleness can be pushed without sacrificing learning stability or reasoning performance, motivated by the observation that larger $\mu$ amortizes rollout generation over many more policy updates.

Diagnosis: stale rollouts are useful until a localized failure mode appears

The paper's central empirical finding is that naive high-staleness GRPO fails in a specific, diagnosable way rather than through generic off-policy collapse. At $\mu = 1024$ on Qwen2.5-Math-7B, standard clipping $[0.8, 1.2]$ clips an increasing fraction of tokens and plateaus at lower accuracy. Relaxing the clip range to $[0, 5]$ recovers strong early learning — evidence that stale rollouts retain useful gradient signal — but collapses mid-training. This "clipping dilemma" frames the design space: tight clipping is stable but discards signal; relaxed clipping exposes signal but is eventually destabilizing.

The authors localize the collapse to negative-advantage tokens. In collapsed runs, the expected importance ratio on negative-advantage tokens, $\mathbb{E}[\rho_t \mid A_t < 0]$, drops sharply, while stable runs keep it near one. Repeated suppression of behavior-sampled low-reward responses under relaxed clipping is therefore the precursor to collapse. Masking ablations then identify precisely which updates are harmful. Removing only the trigger tokens (those with $A_t < 0$ and $\rho_t < \tau_c = 10^{-4}$) still collapses; removing the post-trigger suffix stabilizes training. The mechanism is a prefix-support mismatch: once a trigger token appears at position $\kappa$, its tiny ratio enters every subsequent prefix occupancy ratio $R_{t-1}(\theta) = \prod_{j<t}\rho_j(\theta)$, so teacher-forced updates continue to modify suffix tokens conditioned on prefixes the current policy is unlikely to generate. Token-level clipping misses this because it weights each token by its local ratio alone. An appendix formalizes this via a reverse Pearson divergence bound showing that retained suffix tokens with benign local ratios ($\mu$0) can nonetheless induce arbitrarily large prefix divergence — the blow-up is compatible with non-tiny local ratios.

Method

$\mu$1-GRPO organizes training into a small number of large sequential stages (four by default). Each stage first freezes the current policy as the behavior policy $\mu$2 and generates a large static rollout dataset, storing group-normalized advantages and tokenwise behavior log-probabilities emitted during decoding (avoiding an extra forward pass). The policy is then optimized for $\mu$3 updates — on the order of thousands — against this static dataset using stored log-probabilities as anchors. Two stabilization mechanisms address the diagnosed failure mode:

• Relaxed clipping: the clip range $\mu$4 preserves useful stale-rollout gradients that standard $\mu$5 clipping would suppress.
• Negative-advantage veto (NAV): for each negative-advantage response, if any token's importance ratio falls below $\mu$6, the entire response is masked (sequence-level scope by default), recomputed per update and not differentiated through.

The system-level benefit is substantial: over 4096 model updates, standard GRPO with $\mu$7 requires 1024 rollout refreshes and weight synchronizations, whereas $\mu$8-GRPO requires four. Because rollout generation is a standalone phase, training state need not remain co-resident during inference, freeing device memory for larger rollout batches and higher decoding parallelism.

Main results

Across five models (Llama-3.2-3B-Instruct, Qwen2.5-Math-7B, Qwen3-Base-1.7B/8B, DeepSeek-7B) trained on DeepScaleR and evaluated on AMC23/24, AIME24/25, and Math500, all under an aligned wall-clock protocol on $\mu$9H200 GPUs:

Comparison	Accuracy outcome	Efficiency outcome
vs. high-staleness GRPO ($\mu = 1024$0)	Recovers accuracy loss (e.g., 41.8% → 48.0% on Qwen2.5-Math-7B; 60.2% → 68.6% on DeepSeek-7B)	Retains large-$\mu = 1024$1 efficiency
vs. standard GRPO ($\mu = 1024$2)	Matches or improves average accuracy on 4 of 5 models (best gain: 68.6% vs. 66.1% on DeepSeek-7B)	$\mu = 1024$3 reduced 1.82×, $\mu = 1024$4 reduced 1.53× on average
vs. M2PO ($\mu = 1024$5)	Competitive accuracy	$\mu = 1024$6 reduced 1.87×, $\mu = 1024$7 reduced 1.49×

On DeepSeek-7B specifically, $\mu = 1024$8-GRPO reaches GRPO's performance with a 2.2× wall-clock speedup. A coding experiment transfers the result beyond math: on Qwen2.5-Coder-7B trained on code_contests and evaluated on LiveCodeBench, $\mu = 1024$9-GRPO matches GRPO accuracy (38.6% vs. 39.3%) while cutting training time from 98.9h to 45.7h (2.16× speedup). In a fully asynchronous setup on Qwen2.5-Math-7B, async $[0.8, 1.2]$0-GRPO maintains comparable accuracy while reducing trainer idle ratio from 24.2% to 0.6%, indicating that the staleness tolerance carries over from serial staged execution to parallel asynchronous pipelines.

Ablations support both components. With a fixed generation budget, smaller minibatches (higher effective staleness) yield better accuracy, confirming that rollout data remain useful across many updates. With a fixed optimization budget, performance is stable from $[0.8, 1.2]$1 to $[0.8, 1.2]$2 but degrades at $[0.8, 1.2]$3 (single stage), so occasional refreshes remain necessary. NAV vetoes only a small fraction of tokens (0.17%–4.03% averaged across stages, concentrated in early stages), and the sequence-level scope slightly outperforms narrower scopes. Sensitivity studies show $[0.8, 1.2]$4 is optimal among $[0.8, 1.2]$5, and removing the clip upper bound entirely degrades accuracy sharply (43.6% vs. 48.0%), so some upper-bound control remains necessary. Notably, at low staleness ($[0.8, 1.2]$6), the $[0.8, 1.2]$7-GRPO update rule slightly outperforms standard GRPO (47.7% vs. 47.3%) with negligible NAV activity, suggesting the standard clip range is overly conservative even near-on-policy — though the authors caution this does not constitute an on-policy versus off-policy comparison.

Limitations and open questions

The paper concedes several boundaries. All experiments use models up to 8B parameters and verifiable binary rewards on math and coding tasks; extending high-staleness training to broader domains and larger models remains future work. Performance at extreme staleness ($[0.8, 1.2]$8, single stage) degrades, indicating that occasional refreshes are still load-bearing and that the staleness ceiling is not fully characterized. The NAV threshold $[0.8, 1.2]$9 and clip upper bound, while shown robust within tested ranges, are hand-set hyperparameters whose interaction with model scale is unexplored. The prefix-support analysis assumes a finite decoding horizon and fixed position weights, and the theoretical result establishes a lower bound on divergence rather than a sufficient condition for stability. Finally, the asynchronous evaluation caps maximum lag to match the synchronous setting; whether tolerance extends to unbounded lag in production-scale heterogeneous systems is left open.

Conclusion

This paper demonstrates that GRPO's low-staleness operating point is a system-driven convention rather than an algorithmic necessity. By diagnosing high-staleness collapse as localized negative-advantage suffix drift behind low-support prefixes, $[0, 5]$0-GRPO combines relaxed clipping with a sparse trigger-based veto to stabilize optimization at $[0, 5]$1, converting staleness tolerance into roughly 2× wall-clock speedups at equal or better accuracy across five models and two task domains. The result reframes rollout freshness as one tunable component of the RLVR efficiency budget rather than a fixed constraint.