GRPODropout: Less is More for Online Reinforcement Learning Rollouts

Abstract: Reinforcement learning (RL) methods such as GRPO substantially improve LLM reasoning but often suffer from policy entropy collapse: the loss of sampling diversity weakens exploration and limits further improvement. Existing methods address this issue either through algorithm-level interventions, such as reward modification and entropy/KL regularization, or through token-level reweighting. We investigate a complementary perspective: entropy collapse can also be mitigated by changing which generated rollouts contribute to policy updates. Under the same sampling budget, not all rollouts contribute positively to an update, and selectively excluding some can improve learning. To address this, we propose GRPODropout: before the standard update, we use a simple strategy that selectively removes a small number of high-probability positive-advantage rollouts and recenters the retained advantages. To motivate this design, we develop a rollout-level theoretical analysis that guides method design and threshold selection. The method changes only rollout usage, and adds negligible computational overhead. Experiments show higher accuracy than original GRPO and higher actor entropy while using fewer rollout samples for updates, illustrating "less is more." This work provides insight into RL rollout usage: removing some rollouts can improve performance. Code is available at https://github.com/hexuandeng/GRPODropout/.

• The paper proposes GRPODropout, which selectively omits high-probability positive-advantage rollouts to enhance entropy and performance in online reinforcement learning for language models, while retaining the original GRPO loss and adding minimal computational overhead.
• Across 10 benchmarks and multiple model sizes, GRPODropout shows improved accuracy by 1.55 percentage points on average over GRPO while reducing the number of update rollouts by 12.1% - it retains an average of 7.03 rollouts.
• The method's effectiveness is demonstrated by increased training and held-out MMLU-Pro entropies, despite using slightly less data.
• “According to GRPODropout, higher entropy, in this circumstance, is useful and contributes significantly to the correctness-oriented activity. This is supported by the greater semantic range of solution coverage and fewer repeat blunders.

Problem setting and central claim

“GRPODropout: Less is More for Online Reinforcement Learning Rollouts” (2610.11854) studies entropy collapse in online reinforcement learning for LLMs, focusing on GRPO-style outcome-supervised policy optimization. The paper’s central claim is deliberately counterintuitive: under a fixed rollout-sampling budget, using every sampled rollout in the policy update is not necessarily optimal. Some high-probability, positive-advantage rollouts may reinforce behavior that the current policy already represents well, reducing policy entropy without providing commensurate learning value.

The proposed method, GRPODropout, selectively removes a small subset of these rollouts before the policy update and recenters the remaining advantages. It leaves the underlying GRPO loss unchanged, does not introduce an auxiliary value model, and requires only rollout-level computations already available during training. The method therefore intervenes at the level of update data rather than modifying rewards, clipping rules, entropy bonuses, KL regularization, or token-level credit assignment.

This perspective complements prior analyses of entropy collapse that identify positive-advantage tokens and high-covariance token updates as important sources of entropy reduction (Cui et al., 28 May 2025). It also differs from token-selection methods such as Beyond 80/20, which retain only a small subset of high-entropy tokens (Wang et al., 2 Jun 2025). GRPODropout instead treats each complete generated sequence as the unit of selection and seeks to preserve the original objective while removing only rollouts predicted to have an unfavorable entropy effect.

Rollout-level entropy analysis

The theoretical development begins with a fixed prompt and a group of $G$ rollouts sampled independently from the old policy. For rollout $g$, the paper defines its surprisal as the negative log probability under the old policy and denotes its group-relative advantage by $A^{(g)}$. The analysis uses a local KL-proximal policy update in sequence space. This update is an idealized distribution-space model rather than the actual shared-parameter AdamW update used in experiments, an assumption that is important when interpreting the theoretical results.

The first proposition decomposes the first-order entropy change into additive rollout contributions. In the paper’s notation, the contribution of rollout $g$ is proportional to

$A^{(g)}\left(s_g-H(\pi_{\mathrm{old}})\right),$

where $s_g$ is rollout surprisal and $H(\pi_{\mathrm{old}})$ is the old-policy entropy. Consequently, two classes of rollouts have negative first-order entropy contributions:

• positive-advantage rollouts with probability higher than the policy average;
• negative-advantage rollouts with probability lower than the policy average.

Conversely, low-probability positive rollouts and high-probability negative rollouts tend to increase entropy under the local model. The interpretation is not that high-probability positive rollouts are incorrect or useless in an absolute sense. Rather, reinforcing a positive trajectory that is already highly probable may provide less exploratory information than reinforcing a less probable successful trajectory.

Under the zero-sum advantage condition used by GRPO, the group-level entropy derivative becomes the empirical covariance between advantage and surprisal:

$$\left.\frac{dH}{d\eta}\right|_{\eta=0} = \operatorname{Cov}(A^{(g)},s_g).$$

This gives a compact criterion: positive correlation between advantage and surprisal corresponds to entropy increase, whereas negative correlation corresponds to entropy decrease. Since high-probability sequences have lower surprisal, a policy that assigns positive advantages preferentially to likely sequences tends toward negative advantage–surprisal covariance and therefore entropy reduction.

The result is conditional and local. It is exact for the specified KL-proximal sequence-distribution model with fixed samples and advantages, but it is not an entropy guarantee for a neural-network optimizer, clipped surrogate, finite update, or reference-regularized objective. The paper explicitly treats the analysis as motivation for algorithm design rather than a complete explanation of training dynamics.

From entropy contributions to rollout filtering

The initial filtering rule estimates each rollout’s entropy contribution using the group mean surprisal as a Monte Carlo estimate of policy entropy. It then greedily removes rollouts with the most negative estimated contributions until the retained set has nonnegative estimated entropy derivative. The selection problem is formulated to maximize the number of retained rollouts subject to this covariance constraint.

In implementation, the paper length-normalizes rollout log probabilities. This is a practical modification because raw sequence surprisal scales directly with response length, whereas the theoretical analysis uses complete-sequence probabilities. The implementation therefore operates on mean token surprisal and should be understood as a proxy rather than a direct evaluation of the theoretical sequence-level entropy.

The method does not retain this initial rule unchanged. The authors further compare positive- and negative-advantage deletion under an independent-logit gradient model with a shared learning rate. When a high-probability positive rollout and a low-probability negative rollout have equal negative entropy contributions under the proximal analysis, the high-probability positive rollout has a larger entropy-decreasing effect under the shared-logit update by a factor equal to their probability ratio. This proposition supports preferentially deleting positive-advantage rollouts.

The comparison is subject to strong conditions: zero-sum advantages, independent sequence logits, equal first-order proximal contributions, and a first-order update. It does not establish that every positive rollout is more harmful than every negative rollout, nor does it directly characterize neural-network updates. The paper uses it as a conditional rationale for a conservative design choice: delete only positive-advantage candidates while retaining negative-advantage rollouts.

Advantage recentering and the complete procedure

Deleting positive-advantage rollouts disrupts the original zero-centered group structure. If the retained advantages are used without correction, the resulting update can induce a normalization shift on unsampled or zero-advantage sequences. In the paper’s auxiliary distribution-space analysis, every zero-advantage sequence receives the same first-order log-probability offset, regardless of its similarity to the retained rollouts. This is an undesirable consequence of normalization rather than a meaningful learning signal.

GRPODropout therefore recenters the retained advantages using old-policy probability weights. If $\mathcal I_{\mathrm{keep}}$ is the retained set, the baseline subtracted from each retained advantage is proportional to the old-policy-weighted mean of the retained advantages. In the implementation, the weights are derived from length-normalized rollout probabilities. This choice forces the retained score to have zero old-policy-weighted mean, cancelling the first-order normalization drift in the auxiliary proximal model.

The algorithm can be summarized as follows:

1. Sample the usual group of rollouts and compute rewards, native advantages, and old-policy surprisals.
2. Estimate rollout-level entropy contributions using the original group.
3. Filter candidate rollouts according to the estimated entropy criterion.
4. Restore all excluded rollouts with nonpositive advantages, thereby restricting deletion to positive-advantage rollouts.
5. Recenter the retained advantages using probability-weighted averaging.
6. Apply the original GRPO loss to the retained rollouts.

Figure 1: GRPODropout filters selected positive-advantage rollouts, recenters retained advantages with old-policy probability weights, and otherwise preserves the GRPO update.

The design has two notable properties. First, it changes the effective update dataset without changing the formal policy objective. Second, it does not attempt to reconstruct an unbiased sample from the original rollout distribution after biased deletion. Instead, recentering is targeted at cancelling the specific normalization drift induced by the retained-score construction.

Experimental design

The experiments use Qwen3 models with 1.7B, 4B, and 8B parameters. All models are trained for one epoch on DAPO-17K mathematics data, with 128 prompts per batch and eight sampled rollouts per prompt. The training configuration uses temperature $1.0$, nucleus sampling with $g$0, AdamW with learning rate $g$1, weight decay $g$2, clipping width $g$3, and a maximum response length of 8,192 tokens. The GRPO baseline uses all eight rollouts and no additional entropy bonus or reference-policy KL regularization.

The evaluation covers seven mathematical benchmarks and three general QA benchmarks. Small mathematical test sets use Mean@32, while the remaining benchmarks use Mean@1. Entropy is measured as token-mean conditional entropy over valid response tokens, avoiding the direct response-length dependence of sequence entropy.

The comparison includes original GRPO, DAPO, Clip-Cov, KL-Cov, and Beyond 80/20. These baselines span clipping-based exploration, token masking, and token-level KL intervention. GRPODropout is therefore tested against both the unmodified objective and several entropy-aware alternatives.

Main accuracy and entropy results

Across the three model sizes and ten benchmarks, GRPODropout improves the average score over original GRPO by 1.55 percentage points on average. It also exceeds KL-Cov, the strongest entropy-aware baseline by average score in the reported comparison, by 0.11 points on average. The improvement is not uniform at every model scale: GRPODropout is below KL-Cov on Qwen3-1.7B and Qwen3-8B in the aggregate score, while it is competitive on Qwen3-4B. This scale-dependent result qualifies the broader claim that rollout deletion is consistently superior to existing entropy-aware methods.

Backbone	GRPO average	GRPODropout average	Difference
Qwen3-1.7B	54.13	55.01	+0.88
Qwen3-4B	63.03	65.85	+2.82
Qwen3-8B	66.33	67.29	+0.96

The strongest efficiency result concerns the number of rollouts actually used in updates. Although each prompt produces eight rollouts, GRPODropout retains an average of 7.03 rollouts, corresponding to 12.1% fewer update rollouts than GRPO, after excluding groups with all-zero advantages. The reduction is modest, which is consistent with the method’s stated objective of minimizing disruption to the training distribution.

Despite this small deletion rate, the reported final actor entropy is substantially higher. For Qwen3-4B, training-set entropy is 446.1% higher than GRPO at the end of training, and held-out MMLU-Pro entropy is 256.0% higher. These results support the paper’s central empirical claim that a small amount of rollout-level filtering can strongly alter the entropy trajectory.

Figure 2: GRPODropout maintains higher training and held-out entropy while reducing the number of rollouts contributing to each update.

The implication is not simply that higher entropy is always preferable. DAPO, for example, achieves relatively high entropy and reasoning-path diversity in some conditions without obtaining the highest Pass@32. The paper’s evidence instead supports a more specific conclusion: GRPODropout increases entropy while preserving, and often improving, correctness-oriented performance, suggesting that its retained entropy is not merely undirected stochasticity.

Ablation evidence for deletion and recentering

The ablations isolate deletion direction, recentering, and probability weighting. The results strongly support the paper’s asymmetric filtering rule. With probability-weighted recentering enabled, deleting only positive-advantage rollouts generally outperforms deleting only negative-advantage rollouts in both entropy and accuracy. Deleting both types produces the largest entropy gains in some settings but slightly reduces accuracy, indicating that aggressive removal can discard useful training signal.

The most severe result concerns deletion without recentering. On Qwen3-1.7B, deleting positive rollouts without recentering reduces the average benchmark score from 54.13 for GRPO to 28.81. On Qwen3-4B, the corresponding score falls to 16.00. These configurations exhibit training collapse despite their high entropy in some cases. The result demonstrates that rollout deletion alone is not a sufficient intervention; the advantage correction is structurally necessary for the tested procedure.

Recentering with probability weighting has a model-dependent effect. On Qwen3-4B, probability weighting improves the average score by only 0.03 points over equal-weight recentering. On Qwen3-1.7B, removing probability weighting lowers the average score by 2.82 points. Thus, the theoretical weighting rule is not uniformly dominant in the reported experiments, but it can materially improve stability in smaller models.

These ablations also constrain the interpretation of the method. The performance gain cannot be attributed solely to lower update density or higher entropy. The particular interaction among positive-only deletion, retained-group recentering, and probability weighting is essential.

Accuracy, incorrect-answer dispersion, and reasoning diversity

The paper evaluates whether increased entropy corresponds to useful exploration rather than indiscriminate output variation. On Qwen3-4B, GRPODropout achieves the best or second-best mean on all eight reported metrics combining Pass@32, incorrect-answer dispersion, reasoning-path distance, Vendi diversity, and final actor entropy. Its overall mean rank is 1.63, compared with 2.63 for Clip-Cov and 3.25 for KL-Cov.

GRPODropout obtains Pass@32 of 75.21, second only to Clip-Cov’s 76.88, while maintaining the highest reported entropy at 0.2291. It also reduces the number of distinct incorrect final answers relative to GRPO: $g$4 decreases from 1.1846 to 0.4923, and $g$5 decreases from 7.6216 to 4.2432. At the same time, it improves reasoning-path diversity on both question subsets relative to GRPO.

Method	Pass@32	$g$6	$g$7	Entropy	Mean rank
GRPO	63.54	1.1846	7.6216	0.0448	5.63
DAPO	69.38	1.6154	8.0811	0.1796	3.88
Clip-Cov	76.88	0.7692	5.2162	0.0900	2.63
KL-Cov	72.71	0.4615	3.0000	0.1509	3.25
Beyond 80/20	73.54	1.1231	7.8378	0.1671	4.00
GRPODropout	75.21	0.4923	4.2432	0.2291	1.63

The authors summarize this as a favorable correctness–diversity trade-off: higher entropy and broader semantic reasoning paths coexist with greater solution coverage and fewer recurring incorrect answers. The conclusion is appropriately narrower than a general claim that entropy itself causes better reasoning. The measurements show that, in this experimental setting, GRPODropout’s entropy increase is associated with useful diversity rather than merely increased error variability.

The diversity analysis relies on an external GLM-5.2 summarizer to compress reasoning paths before embedding them with bge-m3. This introduces an evaluation dependency: the measured semantic diversity reflects both the generated responses and the summarizer’s ability to identify the final reasoning path. The paper’s metrics therefore provide informative evidence about path diversity but are not direct, model-independent measurements of latent exploration.

Limitations and open questions

The theoretical results depend on local update models that differ from the actual training process. The principal entropy propositions assume a KL-proximal update in sequence-distribution space, while the positive-dominance result assumes independent sequence logits and a shared learning rate. Actual GRPODropout training uses a shared neural network, clipped policy objectives, token-level loss averaging, AdamW, and length-normalized implementation statistics. The paper explicitly does not claim that its local propositions guarantee entropy preservation under these dynamics.

The method also depends on within-prompt variation across multiple rollouts. It cannot be applied directly to single-rollout training, and the paper does not evaluate very small group sizes. This leaves open whether reliable entropy-contribution estimates can be obtained when $g$8 is small, or whether selection noise would dominate the signal.

The empirical study is restricted to Qwen3 backbones, one mathematics training corpus, one epoch, and a fixed eight-rollout sampling configuration. GRPODropout does not outperform the strongest baseline at every model scale. In addition, the very large relative entropy improvements are measured against a GRPO trajectory that collapses to a low-entropy endpoint; their practical significance should therefore be interpreted alongside absolute entropy values and task accuracy rather than as a standalone measure.

Finally, the analysis leaves unresolved why a 12.1% deletion rate produces entropy changes as large as 446.1% on the training distribution. The proposed explanation identifies selection bias, positive-advantage reinforcement, and normalization drift, but the paper concedes that this theory does not fully account for the nonlinear shared-parameter training dynamics. A specific open question is whether the same selection rule remains effective when combined with reference KL regularization, alternative clipping schemes, token-level advantages, or larger rollout groups.

Conclusion

GRPODropout presents rollout selection as a distinct mechanism for controlling entropy in online RL for language-model reasoning. Its key claim is that some highly probable successful rollouts can be counterproductive when repeatedly reinforced, and that selectively omitting them can improve both exploration and accuracy. The method combines positive-advantage rollout filtering with probability-weighted advantage recentering, preserving the original GRPO loss and adding little computational overhead.

Across three model sizes and ten benchmarks, the method improves average accuracy over GRPO by 1.55 points while using 12.1% fewer retained rollouts for updates. On Qwen3-4B, it substantially increases training and held-out entropy and achieves a strong joint profile of solution coverage, incorrect-answer concentration, and reasoning-path diversity. The evidence supports rollout usage as an important algorithmic degree of freedom, while the gap between the local theory and shared-parameter training remains a central qualification of the paper’s claims.