Learning to Solve Hard Problems in RL for LLMs by Never Giving Up

Abstract: We demonstrate that training LLMs with RL does not improve performance equally across a dataset. RL shows large improvements on easy problems that an LLM is already good at solving, but small improvements on hard problems. We call this the Matthew Effect in RL for LLMs, after the phenomenon of cumulative advantage from economics and network science summarized as "the rich get richer". The naive explanation is that hard problems require more compute to find a solution. We argue that modern RL methods are exacerbating the issue by wasting too much compute on easy problems and instead should dynamically reallocate how they use compute. We introduce Never Give Up (NGU), a simple adaptive sampling method that keeps generating samples for a problem until one is correct. By leveraging asynchronous RL, this naturally uses fewer samples to filter out easy problems and allocates more compute to solving harder problems. We investigate the design choices that affect NGU, such as off-policy robustness, and develop a set of best practices. On the math benchmark Deepscaler, NGU improves performance per compute, especially on harder problems. On a recent coding task, Manufactoria, standard GRPO with a per-test reward fails to fully solve problems that have a range of easy and difficult tests. NGU iteratively improves, solving harder and harder tests, until it learns to fully solve coding problems.

• The paper identifies the Matthew Effect in RL for LLMs, where standard RL methods disproportionately improve easy problems over hard ones due to insufficient learning signals on hard problems. It develops Never Give Up (NGU) to dynamically reallocate computation and improve hard problem training.
• Experiments across various domains, including mathematics, code completion, and agentic software engineering show that fixed-group sampling methods are inefficient. Consequently, NGU outperform fixed-group sampling approaches on hard problems by dynamically adjusting its allocation strategy.
• NGU leverages the asynchronous optimization inherent in large language model (LLM) RL systems, dynamically reallocating compute from solved to unsolved prompts and using simplicity filters for efficient resource allocation. The authors conclude that dynamically reallocating computation between easy and hard problems can outperform both large and small fixed-group baselines in RL.

Reinforcement learning with verifiable rewards is often described as a mechanism for improving an LLM across the distribution of training problems. “Learning to Solve Hard Problems in RL for LLMs by Never Giving Up” (2609.13443) challenges this assumption. Its central claim is that standard RL systematically favors problems the model already solves relatively well. The authors call this pattern the Matthew Effect in RL for LLMs: performance gains are approximately proportional to initial competence, so easy problems become easier while hard problems receive insufficient learning signal. The proposed remedy, Never Give Up (NGU), dynamically reallocates rollout computation toward unresolved prompts through asynchronous sampling.

The Matthew effect in RL for LLMs

The paper first establishes the phenomenon across three domains and three independently developed RL-trained model families: mathematical reasoning with Olmo 3.1 RL-Zero on AIME, code completion with DeepCoder on LiveCodeBench v5, and agentic software engineering with DeepSWE on SWE-bench Verified. Problems are grouped by difficulty using initial model performance or benchmark-provided difficulty categories.

Across all three settings, RL produces its largest improvements on the easiest subset and its smallest improvements on the hardest subset. This is not presented as a peculiarity of one optimizer, model scale, or benchmark. Rather, the authors argue that the relation between initial competence and post-training improvement is systematic: the policy receives more useful gradient signal from problems for which it already produces a mixture of correct and incorrect outputs, while completely unsolved problems frequently contribute no update.

Figure 1: Across mathematics, code completion, and agentic coding, RL improves performance most on tasks that are already relatively easy for the initial model.

This observation extends the literature on primacy bias and vanishing RL gradients. Prior work has shown that reward variance can control whether a prompt produces an effective policy-gradient update, even when the mean reward is far from optimal (Razin et al., 2023). The present paper distinguishes its phenomenon from classical plasticity-loss accounts: the Matthew effect occurs in pretrained models with strong prior competence, rather than only in agents trained from scratch that lose the ability to adapt.

The claim is also deliberately narrower than the claim that RL cannot learn new skills. The paper shows that conventional RL disproportionately sharpens existing competence; it does not establish that sufficiently different objectives, richer rewards, or external guidance cannot produce genuinely novel behaviors. This distinction is important in light of work arguing that many RLVR methods primarily sharpen solution modes already present in the base model (Mayilvahanan et al., 13 Oct 2025).

Why fixed group sampling is inefficient

The mechanistic analysis focuses on GRPO and related group-relative policy-gradient methods. For a prompt, GRPO samples $K$ completions and assigns each completion an advantage relative to the group mean. If all completions are incorrect, every reward is zero and the prompt contributes no gradient. If all are correct, the group likewise provides no relative signal under the filtering procedure used in the experiments.

This motivates a standard intervention: increase $K$ so that difficult prompts have a higher probability of producing at least one correct completion. The paper reports a counterintuitive result. With a fixed total rollout budget, increasing $K$ from 4 to 8, 16, or 32 does not reliably improve hard-problem learning. In the GSM8K Platinum testbed, the $K=4$ configuration performs best overall and especially on the hardest subset.

Figure 2: Under a fixed total batch size, increasing completions per prompt does not necessarily improve hard-problem performance; $K=4$ outperforms larger group sizes in the reported GSM8K experiments.

The explanation is not that larger groups fail to discover correct completions. Rather, they alter which prompts survive filtering. With larger $K$, an easy prompt has more opportunities to produce at least one incorrect completion, so it is more likely to remain in the training batch and consume computation. Conversely, smaller $K$ more aggressively filters easy prompts once they become nearly solved. Thus, $K$ controls not only exploration within a prompt but also the implicit distribution of prompts receiving updates.

Early in training, large $K$ can include more hard prompts because it increases the chance that a difficult prompt contains both successful and unsuccessful samples. Later, however, the dynamics reverse: small $K$ filters solved prompts more efficiently, causing the training batch to become increasingly concentrated on difficult examples.

Figure 3: Smaller group sizes increasingly concentrate later training batches on difficult prompts by filtering solved or low-information easy prompts more aggressively.

The authors therefore replace the signal loss interpretation—insufficient samples to find a positive completion—with a signal efficiency interpretation. The problem is not only that hard prompts are undersampled; it is also that easy prompts are oversampled. This reframing complements adaptive sampling methods such as Reinforce-Ada (Xiong et al., 6 Oct 2025), which emphasize recovering positive signal on difficult prompts, and compute-allocation studies such as IsoCompute (Cheng et al., 12 Mar 2026), which treat rollout allocation as a compute-constrained optimization problem.

Never Give Up

NGU is designed to obtain the benefits of both small and large $K$0 without fixing one group size for every prompt. The procedure initially samples a small group, such as four completions. If the prompt is solved, sampling stops. If all completions fail, standard GRPO would discard the prompt and replace it with another. NGU instead resamples the same prompt with probability $K$1; with probability $K$2, it abandons the prompt. The resulting number of sampling rounds has a geometric distribution, allowing difficult prompts to receive substantially more computation while keeping the expected rollout cost finite.

The implementation depends on asynchronous RL. Easy prompts that terminate quickly are removed from the active queue, and their freed capacity is used to continue sampling unresolved prompts. In a synchronous system, fixed-shape batches would largely eliminate this compute-reallocation benefit. This system dependence is substantive rather than incidental: asynchronous training is also central to prior LLM RL systems that decouple rollout generation and optimization (Noukhovitch et al., 2024), although NGU uses asynchrony as an algorithmic sampling mechanism rather than only as a throughput optimization.

On GSM8K, NGU with $K$3 and $K$4 improves overall pass@1 and obtains its clearest gains on the extra-hard subset.

Figure 4: NGU improves overall pass@1 while concentrating its gains on the hardest GSM8K problems.

The training-batch composition explains the result. NGU initially retains the high hard-prompt fraction associated with large $K$5, while later filtering easy prompts as aggressively as small $K$6. It therefore approaches a Pareto-efficient allocation of training examples across difficulty levels.

Figure 5: NGU combines the early hard-prompt coverage of larger groups with the later easy-prompt filtering of smaller groups.

This result is stronger than simply increasing the rollout budget uniformly. Uniformly larger $K$7 spends additional samples on every prompt, including prompts that are already easy. NGU conditions computation on the observed outcome of the current samples and therefore adapts allocation to the model’s evolving competence.

Maintaining successful and failed completions

NGU can preserve previous completions for a prompt and combine them with newly generated completions when a correct answer is eventually found. This creates a larger GRPO group in which rare positives are contrasted against a larger set of failures. The resulting positive completions receive a stronger relative advantage than they would under an isolated group of four samples.

The benefit is constrained by off-policy staleness. Older negative completions were generated by earlier policies and may become inconsistent with the current policy. In the GSM8K ablation, retaining completions up to four training steps old improves performance, whereas allowing substantially older samples degrades it.

Figure 6: NGU benefits from moderately stale completions but deteriorates when off-policy samples exceed the useful staleness window.

The paper does not discard stale completions entirely. Instead, it uses them to construct a baseline while filtering them from the policy update. Because stale samples are predominantly negative—the sampling process stops after a positive is found—the resulting baseline can become miscentered if the old negatives are excluded without correction. The authors propose anchoring the positives: retain the advantages of current positive completions and rescale the remaining negative advantages so that their sum is zero. This preserves the centered structure of the GRPO update while retaining information from the filtered completions.

Anchoring the positives outperforms both an uncorrected baseline and downsampling of negative completions. The latter result is notable because it contradicts the intuition that balancing positive and negative examples is necessarily beneficial: in this setting, discarding informative negative completions harms learning on the hardest prompts.

Figure 7: On Deepscaler, NGU provides a better hard/easy performance trade-off than fixed group sizes, with the continuation probability controlling the allocation toward difficult problems.

Scaling to Deepscaler mathematics

The larger-scale mathematical experiment uses Qwen 3 4B Base trained on a 10,000-example Deepscaler subset and evaluated on AIME 2025 and BRUMO November 2025. Difficulty is determined by initial pass@64: hard prompts have pass@64 equal to zero, medium prompts have an average pass rate of 2.9%, and easy prompts have an average pass rate of 46.1%.

The comparison is compute-matched at approximately 120 H100 hours per run, with three random seeds. Fixed-group baselines use $K$8; NGU uses $K$9 and varies $K$0 to obtain theoretical average sampling budgets corresponding approximately to $K$1, 64, and 128.

NGU improves the trade-off between easy- and hard-problem performance. Increasing fixed $K$2 can improve difficult prompts but sacrifices performance on easy prompts. NGU improves the hardest subset more substantially while avoiding the same degree of degradation on easy problems. A curriculum baseline that assigns larger $K$3 to initially difficult prompts can match NGU on the hardest subset, but it harms easy-problem performance because initial difficulty estimates become obsolete as the model adapts.

This comparison establishes an important limitation of static curricula: prompt difficulty is policy-dependent and changes during training. NGU does not require an externally maintained difficulty label; its allocation is generated online from the current sampling outcomes.

Coding and test-level difficulty

The Manufactoria experiment extends the analysis from prompt-level difficulty to test-level difficulty within a single coding problem. Each problem contains 14–30 tests, with later tests often substantially harder than earlier ones. A per-test reward allows standard GRPO to improve aggregate test pass rate to approximately 80%, but the model rarely learns to pass all tests for any problem.

The paper interprets this as a second form of signal inefficiency. Easy tests are repeatedly solved, while medium tests generate fluctuating reward differences and the hardest tests remain almost untouched. The aggregate scalar reward therefore rewards partial progress without creating sufficient pressure toward complete problem solutions.

A further complication is harness adaptation. Difficulty measured from the initial model does not initially reveal a clear Matthew effect because the model is still learning the novel prompt format and execution harness. After approximately 100 training steps, the model has adapted to the harness, and difficulty measured at that point produces the expected pattern: RL improves easy prompts more than hard prompts.

Figure 8: Difficulty must be measured after prompt and harness adaptation to expose the Matthew effect in Manufactoria.

With the same per-test reward and $K$4, NGU initially trails standard GRPO on aggregate test pass rate but continues improving on hard tests after GRPO plateaus. It eventually learns to pass every test on some problems, whereas standard GRPO does not fully solve even a single example in the reported run.

Figure 9: NGU continues improving on the hardest Manufactoria tests and transfers this progress into complete problem solutions, while standard GRPO plateaus.

The comparison with an all-tests reward is particularly informative. Prior work required first training with per-test rewards and then restarting from a plateaued checkpoint using a reward granted only when all tests pass. NGU applied to the per-test reward approximately matches the compute efficiency of this staged all-tests procedure. The authors consequently argue that NGU can recover from a locally useful but globally incomplete objective rather than being irreversibly determined by early training, contrasting with plasticity-loss results for RL agents trained from scratch.

Limitations and open questions

The empirical evidence is broad across domains but remains concentrated in correctness-verifiable tasks and relatively specific RL infrastructure. NGU is evaluated primarily with GRPO-style objectives, asynchronous rollout systems, and binary or near-binary reward signals. Its behavior with dense process rewards, preference rewards, multi-turn environments, or long-horizon MDPs is not established.

The method also introduces several implementation-sensitive quantities: the initial group size, the continuation probability, the completion staleness threshold, and the rule for incorporating stale samples into the baseline. Although the paper provides useful ablations, it does not derive a generally optimal policy for these parameters. The geometric continuation mechanism prevents infinite sampling in expectation, but difficult or impossible prompts can still consume substantial tail compute.

The Matthew effect itself depends on how difficulty is defined. Initial pass rate can be misleading when the model has not adapted to a prompt or harness, as the Manufactoria analysis demonstrates. Difficulty buckets based on pass@1 or pass@64 are also policy-dependent and may change as training proceeds. The paper therefore leaves open whether NGU’s gains persist under more principled online estimates of learning potential, or whether methods such as Bayesian prompt-difficulty estimation (Qu et al., 7 Jul 2025) can improve its allocation.

Finally, the paper does not isolate how much of NGU’s improvement comes from dynamic prompt allocation, how much comes from larger effective GRPO groups, and how much comes from the positive-anchoring baseline. The ablations identify each component as relevant, but a complete decomposition of their interaction remains unresolved.

Conclusion

“Learning to Solve Hard Problems in RL for LLMs by Never Giving Up” (2609.13443) argues that standard RL for LLMs exhibits a Matthew effect: training disproportionately improves tasks that the model already solves. Its central diagnosis is signal inefficiency, not merely insufficient group size. Fixed-compute GRPO wastes rollouts on easy prompts while difficult prompts are abandoned after unsuccessful samples.

NGU addresses this asymmetry through asynchronous, probabilistic resampling of unresolved prompts. Across GSM8K, Deepscaler, and Manufactoria, it reallocates computation toward hard mathematical problems and hard coding tests, while preserving performance on easier cases more effectively than fixed-$K$5 sampling or static curricula. The principal open question is whether this adaptive allocation principle extends beyond contextual-bandit-style RLVR to environments where difficulty is state-dependent, rewards are dense or preference-based, and successful exploration requires multi-step credit assignment.