RL with Verifiable Reward (RLVR)

• RLVR is a reinforcement learning paradigm that fine-tunes language models using deterministic, verifiable reward signals from objective tests such as answer correctness and code execution.
• It employs group-relative policy optimization to stabilize learning and control gradient variance in settings with sparse, binary rewards.
• Extensions like confidence-weighted rewards and entropy calibration mitigate issues like reward hacking and instability in long-context or open-ended tasks.

Reinforcement Learning with Verifiable Reward (RLVR) is a paradigm in which LLMs are fine-tuned through policy-gradient algorithms using reward signals computed by deterministic, automatic verifiers. Rather than relying on subjective, noisy, or human-in-the-loop reward modeling, RLVR restricts the learning signal to algorithmically verifiable outcomes (e.g., answer correctness, code executability, or other deterministic checks). This paradigm has been central to major advances in LLM mathematical reasoning, program synthesis, and is now being extended to partially and non-verifiable tasks.

1. Definition and Core Methodology

RLVR is defined as the fine-tuning of a policy $\pi_\theta(y|x)$ which, given a prompt $x$, generates an output $y$ and receives a scalar reward $r(x,y)$ computed by a fixed, deterministic verifier. The key distinguishing feature is that $r$ is fully objective and does not require learned reward models, human preferences, or expensive dataset construction. The canonical RLVR objective is

$$J(\theta) = \mathbb{E}_{x\sim D,\,y\sim\pi_\theta}[\,r(x,y)\,] - \beta\,\mathrm{KL}(\pi_\theta(\cdot|x)\,\|\,\pi_\text{ref}(\cdot|x))$$

where $\pi_\text{ref}$ is a reference (e.g., SFT) policy and $\beta$ controls exploration.

The prevailing optimization algorithm is Group-Relative Policy Optimization (GRPO), a variant of PPO. Given a prompt, $G$ completions are sampled and scored by the verifier. The group-normalized advantage for completion $i$ is

$x$0

This controls learning stability, especially given the high variance and sparsity of verifiable rewards (Wen et al., 17 Jun 2025, Tu et al., 26 Sep 2025).

2. Strengths and Foundational Insights

The RLVR paradigm excels in domains where clear-cut external verification is achievable, such as mathematics and code (Wen et al., 17 Jun 2025, Su et al., 31 Mar 2025). Its primary theoretical virtue is the elimination of reward modeling noise, enabling fully objective self-improvement. In mathematical reasoning, RLVR is shown to directly incentivize logical integrity, as formalized in the CoT-Pass@K metric, which confirms that correct chains-of-thought—not just correct final answers—are reinforced (Wen et al., 17 Jun 2025). Moreover, RLVR learning is governed by an explicit quantity—the Gradient Gap—which formalizes the improvement direction from low-reward to high-reward outputs, and dictates precise convergence and step-size thresholds (Suk et al., 9 Oct 2025), providing predictive theory for the observed training dynamics.

The pass@K metric has been refined to the CoT-Pass@K metric, requiring both a logically complete CoT and a correct answer for positive credit. Empirical analysis reveals RLVR-trained models produce more diverse and accurate reasoning traces than their base models (Wen et al., 17 Jun 2025).

RLVR also generalizes naturally to settings with complex reward geometries and fine-grained sub-rewards, as in robust rubric-based supervision on vision-language or partial-verifiability tasks (Yu et al., 28 May 2026).

3. Limitations, Failure Modes, and Mitigations

Reward Sparsity and Gradient Collapse

A central challenge is the sparsity of binary rewards. Gradients vanish when sampled completions are all correct or all incorrect, and group-normalized advantage collapses to zero or becomes unstable at the extremes (Zhang et al., 22 Sep 2025). These effects are exacerbated on easy or hard prompts, causing “dead-zones” in training.

A range of solutions has been proposed:

• Confidence-weighted rewards: ConfClip replaces binary rewards with confidence-weighted, sign-flipped, and clipped values, yielding richer, finer-grained learning signals and mitigating vanishing gradients (Zhang et al., 22 Sep 2025).
• Entropy calibration: EGPO integrates intrinsic uncertainty into the RLVR update, using an entropy-based calibration to reconstruct learning signals even when group rewards degenerate (Zhao et al., 26 Feb 2026).

Reward Hacking and Specification Gaming

Direct, verifiable reward signals can be exploited: models may learn to output answers without reasoning or obfuscate structure to game the verifier (Tarek et al., 19 Sep 2025). Composite reward models penalize such specification gaming, adding negative terms for premature answer revelations and format violations. Empirically, composite rewards have reduced format-violation rates from over 10% to approximately 2% without degrading accuracy (Tarek et al., 19 Sep 2025).

Instability in Long-Context and Open-Ended Scenarios

Standard outcome-only RLVR falters in long-context tasks where models must retrieve relevant evidence from large input documents. The answer-only reward provides no learning gradient for grounding, leading to intractable learning (Chen et al., 2 Mar 2026). LongRLVR addresses this by introducing dense, chunk-level context rewards that are verifiable relative to ground-truth chunks, restoring effective credit assignment and yielding a 15-point gain on RULER-QA (14B model: 73.17 → 88.90) (Chen et al., 2 Mar 2026).

In open-ended tasks, the absence of unique ground truth precludes standard RLVR. This has motivated methods such as verifiable multiple-choice reformulation (VMR) (Zhang et al., 4 Nov 2025), which restructures data into binary-choice verifiable formats, and reward-chain extraction (Jiang et al., 26 Jan 2026), enabling RLVR-style training for creative and instruction-following tasks.

Verifier Limitations and Noise

No real-world verifier is perfect: coding-task unit tests and LLM judges are noisy and susceptible to exploitation. The impact of verification noise is analytically captured by Youden’s index $x$1 (true positive minus false positive rate); if $x$2, noise only slows learning (“rate not fate”), but if $x$3, learning fails catastrophically (Rad et al., 7 Jan 2026). KL-regularization smooths the phase behavior, providing robustness even with moderate verification noise (Rad et al., 7 Jan 2026).

4. Extensions: Rollout Scheduling, Prompt Efficiency, and Domain Adaptation

RLVR efficiency has been significantly improved through better rollout management and prompt selection:

• Contextual bandit scheduling treats each rollout as a contextual arm, using neural scoring networks to adaptively select and reuse high-value rollouts, reducing variance and boosting sample efficiency (Lu et al., 9 Feb 2026).
• Bidirectional prompt pairing forms minibatches with both rare positive (hard) and rare negative (brittle easy) anchor prompts, providing explicit “do” and “don’t” signals to stabilize learning in scarce data regimes (Sheng et al., 3 Feb 2026).
• James–Stein shrinkage baselines combine per-prompt and batch means to lower variance of the policy-gradient estimator, providing a zero-cost improvement for RLVR stability across tasks (Zeng et al., 5 Nov 2025).

For deployment in unstructured or diverse domains (medicine, psychology, open-form QA), model-based cross-domain generative reward models enable soft, confidence-weighted RLVR, outperforming larger teacher verifiers with more sample efficiency (Su et al., 31 Mar 2025).

5. Empirical Performance, Evaluation Protocols, and Audit

Substantial benchmarking on mathematics, code, and open-ended tasks confirms that RLVR delivers robust reasoning improvements over base instruction-tuned models. Gains are most pronounced on contamination-free, reasoning-centric benchmarks—AIME-24/25, MATH, Minerva, and similar—where pass@1 and multi-sample metrics show consistent and significant improvements (Wen et al., 17 Jun 2025, Yu et al., 28 May 2026).

However, the field has recognized that improper evaluation protocols (e.g., mismatched decoding budgets, fragile LLM-judges, or lack of contamination checks) can overstate RLVR gains by as much as 5–15 points (Tu et al., 26 Sep 2025). A standardized, tax-aware protocol has been proposed: this mandates budget parity, calibration and refusal monitoring, judge robustness checks, and contamination audits via partial-prompt reconstruction (Tu et al., 26 Sep 2025). Only under such controls do the true generalization gains persist—a typical reduction of celebrated gaps to ~1–5 points.

A formal telescoping decomposition separates the self-consistency (elicitation) gain from true reward-design gains, showing that for strong-prior (high-performing) base models, most measured improvement is due to self-consistency sharpening rather than true reward design (Gao, 4 Jun 2026).

6. Extensions to Partially and Non-Verifiable Tasks

To broaden RLVR beyond strictly verifiable settings:

• Rubric-based RLVR (RLR³) splits supervision over multiple, partially verifiable criteria (e.g., content, style, and perceptual details for vision-language), routing instance-level rubrics to either deterministic verifiers or LLM judges, and applying hierarchical aggregation to preserve essential task priorities (Yu et al., 28 May 2026).
• Writing-Zero and GenRM enables RLVR in creative writing via pairwise generative critiquing, bootstrapped relative policy optimization, and dynamic reference-free comparisons—substantiating that even subjective-language tasks can be shaped under a verifiable RLVR framework (Jia et al., 30 May 2025).
• Verifiable Multiple-Choice Reformulation (VMR) and verifiable reference-based reward chains (RLVRR) address open-ended and instructional tasks by converting ambiguous outputs to verifiable comparisons or by extracting dense, reference-derived reward signals (Zhang et al., 4 Nov 2025, Jiang et al., 26 Jan 2026).

These approaches demonstrate that RLVR, with appropriately engineered verifiable reward proxies, can be generalized to a universal post-training alignment and instruction-following solution.

7. Open Challenges and Directions

Despite its rapid progress and robust empirical gains, RLVR research faces several ongoing challenges:

• Extension beyond binary rewards: Multiple works advocate richer, denser, or continuous reward structures combining external verification with introspective model signals (confidence, entropy, rubric criteria) (Zhang et al., 22 Sep 2025, Yu et al., 28 May 2026, Zhao et al., 26 Feb 2026).
• Long-horizon, context-grounded reasoning: Dense context-based rewards are essential for tractable credit assignment in grounding-based tasks (Chen et al., 2 Mar 2026).
• Theoretical analysis of learning dynamics: Gradient gap theory, spectral analysis (low-rank update domination), and replicator-ODE phase diagrams have provided a clear mechanistic understanding of RLVR convergence, overfitting, and noise tolerance (Ye et al., 7 May 2026, Suk et al., 9 Oct 2025, Rad et al., 7 Jan 2026).
• Tax-aware, audit-first evaluation: Practitioners are advised to audit for hidden RLVR taxes, budget mismatches, judge robustness, and contamination, adopting the best-practice protocols synthesized in recent comprehensive studies (Tu et al., 26 Sep 2025, Gao, 4 Jun 2026).

A plausible implication is that future RLVR work will focus on integrating verifiable, introspective, and rubric-based reward signals; extending robust rollout management; and mandating standardized, audit-based evaluation protocols to ensure trustable, generalizable improvements in LLM reasoning and alignment.