Reinforcement Learning with Verifiable Rewards

• Reinforcement learning with verifiable rewards (RLVR) is a framework where policy optimization is driven by objective signals from verifiers, such as unit tests and symbolic checks, rather than human preferences.
• It employs KL-regularized policy optimization and diverse reward constructions (e.g., CER, reference-derived reward chains) to enable scalable, structured reasoning across tasks like math, coding, and negotiation.
• Practical challenges include credit assignment, verifier noise, and evaluation artifacts—collectively known as the 'RLVR tax'—which impact calibration, safety, and efficiency.

Searching arXiv for papers on reinforcement learning with verifiable rewards and closely related variants. Reinforcement learning with verifiable rewards (RLVR) is a post-training paradigm for LLMs in which policy optimization is driven by rewards that are objectively and automatically checkable by a verifier, rather than by human preference models. In its canonical form, a prompt is sampled, the policy generates one or more responses, and a deterministic or reference-grounded verifier assigns reward from correctness, execution outcomes, constraint satisfaction, or analogous mechanically evaluable signals. RLVR has been most successful in mathematics, coding, factual QA, and other structured reasoning tasks, but recent work expands the notion of verifiability to free-form reference-based grading, score-based optimization without ground-truth solutions, dense process supervision, negotiation, and open-ended generation (Tu et al., 26 Sep 2025, Xiao et al., 11 Mar 2026, Lin et al., 25 Jun 2026, Jiang et al., 26 Jan 2026).

1. Definition, scope, and canonical formulation

A standard formulation treats RLVR as the optimization of a policy $\pi_\theta$ under verifier-computed reward. One common objective is

$$J(\theta) = \mathbb{E}_{x\sim D,\; y\sim \pi_\theta(\cdot\mid x)}[r(x,y)],$$

with $r(x,y)$ produced by exact-answer matching, symbolic equivalence, unit tests, parser-based checks, or other objective procedures (Zeng et al., 5 Nov 2025, Su et al., 31 Mar 2025). In this sense, RLVR differs from RLHF-style pipelines because the reward source is not a learned human-preference model, and it differs from generic RL formulations because many prominent RLVR settings have sparse, terminal, verifier-grounded rewards rather than dense environment dynamics.

Within contemporary LLM post-training, RLVR is commonly instantiated with PPO-, GRPO-, or DAPO-style policy optimization under a KL penalty to a base or reference policy, often with online rollouts, offline bootstrapping, entropy control, and response filtering or gating (Tu et al., 26 Sep 2025). The central practical appeal is scalability: where verifiers exist, RLVR avoids dense human annotation for every trajectory and can exploit large volumes of cheap programmatic supervision. This has made it a major approach for reasoning-oriented post-training in domains where correctness is externally checkable (Mroueh, 9 Mar 2025).

The scope of RLVR is no longer limited to single-answer tasks. Some work still studies the classical single-turn setting in which an environment samples a problem instance, the model emits an answer, and a verifier returns reward; other work extends the same basic principle to multi-turn agents and strategic interaction, where terminal outcomes or intermediate actions remain objectively checkable (Stojanovski et al., 30 May 2025, Yuan et al., 11 May 2026).

2. Verifiers and reward construction

The original RLVR regime is dominated by explicit rule-based verifiers. These include exact string match, algebraic equivalence, unit-test pass/fail, parser-based format checks, executable semantics, and deterministic refusal or citation-grounding checks (Tu et al., 26 Sep 2025, Stojanovski et al., 30 May 2025). In structured tasks this produces a clean verifier signal, but recent work has broadened the reward design space in several directions.

One influential proposal argues that RLVR should not optimize correctness in isolation and introduces a componentized default reward

$$\mathcal{R}=\alpha\,\text{Correct}+\beta\,\text{Grounding}+\gamma\,\text{Refusal},$$

where correctness, citation or evidence sufficiency, and calibrated abstention are optimized jointly and on a staged schedule (Tu et al., 26 Sep 2025). This formulation makes explicit that gains in answer accuracy can coexist with degradation in grounding or refusal behavior.

A second line of work replaces handcrafted external verification with model-intrinsic verification. Conditional Expectation Reward (CER) defines reward as the expected likelihood of generating the reference answer conditioned on the generated answer, thereby yielding a soft, graded signal rather than binary pass/fail. CER is presented as an extension of RLVR to free-form domains and is proven to be equal in expectation to exact-match reward, while offering denser supervision in tasks where valid answers have diverse surface forms (Xiao et al., 11 Mar 2026).

A third line extends RLVR to broad reference-based free-form QA. In this setting, a verifier model consumes the prompt, an expert-written reference answer, and the model output, and returns either a binary judgment or a soft confidence-derived reward. Empirically, binary verification judgments across math and multi-subject college-level tasks exhibit high cross-model agreement, which enables reward-model distillation without extensive domain-specific human annotation (Su et al., 31 Mar 2025).

A fourth line targets open-ended generation through reference-derived “reward chains” rather than single final-answer checks. RLVRR decomposes reward into content and style,

$r_\theta(x,y)=F\big(r_c(x,y,z),\, r_s(x,y,z)\big),$

where content is verified by longest-common-subsequence matching over reference-derived keyword sequences and style is evaluated by LLM-generated Python checkers over structural properties such as headings, formatting, or length range (Jiang et al., 26 Jan 2026). This preserves flexibility in wording while making key semantic and stylistic constraints verifiable.

Finally, RLVR has been generalized to settings without ground-truth solutions. RiVER treats deterministic execution outcomes on score-based optimization tasks as verifiable supervision, not by exact correctness but by within-instance ranking and winner-heavy bounded reward shaping. In this regime, verifiability means feasibility checking and objective comparison among sampled candidates rather than comparison to a known gold answer (Lin et al., 25 Jun 2026).

3. Optimization foundations and dynamics

The dominant optimization family for RLVR is KL-regularized policy optimization. In practice this includes PPO-like clipped objectives, GRPO group-relative normalization, RLOO-style leave-one-out baselines, and DAPO variants, usually applied to groups of sampled completions per prompt (Tu et al., 26 Sep 2025). What distinguishes the RLVR case is that reward is often binary or bounded and verifier-derived, which makes the optimization problem more analytically tractable than preference-based RL.

For binary rewards, GRPO admits an especially explicit characterization. It can be rewritten as a KL-regularized contrastive objective over successful and unsuccessful synthetic trajectories sampled from the old policy, and the optimal policy update has a closed-form exponential-tilting expression relative to the reference policy (Mroueh, 9 Mar 2025). The same analysis yields a one-dimensional recurrence for the success probability $p_n(q)$, showing that under mild conditions the fixed point $p^*$ exceeds the reference success rate $p_{\mathrm{ref}}(q)$. In this precise sense, GRPO acts as a success amplifier rather than merely a local optimizer (Mroueh, 9 Mar 2025).

This theoretical picture supports a broader empirical debate. One interpretation is that RLVR improves reasoning by amplifying rare successful behaviors already present in the base policy. Another is that, in some regimes, prolonged RL with KL control and related interventions can expand capability beyond mere search or sampling efficiency. The position paper on RLVR argues that many practical gains are real but often overstated, and that a substantial fraction of reported improvement can be explained by distributional sharpening, search budget, or memorization unless parity-controlled evaluation is enforced (Tu et al., 26 Sep 2025).

The role of KL regularization is central in both theory and practice. In the analytical GRPO view, smaller $\beta$ increases reweighting toward successful outputs and strengthens amplification, while larger $\beta$ keeps the updated policy closer to the reference and weakens amplification (Mroueh, 9 Mar 2025). In empirical training recipes, KL to the base model is monitored alongside entropy and reward variance because large drift can coincide with calibration degradation, reward hacking, or collapse into narrow high-reward behaviors (Tu et al., 26 Sep 2025).

4. Credit assignment, variance reduction, and sample efficiency

A large methodological literature within RLVR addresses the fact that verifier signals are sparse, group sizes are small, and rollout generation is expensive. The interventions differ, but they share a common target: improving the fidelity of the update without abandoning verifier-grounded supervision.

Outcome-level group methods usually broadcast one sequence-level advantage to all tokens. GRAIL replaces this with token-wise reweighting derived from gradient-activation saliency of the final answer loss, producing average gains of $$J(\theta) = \mathbb{E}_{x\sim D,\; y\sim \pi_\theta(\cdot\mid x)}[r(x,y)],$$0 in accuracy and $$J(\theta) = \mathbb{E}_{x\sim D,\; y\sim \pi_\theta(\cdot\mid x)}[r(x,y)],$$1 in Pass@3 across five models while avoiding process-level supervision (Pala et al., 3 Jun 2026). At the estimator level, James–Stein-inspired shrinkage baselines improve prompt-level mean estimation and reduce gradient variance in low-generation RLVR, while Discounted Beta–Bernoulli reward estimation uses prompt-wise discounted historical reward statistics to stabilize Bernoulli reward estimation and avoid variance collapse without replay overhead (Zeng et al., 5 Nov 2025, Kim et al., 19 Mar 2026).

Other methods improve which data are optimized on. Contextual Rollout Bandits formulate rollout scheduling as a contextual bandit problem, using a neural scheduler to select high-value current or historical rollouts and reporting consistent performance and training-efficiency gains across GRPO, DAPO, and GSPO (Lu et al., 9 Feb 2026). FEST addresses rare-success regimes by combining GRPO on a large answer-only dataset with only 128 randomly selected demonstrations and finds that supervised signal, on-policy signal, and decaying demonstration weight are all required for success (Yan et al., 14 May 2026).

Still other approaches expand the supervised behaviors inside RLVR. RISE jointly trains a model to solve problems and verify its own solutions online, using the same outcome verifier to supervise both generation and self-verification; the largest effect is on verification accuracy, with up to $$J(\theta) = \mathbb{E}_{x\sim D,\; y\sim \pi_\theta(\cdot\mid x)}[r(x,y)],$$2 improvement reported in the abstract (Liu et al., 19 May 2025). RLAVR treats RLVR as a budgeted active-labeling problem: it acquires ground-truth labels for only a subset of prompts, defines supervision value through the Corrective Advantage Gap, and uses CARE to decide which pseudo-labeled samples to trust, correct, or discard (Wang et al., 25 May 2026).

Method family	Main intervention	Representative paper
GRAIL	Token-wise saliency reweighting of sequence advantage	(Pala et al., 3 Jun 2026)
Shrinkage baselines	James–Stein-style prompt baseline shrinkage	(Zeng et al., 5 Nov 2025)
DBB	Discounted Beta–Bernoulli reward estimation	(Kim et al., 19 Mar 2026)
CBS	Contextual rollout selection and reuse	(Lu et al., 9 Feb 2026)
FEST	Few-shot demonstration-guided RLVR with 128 demonstrations	(Yan et al., 14 May 2026)
RISE	Joint generation and self-verification	(Liu et al., 19 May 2025)
RLAVR/CARE	Active acquisition of ground-truth labels	(Wang et al., 25 May 2026)

Taken together, these methods indicate that the dominant bottlenecks in RLVR are not only reward definition but also credit assignment, baseline quality, rollout selection, and label allocation.

5. Evaluation, measurement gaps, and the RLVR tax

A major current theme in the RLVR literature is that measured gains can be inflated by evaluation artifacts. The most explicit formulation is the “RLVR tax,” which groups hidden costs into hallucination and overconfidence, instruction-following erosion, and safety or privacy exposure (Tu et al., 26 Sep 2025). In factual QA controls, RL reasoning models can sharply reduce abstention while leaving shared accuracy nearly unchanged or worse; for example, Qwen2.5 14B moves from 1136 unattempted questions in the base instruct model to 103 in the RL reasoning variant, while shared accuracy drops from $$J(\theta) = \mathbb{E}_{x\sim D,\; y\sim \pi_\theta(\cdot\mid x)}[r(x,y)],$$3 to $$J(\theta) = \mathbb{E}_{x\sim D,\; y\sim \pi_\theta(\cdot\mid x)}[r(x,y)],$$4 and ECE worsens from $$J(\theta) = \mathbb{E}_{x\sim D,\; y\sim \pi_\theta(\cdot\mid x)}[r(x,y)],$$5 to $$J(\theta) = \mathbb{E}_{x\sim D,\; y\sim \pi_\theta(\cdot\mid x)}[r(x,y)],$$6 (Tu et al., 26 Sep 2025).

The same paper argues that strict budget parity is necessary for credible comparison: same $$J(\theta) = \mathbb{E}_{x\sim D,\; y\sim \pi_\theta(\cdot\mid x)}[r(x,y)],$$7, same decoding settings, same verifier, same prompt family, and multiple seeds. Its standardized evaluations estimate pass@1 by averaging 32 independent single-sample decodes, namely avg@32 of pass@1, under matched settings. Under such controls, some previously reported margins shrink materially; for instance, Open-RS3-1.5B falls from reported AIME-24 pass@1 of 46.70 to 30.94 under standardization (Tu et al., 26 Sep 2025).

Data contamination is treated as a separate failure mode because memorization can be mistaken for reasoning. The proposed audit reveals only the first $$J(\theta) = \mathbb{E}_{x\sim D,\; y\sim \pi_\theta(\cdot\mid x)}[r(x,y)],$$8 of a problem, with $$J(\theta) = \mathbb{E}_{x\sim D,\; y\sim \pi_\theta(\cdot\mid x)}[r(x,y)],$$9, and measures ROUGE-L@x, EM@x, and ACC@x. On legacy math sets, several Qwen models can reconstruct hidden suffixes at high rates, whereas the same phenomenon collapses on the fresh AIME-2025 benchmark; this asymmetry is used as evidence of contamination rather than generic suffix-completion skill (Tu et al., 26 Sep 2025).

Inference cost is also part of the tax. In standardized math evaluations, reasoning-tuned models often emit dramatically longer sequences than instruct baselines: Qwen2.5-Math-1.5B-Instruct averages 783 generated tokens, whereas DeepSeek-R1-Distill-Qwen-1.5B averages 11,394, and Polaris-4B-Preview (Thinking) averages 20,179 versus 3,516 for the non-thinking variant (Tu et al., 26 Sep 2025). The practical implication is that accuracy claims cannot be separated from latency, exposure surface, and compute budget.

A related theoretical perspective studies verifier noise directly. In a mean-field analysis of GRPO-style RLVR with false positives and false negatives, the long-run drift of incorrect reasoning modes depends only on Youden’s index

$r(x,y)$0

If $r(x,y)$1, incorrect mass is driven toward extinction; if $r(x,y)$2, the dynamics are neutral; if $r(x,y)$3, incorrect modes amplify and dominate. This introduces a sharp distinction between rate effects and fate effects: verifier noise can merely slow learning, or it can reverse the direction of learning entirely (Rad et al., 7 Jan 2026).

6. Environments and application domains

The most mature RLVR environments remain math, code, and structured reasoning, but the domain frontier is broader. Reasoning Gym provides over 100 procedural generators and verifiers spanning algebra, arithmetic, algorithms, cognition, geometry, graphs, logic, games, and code, with effectively unlimited data and explicit difficulty control (Stojanovski et al., 30 May 2025). In that framework, curriculum RLVR increases difficulty when performance exceeds $r(x,y)$4 over the last 20 training steps, and RL on synthetic verifiable tasks transfers both within and across domains as well as to external benchmarks such as MATH, GSM8K, Big-Bench Hard, and MMLU-Pro (Stojanovski et al., 30 May 2025).

Another expansion targets multi-turn agents. Verifiable Process Rewards studies densely-verifiable agentic reasoning problems in which intermediate actions can be checked by symbolic or algorithmic oracles. It instantiates turn-level verification in Tic-Tac-Toe through MCTS-optimal moves, in Sudoku through consistency with the unique solution grid, and in Minesweeper through posterior-based action validity; VPR outperforms outcome-only and rollout-based process reward baselines in these controlled environments and transfers to general and agentic reasoning benchmarks (Yuan et al., 11 May 2026).

Strategic interaction is now also בתוך the RLVR scope. In bilateral buyer–seller negotiation, the final price, budget feasibility, and seller-cost feasibility are mechanically checkable, so reward can be grounded directly in normalized buyer surplus. Using GRPO with this reward, a 30B buyer model learns a four-phase strategic evolution from naive bargaining to aggressive anchoring, deadlock, rational concession, and advanced persuasion, and achieves reward 0.7664 on held-out products, exceeding the strongest reported zero-shot buyer baseline in the same table (Liu et al., 10 Apr 2026).

Cross-domain free-form QA and open-ended generation further extend the range of verifiable supervision. Reference-grounded generative verification supports RL in medicine, chemistry, psychology, economics, law, and education when expert-written answers exist (Su et al., 31 Mar 2025). RLVRR shows that reference-derived content and style chains can support RL on open-ended instruction-following benchmarks, outperforming SFT trained with ten times more data in the reported setup (Jiang et al., 26 Jan 2026).

Setting	Verifiable signal	Representative paper
Procedural reasoning environments	Built-in generators and deterministic verifiers	(Stojanovski et al., 30 May 2025)
Agentic reasoning	Turn-level symbolic or algorithmic process rewards	(Yuan et al., 11 May 2026)
Negotiation	Economic surplus and budget feasibility	(Liu et al., 10 Apr 2026)
Cross-domain free-form QA	Reference-grounded generative grading	(Su et al., 31 Mar 2025)
Open-ended generation	Reference-based content and style reward chain	(Jiang et al., 26 Jan 2026)
Score-based optimization	Deterministic execution and within-instance ranking	(Lin et al., 25 Jun 2026)

7. Debates and open problems

Several unresolved questions structure current research. One is whether RLVR produces genuine capability expansion or mainly distributional sharpening. The GRPO theory proves success amplification relative to a reference model, but the position paper argues that many headline gains shrink under matched-budget, contamination-controlled evaluation and that the observed improvements are often entangled with search budget or memorization (Mroueh, 9 Mar 2025, Tu et al., 26 Sep 2025). This suggests that claims about “new reasoning capability” require stricter evidence than raw benchmark improvement.

A second open problem concerns the boundary of verifiability itself. Extensions such as CER, cross-domain reference-based grading, RLVRR, and RiVER all broaden the notion of verifier beyond exact symbolic checks, but they also import new assumptions: calibration of model-intrinsic probabilities, adequacy of expert-written references, reliability of generated style checkers, or trustworthiness of deterministic score evaluators (Xiao et al., 11 Mar 2026, Su et al., 31 Mar 2025, Jiang et al., 26 Jan 2026, Lin et al., 25 Jun 2026). These are not objections to RLVR; they redefine where the method remains objective enough to justify the name.

A third issue is oracle quality. VPR demonstrates directly that dense process supervision is beneficial only when the verifier is strong: in Tic-Tac-Toe, a weak MCTS oracle with $r(x,y)$5 simulations performs worse than the base model, whereas $r(x,y)$6 yields the best in-domain and OOD results (Yuan et al., 11 May 2026). The noisy-verifier analysis reaches a parallel conclusion in mean-field form through the $r(x,y)$7 phase transition (Rad et al., 7 Jan 2026). Both results make verifier quality, not just algorithm choice, a primary determinant of RLVR stability.

A fourth issue is faithful evaluation beyond answer correctness. The same works that make RLVR attractive also expose calibration drift, abstention collapse, unsafe long reasoning traces, and prompt-sensitive judge evaluations (Tu et al., 26 Sep 2025). The constructive position emerging from this literature is not anti-RLVR. Rather, it is that RLVR is valuable and industry-ready, but only if training and evaluation jointly track accuracy, calibration, abstention, provenance, budget parity, and safety surface (Tu et al., 26 Sep 2025).

In that broader sense, RLVR is no longer a narrow technique for exact-match math problems. It is becoming a general framework for post-training LLMs whenever correctness, feasibility, ranking, or process validity can be made sufficiently objective to support reliable optimization. The field’s present challenge is therefore not whether verifiable rewards work, but how far verifiability can be extended without losing the measurement discipline that made RLVR compelling in the first place.