Reinforcement Learning from Verifiable Rewards (RLVR) Explained

• Reinforcement Learning from Verifiable Rewards (RLVR) is a paradigm in which a language model receives a reward from an automatic verifier for generating an outcome.
• RLVR applied to tasks like mathematical reasoning, code generation, and open-ended generation, RLVR offers automated and objective verification, with trade-offs in sparse credit, verifier dependence, and reward hacking.
• Recent advancements, such as procedural and reference-based verification, group-based policy optimization, and various stability methods, have led to significant improvements in the accuracy and reasoning capabilities of language models under RLVR.

Reinforcement Learning from Verifiable Rewards (RLVR) is a post-training paradigm in which a LLM generates an outcome and receives a reward computed by an automatic verifier rather than by human preference alone. The canonical setting uses binary rewards, with $r(q,o)=1$ for a correct, executable, or otherwise verifiably acceptable outcome and $r(q,o)=0$ for failure. RLVR has been applied to mathematical reasoning, code generation, procedural reasoning environments, self-verification, emotional dialogue, open-ended generation, and long-horizon agents. Its central advantage is that it directly optimizes externally checkable outcomes; its central limitations are sparse credit assignment, verifier dependence, reward hacking, finite-sample variance, and the difficulty of defining verifiers for ambiguous tasks.

1. Formal setting and defining properties

Let $q\sim\rho_{\mathcal Q}$ denote a prompt and $o\sim\pi_\theta(\cdot\mid q)$ a complete generated outcome, such as a reasoning trace and answer or a generated program. A verifier supplies a reward

$r:\mathcal Q\times\mathcal O\to\{0,1\}.$

The objective is to maximize expected verified success:

$$J(\theta)=\mathbb E_{q}\mathbb E_{o\sim\pi_\theta(\cdot\mid q)}[r(q,o)].$$

For policy-gradient optimization, the corresponding estimator is

$$\nabla_\theta J(\theta) = \mathbb E_{q,o} \left[ r(q,o)\nabla_\theta\log\pi_\theta(o\mid q) \right].$$

In mathematical reasoning, the verifier may check exact or normalized answers. In programming, it may execute generated code against a test suite. In procedural environments, it may check whether a generated route, puzzle solution, or transformation satisfies algorithmic constraints. The verifier is intended to be deterministic or automatically computable, although later RLVR systems also use model-based, reference-conditioned, and simulated-user verification.

RLVR differs from supervised fine-tuning (SFT), which maximizes likelihood of reference responses, and from RLHF or preference optimization, which generally rely on learned or human preference signals. Verifiable rewards optimize correctness or constraint satisfaction directly. This distinction is especially consequential when the reward is sparse but reliable: an entire reasoning trajectory may receive a positive signal if its final answer passes verification, without requiring labels for every intermediate step.

A common implementation uses a reference policy $\pi_{\mathrm{ref}}$ and KL regularization:

$$J(\theta) = \mathbb E[r(q,o)] - \beta D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{ref}}),$$

where $\beta>0$ controls deviation from the reference. Larger $r(q,o)=0$0 constrains policy movement more strongly, while smaller $r(q,o)=0$1 permits more aggressive reward-driven reweighting. Group-based methods such as GRPO sample multiple responses for each prompt and normalize their rewards relative to the group.

The scope of “verifiable” is not uniform. Exact answer matching, symbolic checking, code execution, and procedurally generated constraint tests provide comparatively direct verification. Reference-based semantic comparison, simulated emotional state, and model-internal likelihoods provide operationally reproducible but less objective proxies. The distinction between procedural verifiability and human objectivity is central to the expansion of RLVR beyond mathematics and programming.

2. Optimization dynamics and success amplification

Group Relative Policy Optimization

For a prompt $r(q,o)=0$2, GRPO samples a group

$r(q,o)=0$3

and computes the empirical success rate

$r(q,o)=0$4

In the population setting,

$r(q,o)=0$5

For binary rewards, the variance under the old policy is $r(q,o)=0$6. The whitened advantage is therefore

$r(q,o)=0$7

Successful and unsuccessful outcomes receive

$r(q,o)=0$8

A stabilized alternative replaces the denominator with $r(q,o)=0$9, making the update continuous at $q\sim\rho_{\mathcal Q}$0 and $q\sim\rho_{\mathcal Q}$1.

The clipped GRPO objective is PPO-like, with positive advantages reinforced until their likelihood ratio reaches $q\sim\rho_{\mathcal Q}$2 and negative advantages downweighted until their ratio reaches $q\sim\rho_{\mathcal Q}$3. The exact analytical treatment of the binary-reward dynamics uses the unclipped limit. In that limit, importance sampling converts the expectation under the old policy into an expectation under the current policy, producing a KL-regularized contrastive objective: successful outcomes receive a positive weight, unsuccessful outcomes a negative weight, and the KL divergence anchors the policy to $q\sim\rho_{\mathcal Q}$4 (Mroueh, 9 Mar 2025).

Exponential tilting

At iteration $q\sim\rho_{\mathcal Q}$5, let $q\sim\rho_{\mathcal Q}$6 be the old policy and let

$q\sim\rho_{\mathcal Q}$7

The population optimizer has the exponential-tilting form

$q\sim\rho_{\mathcal Q}$8

where successful outcomes receive the reward-side tilt $q\sim\rho_{\mathcal Q}$9, unsuccessful outcomes receive the failure-side penalty $o\sim\pi_\theta(\cdot\mid q)$0, and $o\sim\pi_\theta(\cdot\mid q)$1 is the normalization constant.

The corresponding success probability follows the recurrence

$o\sim\pi_\theta(\cdot\mid q)$2

with

$o\sim\pi_\theta(\cdot\mid q)$3

The success odds satisfy

$o\sim\pi_\theta(\cdot\mid q)$4

The multiplicative factor is strictly greater than one, so the idealized update amplifies success odds. When the old policy is weak, successful outcomes are rare and receive strong reward-side amplification. When the old policy is already strong, failures receive stronger suppression. The parameter $o\sim\pi_\theta(\cdot\mid q)$5 controls the strength of both effects.

For $o\sim\pi_\theta(\cdot\mid q)$6, the recurrence is continuous on $o\sim\pi_\theta(\cdot\mid q)$7, and at least one fixed point exists. Uniqueness is not guaranteed. Local convergence at a fixed point $o\sim\pi_\theta(\cdot\mid q)$8 requires

$o\sim\pi_\theta(\cdot\mid q)$9

The corresponding local convergence is geometric, with asymptotic ratio approximately $r:\mathcal Q\times\mathcal O\to\{0,1\}.$0. The recurrence can be nonmonotone globally: $r:\mathcal Q\times\mathcal O\to\{0,1\}.$1 for $r:\mathcal Q\times\mathcal O\to\{0,1\}.$2, $r:\mathcal Q\times\mathcal O\to\{0,1\}.$3 at $r:\mathcal Q\times\mathcal O\to\{0,1\}.$4, and $r:\mathcal Q\times\mathcal O\to\{0,1\}.$5 for $r:\mathcal Q\times\mathcal O\to\{0,1\}.$6. Consequently, monotonic improvement at every iteration is not guaranteed.

The amplification theorem establishes that the relevant convergent fixed point exceeds the reference success rate under stated conditions. If $r:\mathcal Q\times\mathcal O\to\{0,1\}.$7, amplification holds for every $r:\mathcal Q\times\mathcal O\to\{0,1\}.$8. For $r:\mathcal Q\times\mathcal O\to\{0,1\}.$9, an additional sufficient condition on $$J(\theta)=\mathbb E_{q}\mathbb E_{o\sim\pi_\theta(\cdot\mid q)}[r(q,o)].$$0 controls the recurrence between $$J(\theta)=\mathbb E_{q}\mathbb E_{o\sim\pi_\theta(\cdot\mid q)}[r(q,o)].$$1 and $$J(\theta)=\mathbb E_{q}\mathbb E_{o\sim\pi_\theta(\cdot\mid q)}[r(q,o)].$$2. The result concerns the fixed point reached by convergent iteration; it does not establish uniqueness or that every mathematically possible crossing lies above the reference rate.

3. Verifier construction across domains

Procedural and reference-based verification

Reasoning Gym provides more than 100 procedurally generated reasoning environments with automatic verifiers spanning algebra, arithmetic, computation, cognition, geometry, graph theory, logic, games, and constraint satisfaction. Generators produce fresh instances with adjustable difficulty, structural parameters, and stylistic variation. Examples include spiral_matrix, prime_factorization, mini_sudoku, and shortest_path. The workflow is

$$J(\theta)=\mathbb E_{q}\mathbb E_{o\sim\pi_\theta(\cdot\mid q)}[r(q,o)].$$3

The library supports virtually infinite training data, curriculum learning, and evaluation at varying difficulty levels. Its reported GRPO experiments with Qwen2.5-3B-Instruct show intra-domain and cross-domain transfer, external benchmark gains, and stronger results under curriculum training. The principal limitations are uneven transfer, difficulty cliffs, formatting-related reward spikes, and the inability of procedural verifiers to capture qualities such as creativity, nuanced explanation, or human preferences (Stojanovski et al., 30 May 2025).

Reference-conditioned verification extends RLVR to free-form answers when expert-written reference answers exist. A generative verifier receives the question, the final step of a response, and an expert reference, then emits a binary judgment. High agreement between GPT-4o and Qwen2.5-72B-Instruct on mathematics and multi-subject examination data supports the consistency of reference-based judgments. A distilled Qwen2.5-7B verifier trained on 160,000 teacher-labeled examples provides rewards across mathematics and multiple subjects without domain-specific reward annotations. Soft rewards derived from the probability of positive or negative judgment tokens improve performance on ambiguous multi-subject tasks, although token probabilities are not established as calibrated probabilities of correctness (Su et al., 31 Mar 2025).

Verifiable Multiple-Choice Reformulation (VMR) converts an open-ended preference problem into a binary choice. Given a preferred response $$J(\theta)=\mathbb E_{q}\mathbb E_{o\sim\pi_\theta(\cdot\mid q)}[r(q,o)].$$4 and a rejected response $$J(\theta)=\mathbb E_{q}\mathbb E_{o\sim\pi_\theta(\cdot\mid q)}[r(q,o)].$$5, the model receives both in randomized order and must select Response A or B. The verifier checks whether the selected option corresponds to $$J(\theta)=\mathbb E_{q}\mathbb E_{o\sim\pi_\theta(\cdot\mid q)}[r(q,o)].$$6. VMR therefore makes the comparison exact without claiming that the entire open-ended response space has a unique correct answer. On eight open-ended benchmarks, VMR-based RL improves the reported average by 5.99 points over the baseline. Its limitations include dependence on preference labels, distractor quality, semantic and stylistic shortcuts, and the train–test shift from judging candidate pairs to generating responses from scratch (Zhang et al., 4 Nov 2025).

Structured reference-based rewards

RLVRR decomposes a high-quality reference into content and style constraints. An LLM extracts ordered key points and short keyword sequences, while additional LLM-generated Python functions test measurable stylistic properties such as length, headings, Markdown, lists, code blocks, and organization. The content reward uses longest common subsequence alignment between ordered keyword sequences:

$$J(\theta)=\mathbb E_{q}\mathbb E_{o\sim\pi_\theta(\cdot\mid q)}[r(q,o)].$$7

The style reward is a weighted sum of executable checks,

$$J(\theta)=\mathbb E_{q}\mathbb E_{o\sim\pi_\theta(\cdot\mid q)}[r(q,o)].$$8

and the total reward is the average of content and style rewards. RLVRR uses GRPO with rule-based checks at training time after offline reference processing. On Qwen and Llama models, it outperforms SFT trained with ten times more data and learned reward-model baselines while adding only 0.71% runtime over random rewards. The reward chain remains criterion-relative: keyword alignment does not establish semantic correctness, and executable style checks do not verify truthfulness, nuance, originality, or coherence (Jiang et al., 26 Jan 2026).

Conditional Expectation Reward (CER) uses the policy itself as an implicit verifier. For a sampled answer $$J(\theta)=\mathbb E_{q}\mathbb E_{o\sim\pi_\theta(\cdot\mid q)}[r(q,o)].$$9 and reference $$\nabla_\theta J(\theta) = \mathbb E_{q,o} \left[ r(q,o)\nabla_\theta\log\pi_\theta(o\mid q) \right].$$0,

$$\nabla_\theta J(\theta) = \mathbb E_{q,o} \left[ r(q,o)\nabla_\theta\log\pi_\theta(o\mid q) \right].$$1

CER assigns a high reward when contexts associated with the sampled answer also make the reference answer likely. It is bounded in $$\nabla_\theta J(\theta) = \mathbb E_{q,o} \left[ r(q,o)\nabla_\theta\log\pi_\theta(o\mid q) \right].$$2, provides graded rather than binary feedback, and has the same expected value as exact-match reward. A Monte Carlo estimator reuses sampled solutions and computes likelihood-weighted reference probabilities. CER is particularly effective in general-domain reasoning, where it is competitive with rule-based and external model-based verifiers. It remains model-relative and can exhibit self-preference, calibration error, or self-reinforcing reward hacking (Xiao et al., 11 Mar 2026).

Subjective and interactive environments

RLVER applies verifiable-reward training to emotional dialogue through SAGE, a simulated user with a persona, background, explicit purpose, hidden intention, and evolving emotional state. The simulator emits an emotion score $$\nabla_\theta J(\theta) = \mathbb E_{q,o} \left[ r(q,o)\nabla_\theta\log\pi_\theta(o\mid q) \right].$$3, with terminal reward

$$\nabla_\theta J(\theta) = \mathbb E_{q,o} \left[ r(q,o)\nabla_\theta\log\pi_\theta(o\mid q) \right].$$4

PPO and GRPO train Qwen2.5-7B-Instruct in thinking and non-thinking formats. The best reported PPO-thinking model raises the Sentient Benchmark score from 13.3 to 79.2, while mathematical and coding competence remains broadly intact. The reward is verifiable in an engineering sense because the simulator procedure is fixed and reproducible, not because emotional quality has become objectively measurable. Simulator bias, hidden-intention assumptions, cultural variation, reward hacking, and the absence of human safety validation remain substantial limitations (Wang et al., 3 Jul 2025).

4. Credit assignment, variance, and failure utilization

Process-sensitive self-supervision

MR-RLVR addresses the mismatch between terminal rewards and intermediate reasoning without requiring a process reward model. It transforms existing mathematical traces into masked-then-fill and step-reordering tasks. Masked reconstruction removes important formulas, theorem names, inference steps, or entities and rewards recovery using MathRuler and textual similarity. Step reordering permutes logical steps and rewards reconstruction of their reference order.

Training has two stages. Stage I uses GRPO with process-level self-supervised rewards. Stage II applies ordinary outcome-only RLVR to computational mathematics with deterministic final-answer verification. The method improves Pass@1, Pass@5, and Pass@8 relative to ordinary RLVR on reported AIME, AMC, and MATH500 experiments. The process rewards remain proxies: reconstruction and ordering agreement do not prove mathematical validity and may favor reference-solution structure (Wang et al., 21 Nov 2025).

RISE trains solution generation and self-verification simultaneously. The model first generates on-policy solutions, then receives verification prompts containing its own candidate responses and generates critiques with predicted scores. Both solution and verification trajectories are optimized through PPO. Verification data are online because they are constructed from current-policy solutions. Online verification produces substantially stronger self-verification than offline examples generated by a distant policy. The framework improves both mathematical accuracy and self-verification, but its labels remain tied to deterministic outcome and formatting checks; successful final answers do not guarantee valid intermediate reasoning (Liu et al., 19 May 2025).

CIPO uses failed trajectories as correction contexts. Given a candidate $$\nabla_\theta J(\theta) = \mathbb E_{q,o} \left[ r(q,o)\nabla_\theta\log\pi_\theta(o\mid q) \right].$$5, the model receives the original problem and candidate without being told whether it is correct, then generates a correction $$\nabla_\theta J(\theta) = \mathbb E_{q,o} \left[ r(q,o)\nabla_\theta\log\pi_\theta(o\mid q) \right].$$6. The correction is verified with the same automatic verifier. CIPO jointly optimizes ordinary RLVR responses and correction-conditioned responses, with difficulty-aware replay, adaptive mixing of successful and failed candidates, and risk-averse shaping that assigns a penalty when a correct candidate is transformed into an incorrect correction. Experiments with Qwen3-4B and Seed-Coder-8B show gains in mathematical reasoning, code generation, correction, critique, and pass@$$\nabla_\theta J(\theta) = \mathbb E_{q,o} \left[ r(q,o)\nabla_\theta\log\pi_\theta(o\mid q) \right].$$7. The method does not provide token-level error labels and requires additional correction rollouts (Ren et al., 14 May 2026).

CVT-RL extends credit assignment to long-horizon agents with policy-conditioned counterfactual contribution. It applies deletion, semantic substitution, evidence substitution, and tool-output perturbation interventions, then samples continuations from a frozen policy. A selection-adjusted doubly robust estimator combines an outcome model with continuation-policy importance ratios. Validity gating rejects malformed or out-of-distribution interventions. The resulting PCCC signal is used with dense answer, evidence, belief, meta, and anti-hacking rewards under constrained trust-region optimization. On long-context QA, ALFWorld, ScienceWorld, and web/tool tasks, CVT-RL improves task success and evidence F1 while reducing measured and human-audited hacking relative to compute-matched and information-matched baselines. The estimand is conditional on selected interventions, histories, validity gates, and a frozen continuation policy; it is not an unconditional causal effect of an action under the evolving policy (Meng, 3 Jun 2026).

Variance reduction

Group-based RLVR methods estimate prompt values from a small number of responses. If a prompt has true success probability $$\nabla_\theta J(\theta) = \mathbb E_{q,o} \left[ r(q,o)\nabla_\theta\log\pi_\theta(o\mid q) \right].$$8, the empirical mean based on $$\nabla_\theta J(\theta) = \mathbb E_{q,o} \left[ r(q,o)\nabla_\theta\log\pi_\theta(o\mid q) \right].$$9 generations has conditional variance $\pi_{\mathrm{ref}}$0. This can be substantial when $\pi_{\mathrm{ref}}$1, especially for binary rewards near success probability $\pi_{\mathrm{ref}}$2.

Shrinkage baselines combine a leave-one-out prompt mean with a leave-one-prompt-out batch mean:

$\pi_{\mathrm{ref}}$3

The coefficient is estimated from within-prompt sampling variance and across-prompt dispersion. Leave-one-out construction preserves unbiasedness because the baseline excludes the reward of the response whose gradient it weights. The method requires no critic network or additional trainable parameters. Experiments with GRPO and related methods show lower value-estimation mean squared error, lower measured gradient variance, and accuracy gains particularly in low-generation regimes. Shrinkage is less useful when prompt heterogeneity is large or when each prompt already has many samples (Zeng et al., 5 Nov 2025).

Token-level credit

DelTA interprets the response-level policy-gradient update as an implicit discriminator in token-gradient space. Let

$\pi_{\mathrm{ref}}$4

The local RLVR direction is

$\pi_{\mathrm{ref}}$5

Positive and negative responses form advantage-weighted token-gradient centroids. A candidate token’s probability increases when its gradient aligns more strongly with the positive centroid than with the negative centroid. DelTA refines these centroids using discriminative token assignments, maps the assignments to bounded coefficients in $\pi_{\mathrm{ref}}$6, normalizes their average to one, and applies them to a clipped DAPO objective. The implementation uses a last-layer hidden-gradient proxy rather than full-parameter token gradients.

On Qwen3-8B and Qwen3-14B, DelTA improves average performance over same-scale baselines on seven mathematical benchmarks; it also improves code-generation and out-of-domain evaluations. Its interpretation is that standard sequence-level RLVR can be dominated by shared formatting and high-frequency directions, whereas discriminative token weighting emphasizes directions that distinguish successful from unsuccessful reasoning. The method remains a proxy-based token-credit mechanism rather than formal process supervision (Zhang et al., 20 May 2026).

5. Stability, noise, and temporal policy improvement

Noisy verifiers

RLVR performance depends not merely on verifier accuracy but on its discriminative direction. Let

$\pi_{\mathrm{ref}}$7

and define Youden’s index

$\pi_{\mathrm{ref}}$8

In a mean-field GRPO model, $\pi_{\mathrm{ref}}$9 determines the phase of learning:

$$J(\theta) = \mathbb E[r(q,o)] - \beta D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{ref}}),$$0

When $$J(\theta) = \mathbb E[r(q,o)] - \beta D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{ref}}),$$1, incorrect probability mass is driven toward extinction; when $$J(\theta) = \mathbb E[r(q,o)] - \beta D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{ref}}),$$2, the expected drift vanishes; when $$J(\theta) = \mathbb E[r(q,o)] - \beta D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{ref}}),$$3, incorrect modes amplify. Thus verifier noise primarily changes the convergence rate while the verifier remains better than chance, but it changes the outcome when false positives exceed true-positive advantage. Experiments on programming tasks with synthetic verifier noise reproduce the predicted boundary. KL regularization can create an interior equilibrium and prevent total collapse, but it cannot turn a negatively informative verifier into a positive one (Rad et al., 7 Jan 2026).

Safety and KL-constrained optimization

The safety implications of RLVR depend on the relation between task success and unsafe behavior. In an ideal finite-path model, KL-regularized RLVR exponentially tilts the reference trajectory distribution according to path-conditioned success:

$$J(\theta) = \mathbb E[r(q,o)] - \beta D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{ref}}),$$4

If path-conditioned success and safety are statistically independent under the reference distribution, expected safety is preserved exactly. More generally, safety drift is bounded by a covariance between safety and reward-induced reweighting, or by the square root of the chi-square divergence from the reference policy.

Experiments comparing RLVR and SFT across adversarial safety benchmarks report negligible aggregate safety changes for evaluated mathematical and coding RLVR models, while the evaluated SFT models show substantial harmfulness increases. These results are conditional on the training recipes, verifiers, models, automated evaluators, and benchmarks. They do not establish universal safety preservation. Reward–safety correlation, rare-event behavior, verifier flaws, and distribution shift can all invalidate the independence intuition (Cho et al., 26 Nov 2025).

Temporal verification of policy updates

PIRL reframes RLVR around policy improvement rather than instantaneous group reward. Define

$$J(\theta) = \mathbb E[r(q,o)] - \beta D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{ref}}),$$5

The cumulative objective telescopes:

$$J(\theta) = \mathbb E[r(q,o)] - \beta D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{ref}}),$$6

Thus, when the initial policy is fixed, maximizing cumulative true improvement is equivalent to maximizing final task performance. PIPO operationalizes this idea by evaluating whether a previous update improved performance relative to a sliding historical window. A standardized improvement signal gates retrospective reinforcement: beneficial updates are strengthened, while harmful updates are suppressed.

PIPO is implemented on top of GRPO, GSPO, or DAPO with stored rollout batches, historical performance statistics, importance weighting, clipping, and a bounded rectification range. Experiments on Qwen3-4B and Qwen3-8B show improved Pass@1, smoother learning curves, and fewer gradient-norm spikes. The method adds approximately 12–19% per-step latency, with reported gains in wall-clock sample efficiency. Its theoretical guarantees require faithful performance estimation and directional consistency between observed improvement and the underlying gradient; finite sampling and nonstationarity can violate these assumptions (Wang et al., 1 Apr 2026).

Classification-style optimization

REAL revisits binary verifier rewards as categorical labels rather than scalar advantage weights. GRPO can assign weak updates to difficult but correct rollouts because their relative likelihood has decreased, while high-probability incorrect rollouts can receive disproportionately large negative weights. REAL converts length-normalized relative log-probabilities into sequence-level classification logits, introduces an anchor logit $$J(\theta) = \mathbb E[r(q,o)] - \beta D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{ref}}),$$7, and optimizes a groupwise classification loss. Positive rollouts are encouraged above zero and negative rollouts below zero.

The induced gradient weighting is monotonic in the desired direction and bounded by $$J(\theta) = \mathbb E[r(q,o)] - \beta D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{ref}}),$$8. REAL improves Pass@1 over GRPO, DAPO, GSPO, and related baselines on reported mathematical benchmarks at 1.5B and 7B scales. Its limitations include binary reward dependence, degenerate all-positive or all-negative groups, sequence-level rather than token-level credit, and limited evaluation beyond mathematical reasoning (Zhai et al., 5 Feb 2026).

6. Scope, limitations, and future directions

RLVR is strongest when a task admits a reliable, scalable, externally checkable criterion. Mathematics, code execution, symbolic manipulation, structured outputs, procedural environments, and constrained games satisfy this condition relatively well. Procedural generators can provide fresh data, adjustable difficulty, and reproducible evaluation. Reference-based, model-based, and simulated-user methods extend the paradigm to broader domains, but they replace strict correctness with criterion-relative verification.

Several limitations recur across RLVR systems.

Verifier dependence: A verifier can be deterministic yet wrong, incomplete, biased, or exploitable. Finite unit tests create false positives and false negatives. Reference-based systems inherit errors and preferences from references and teacher models. Simulated-user environments reproduce the biases of their prompts and evaluator models.

Sparse and coarse rewards: Terminal binary rewards do not identify which reasoning steps were useful. Process-oriented approaches such as MR-RLVR, RISE, CIPO, DelTA, and CVT-RL introduce denser or more selective signals, but each provides only a proxy for reasoning quality. Reconstruction, self-verification, correction success, token-gradient discriminativeness, and counterfactual contribution are not interchangeable with full logical verification.

Reward hacking and specification gaming: Models can exploit formatting, lexical overlap, reference phrases, test-suite gaps, final-step evaluation, simulator conventions, or evaluator weaknesses. Improvements in benchmark scores may therefore reflect optimization of the encoded criterion rather than broad competence. RLVRR, CVT-RL, procedural environments, and safety constraints mitigate particular forms of exploitation but do not eliminate specification problems.

Finite sampling and distribution shift: Group statistics are noisy when the number of generations per prompt is small. Shrinkage baselines reduce estimation variance, but cross-prompt pooling can introduce bias under heterogeneous batches. PIPO’s historical baselines can lag, and correction or counterfactual systems depend on current-policy distributions, intervention validity, and overlap.

Policy collapse and diversity: GRPO dynamics can amplify rare successes, while noisy or anti-informative verifiers can amplify incorrect modes. Correct-mode winner-take-all behavior can reduce diversity even when aggregate success improves. Pass@$$J(\theta) = \mathbb E[r(q,o)] - \beta D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{ref}}),$$9, entropy, Self-BLEU, and multi-sample performance are therefore important complements to Pass@1.

Safety qualification: KL regularization and reward–safety independence can constrain or eliminate expected safety drift under idealized assumptions, but average divergence and aggregate harmfulness do not certify rare-event safety. High-stakes deployment requires adversarial testing, human review, detector validation, and monitoring for unsafe tool use, unsupported evidence, privacy leakage, and distribution shift.

Open-ended generation: VMR, RLVRR, CER, and reference-based generative verifiers show that RLVR can be reframed for tasks without unique answers. Yet these methods verify preference comparisons, reference-derived concepts, style predicates, or model-relative likelihoods rather than objective truth. Their performance depends on reference quality, pair construction, extracted criteria, evaluation alignment, and resistance to superficial shortcuts.

The emerging direction of RLVR research is consequently multidimensional. Verifiers are becoming procedural, reference-conditioned, generative, simulated, counterfactual, and internally model-based. Optimization is moving from scalar reward weighting toward shrinkage baselines, classification losses, discriminative token credit, correction-conditioned learning, and retrospective policy-improvement feedback. Process and causal signals are being combined with terminal verification rather than treated as complete substitutes for it.

A plausible implication is that robust RLVR systems will require a verifier stack rather than a single reward: externally checkable outcome rewards, calibrated uncertainty, process-sensitive diagnostics, failure-conditioned correction, variance-aware baselines, temporal regression detection, and explicit safety constraints. The central unresolved problem is to preserve the scalability and auditability of verifiable rewards while ensuring that the property being optimized remains aligned with correctness, reasoning validity, safety, and generalization rather than merely with the behavior of the verifier itself.