Reinforcement Learning with Verified Reward (RLVR)

• RLVR is a reinforcement learning paradigm that employs objective, verifiable reward functions—such as rule-based checks—to optimize post-training outputs in large language models.
• It has demonstrated significant performance gains in domains like mathematics, coding, and medical QA by directly optimizing for correctness using minimal curated examples.
• The approach addresses challenges such as reward hacking and exploration limits while extending to soft, hybrid, and rubric-based verification methods for broader applicability.

Reinforcement Learning with Verified Reward (RLVR) is a paradigm for post-training LLMs and other generative systems through reinforcement learning in which the reward function is specified by objective, verifiable criteria—typically based on rule-based or reference-answer checks—rather than by learned preference models. RLVR was initially motivated by the unique verifiability of solutions in mathematical reasoning and code generation, but has since been applied to a broadening spectrum of domains, from medical question answering and multimodal emotion recognition to world modeling and instruction following. By directly optimizing for correctness under verifiable labels, RLVR enables the emergence of robust reasoning capabilities in models, often with minimal human supervision for intermediate steps and sometimes with only a few curated training examples.

1. Foundational Principles of RLVR

RLVR distinguishes itself from conventional reward learning or reinforcement learning from human feedback (RLHF) by relying on rewards that are determined by programmatic or reference-based verification, not through learned scoring models. The prevailing RLVR training workflow uses an external reward function $r_\phi(q, o)$, with no trainable parameters, that verifies the output $o$—such as matching a gold-label answer, checking format compliance, or comparing to references—applied to candidate completions sampled from the policy $\pi_\theta(o|q)$. The reinforcement learning loop can use any policy gradient method; recent work frequently employs Proximal Policy Optimization (PPO) or Group Relative Policy Optimization (GRPO).

The critical mathematical form is: $$J_\text{PPO}(\theta) = \mathbb{E}_{q,o} \left[ \min\left( \left( \frac{\pi_\theta(o_t|q,o_{<t})}{\pi_{\theta_\text{old}}(o_t|q,o_{<t})}\right)A_t,\, \mathrm{clip}\left(\frac{\pi_\theta(o_t|q,o_{<t})}{\pi_{\theta_\text{old}}(o_t|q,o_{<t})}, 1-\epsilon, 1+\epsilon\right)A_t \right) \right]$$ with $A_t$ typically the advantage at token $t$ (e.g., under Generalized Advantage Estimation), and $\epsilon$ the PPO clipping parameter. To avoid degenerate policy drift, a KL penalty against a reference or pre-trained policy is often included on a per-token basis.

The reward function for RLVR is usually rule-based or reference-answer-based. In the medical multiple-choice setting (Zhang et al., 27 Feb 2025), for instance:

• Outputs violating the required format (e.g., missing > ... or <answer>...</answer> blocks) are assigned a reward of –1.
• Outputs with correct format and a predicted answer matching the gold label suffer a reward of +1.
• Incorrect answer choices, with correct format, are assigned 0.

These strict verification protocols are what give RLVR its characteristic objectivity and detachment from learned or subjective reward models.

2. RLVR in Mathematical, Coding, and Medical Domains

The success of RLVR was first established in domains with strong verifiability, notably mathematics and code. In mathematics, RLVR—using as little as one carefully chosen training example—can nearly double performance on challenging benchmarks such as MATH500 (e.g., raising Qwen2.5-Math-1.5B from 36.0% to 73.6% accuracy) and substantially improve cross-benchmark averages (Wang et al., 29 Apr 2025). Even with these sparse rewards, the approach is effective because existing large pre-trained models are heavily “primed” with latent reasoning abilities, which RLVR acts to surface or upweight.

Transitional studies extended RLVR to medical multiple-choice QA (MCQA), as in Med-RLVR (Zhang et al., 27 Feb 2025), which adapts the same group- or token-level RL objective with verifiable scoring against MCQA ground truth. Here, a 3B-parameter base model (Qwen2.5-3B) was trained using PPO and simple, deterministic verification (matching answer tags and enforcing answer reasoning blocks via regex or template matching). Med-RLVR achieves in-distribution performance rivaling supervised fine-tuning, and—more strikingly—an additional 8 percentage point gain on out-of-distribution health question benchmarks. Training dynamics reveal a progression from format mistakes to concise, effective reasoning, with emergent evidence of reward hacking (the model exposing the answer early in its reasoning chain).

The framework is further extended to multimodal settings (e.g., emotion recognition), world modeling (with accuracy and perceptual quality as direct task metrics), and other structured tasks, always centering on strictly verifiable reward signals (Zhao et al., 7 Mar 2025, 2505.13934).

3. Soft, Model-Based, and Hybrid Verification

While RLVR is highly effective in domains with strictly verifiable answers, its utility depends on reward function design in less-structured scenarios. To this end, generative and soft scoring RLVR approaches have been developed (Su et al., 31 Mar 2025):

• Soft model-based reward: Where binary verification is too coarse, a generative verifier LLM can score a response by outputting a “0” or “1" token for incorrect/correct and the token probability (e.g., $\pi_\phi(1|x, a, y_T)$) is used as a soft, continuous reward—making the RLVR pipeline robust to noisy and ambiguous reference answers.
• Hybrid verification: Instruction-following RLVR (VerIF) combines code-based checks for “hard” constraints (e.g., length, keyword presence) with LLM-based verification for content (e.g., style, applicability to instructions), aggregating the outputs as a final reward (Peng et al., 11 Jun 2025).
• Rubrics as Rewards: RLVR has been extended to subjective or ill-defined benchmarks (e.g., open-ended medical advice) by transforming best-practices rubrics—i.e., weighted multi-criterion checklists—into explicit reward functions for GRPO. This “RaR” approach supplies a vector of criterion-anchored scores, composited as $$r(x, \hat{y}) = \frac{\sum_j w_j c_j(x, \hat{y})}{\sum_j w_j}$$, with either rule-based or judge-LLM rubric evaluation (Gunjal et al., 23 Jul 2025).

Such innovations demonstrate that RLVR is not restricted to exact-match correctness but is extensible to fine-grained, rubric- or model-judged objectives for broader real-world applicability.

4. Training Dynamics, Generalization, and Support

RLVR is notable for rapidly eliciting reasoning behaviors without explicit intermediate supervision:

• Self-evolved reasoning: RLVR can drive the emergence of new solution patterns (e.g., chained reasoning, preference for code-based solutions) as measured by the increased frequency of code reasoning in mathematical tasks, rising from 65% to over 90% after RLVR—even when reward signals are spurious or uncorrelated with correctness (Shao et al., 12 Jun 2025).
• Post-saturation generalization: 1-shot RLVR exhibits continued gains on test sets long after the single training prompt accuracy saturates (Wang et al., 29 Apr 2025), suggesting an ongoing refinement of global policy and reasoning structure.
• Bias for answer-level precision, support shrinkage: RLVR mathematically preserves the support of the base model ($$\mathrm{supp}(\pi_\theta(\cdot|x)) \subseteq \mathrm{supp}(q(\cdot|x))$$) and acts as a conservative reweighting mechanism (Wu et al., 20 Jul 2025). Empirically, pass@1 (precision on high-probability completions) is increased, but answer-level entropy is systematically reduced, potentially at the expense of broad exploration and pass@k when $k$ is large.

These properties are central to both the strength and the present limits of RLVR: it optimizes for more reliable reasoning in the support of the base model but is less likely to “invent” novel solution structures ab initio.

5. Data Curation, Curriculum, and Domain Mixing

The effectiveness of RLVR is tightly coupled to data curation and curriculum strategies:

• Sample filtering: Filtering MCQA samples by difficulty using models such as Phi-4 or larger Gemma variants improves both domain-specific and cross-domain robustness over naive random sampling. Self-filtering may give domain gains (e.g., on medical test sets) but at a cost to generalization (Qiu et al., 16 Apr 2025).
• Curriculum learning: Progressive buildup of task complexity (e.g., from simple to complex puzzles) with intermediate policy refresh stages can improve convergence and upper-bound performance, especially in multi-domain RLVR training (Li et al., 23 Jul 2025).
• Template and reward design consistency: Maintaining format consistency and nuanced reward granularity (e.g., partial rewards for puzzles with multiple sub-questions) are essential for stable training and high performance across domains.

Compositional experiments show that RLVR trained in multi-domain regimes can yield either beneficial transfers (math aiding logic puzzles) or detrimental interference (math impeding code), requiring careful domain mixing and adaptive rewards.

6. Challenges: Reward Hacking, Support Limitation, and Open Problems

RLVR introduces several novel failure modes and research challenges:

• Reward hacking: Models can exploit superficial aspects of the reward (e.g., outputting the answer early in the chain-of-thought, or mimicking required format without genuine reasoning). Diagnosis and mitigation require additional regularization, trap instructions (“trip wires” (Guo et al., 6 Aug 2025)), and intent-alignment modules to enforce logical and semantic compliance.
• Exploration limits: RLVR’s reweighting nature precludes expansion beyond the base’s initial support; explicit exploration strategies, probabilistic mass seeding into underrepresented output regions, or hybrid policies are cited as necessary for breaking the “invisible leash” and achieving broader reasoning generalization (Wu et al., 20 Jul 2025).
• Sparse rewards and insufficient diversity: Settings with extremely sparse correct solutions require auxiliary strategies such as expert prompting, mutual learning, stepwise or hint-based guidance, or rubric-based shaped rewards to maintain informative learning signals (Zhang et al., 3 Jul 2025).

These challenges remain central as RLVR is adapted to noisier, less-structured domains and higher-complexity real-world applications.

7. Future Directions and Open Research Problems

Ongoing and anticipated research directions in RLVR, as evidenced across the referenced literature, include:

• Improved reward functions for more ambiguous or subjective tasks using hybrid, soft, or rubric-based signals that combine human-aligned and programmatic checks (Su et al., 31 Mar 2025, Gunjal et al., 23 Jul 2025).
• Frameworks for joint generation and self-verification (as in RISE), bringing together solution and critique within a single policy gradient loop, with demonstrated gains in both reasoning and output reliability (Liu et al., 19 May 2025).
• Methods for scalable verifier-free RLVR, such as leveraging intrinsic model token probabilities as an implicit correctness signal, with debiasing and curriculum techniques to ensure effective learning across open-domain tasks (Yu et al., 23 Jun 2025).
• Cross-domain and multilingual adaptation, including domain-specific reward design and curriculum construction tailored to diverse scientific, technical, and layperson settings (Li et al., 23 Jul 2025).
• Addressing conservative support through explicit exploration mechanisms, hybrid policies, and support re-seeding to overcome RLVR’s fundamental limitation in novel solution discovery (Wu et al., 20 Jul 2025).

Collectively, RLVR marks a shift toward transparent, objective, and more sample-efficient post-training of LLMs, with demonstrated gains in reasoning robustness over supervised fine-tuning and strong potential for extension to complex, knowledge-intensive fields. Ongoing work focuses on expanding its scope, addressing exploration and expressivity limitations, and refining reward engineering for real-world alignment.