RL with Verifiable Rewards

• RLVR is a paradigm that uses objectively verifiable reward signals, such as binary tests or model-based probabilities, to guide reinforcement learning.
• It integrates techniques like PPO and REINFORCE with reference comparisons to ensure alignment without extensive supervised fine-tuning.
• RLVR scales across structured and ambiguous domains, yielding measurable improvements over traditional supervised approaches in tasks like math, code, and creative writing.

Reinforcement Learning with Verifiable Rewards (RLVR) is a paradigm in which reinforcement learning agents—particularly LLMs and vision–LLMs (VLMs)—are optimized using reward signals that are objectively and transparently verifiable. Unlike scalar reward signals based on subjective model preferences or extrinsic demonstrations, RLVR employs reward functions that compare generated outputs against reference answers, executable tests, or task-specific criteria, producing binary or continuous rewards grounded in explicit verification. This framework has proven effective in structured domains such as mathematics and code, and recent research has extended it to diverse and less-structured settings including medicine, psychology, dialogue, robotics, and remote sensing.

1. Core Principles and Reward Architectures

RLVR frameworks are unified by the principle of leveraging reward signals derived from direct, objective verification of an agent’s output. The classic setting is mathematical reasoning or programming, where correct outputs can be checked using ground-truth answers or test cases (e.g., r(y) = 1 if output y passes all tests; otherwise 0). The significance of RLVR lies in its ability to train models without the need for extensive supervised fine-tuning, instead relying on the alignment between model outputs and externally verifiable criteria.

Binary verification remains effective in domains with structured outputs, but presents limitations for free-form, narrative, or ill-structured answers. To address these limitations, recent work has introduced generative reward models, which deliver soft, model-based reward signals. In these settings, the reward is defined as a probability produced by a verifier model (e.g., πₚ(1 | x, a, y) for “correct,” where πₚ is the policy of the verifier, x is the input, a is an expert-written reference, and y is the candidate answer). Such model-based or token-level rewards provide finer-grained gradients crucial for domains with ambiguous or open-ended outputs.

A high degree of cross-model consistency is reported for binary verification in settings with structured references (Cohen’s κ > 0.86–0.88 between leading LLM verifiers), allowing distilled smaller models (e.g., a 7B LLM) to serve as practical reward models for RL training without extensive domain-specific supervision.

2. Training and Optimization Methodologies

RLVR leverages reinforcement learning algorithms such as REINFORCE, PPO, Group Relative Policy Optimization (GRPO), or novel domain-specific variants. The central policy optimization step incorporates the verifiable reward into the policy gradient update:

$$J(\theta) = \mathbb{E}_{(x,a) \sim D} \mathbb{E}_{y \sim \pi_{\theta}(\cdot|x)} [r_{\phi}(x, a, y)]$$

$$\nabla_{\theta} J(\theta) = \mathbb{E}_{(x,a)} \mathbb{E}_{y \sim \pi_\theta} [ r_{\phi}(x, a, y) \nabla_\theta \log \pi_\theta(y|x) ]$$

Where $r_{\phi}$ is the reward assigned by the reward model, often with batch z-score normalization and a KL-divergence penalty term to stabilize updates. For model-based rewards, the RL loop typically leverages a distilled verifier (trained on high-agreement data), and supports both binary and soft reward forms, the latter represented by the confidence of the verifier’s decision.

Batch-normalization as well as KL-divergence penalty help to maintain policy stability and regularization. In practice, models are trained using group sampling—multiple outputs are generated per prompt, and policy updates are made by comparing rewards within these groups.

In free-form and ambiguous domains, experiments have shown that soft, model-based reward signals outperform hard binary verification, especially as dataset scale increases.

3. Cross-Domain Experiments and Empirical Outcomes

Debate regarding RLVR’s generalizability is addressed in recent empirical studies across mathematics, multi-subject QA (medicine, psychology, economics, education), writing, and vision-language tasks:

• In mathematics, RLVR-trained policies using model-based reward models yielded up to 8% absolute improvements over supervised fine-tuning (SFT) baselines and rule-based rewards.
• In broad-domain QA tasks (ExamQA), distilled 7B reward models provided significant gains in free-form answer settings, demonstrating that small verifiers can generalize well if distilled from sufficiently strong LLMs.
• Model-based, soft reward schemes adapt gracefully to ambiguous or diverse references, offering conservative signals where binary verifiers would introduce noise or degrade with scale.
• In creative writing, pairwise generative reward models and bootstrapped reference-free policy optimization (BRPO) bridge the gap for non-verifiable tasks, transforming subjective comparison into reliably verifiable, dynamic reward signals. These strategies resist reward hacking artifacts found in scalar reward-based training regimes.
• In vision-language and remote sensing tasks, even one-shot RLVR training with a handful of rule-verified examples is shown to deliver double-digit gains, with 128-shot models matching or exceeding large-scale supervised baselines.

The robustness of RLVR extends to data-synthesis regimes, where pipelines such as SynthRL generate more challenging, guaranteed-verifiable training samples, enhancing reasoning steps and out-of-domain bench performance.

4. Scalability and Robustness

Scaling RLVR to larger and less structured domains surfaces several findings:

• When reward models are distilled from larger, high-quality LLMs, small verifiers can scale to hundreds of thousands of training cases with minimal degradation.
• Binary verification works reliably if there is an expert reference, but classic rule-based verifiers’ performance can degrade at scale due to their inability to handle ambiguous, narrative answers.
• Model-based and soft reward signals show performance that continues to grow with additional data, supporting scalability in both sample size and domain coverage.
• RLVR is robust to moderate label noise, as soft generative verifiers can produce meaningful gradients even in the presence of reference ambiguity.
• The combination of reward normalization, KL regularization, and batch sampling ensures stable learning across long training runs and diverse domains.

5. Limitations, Open Questions, and Future Directions

RLVR’s extension to noisy and weak-label settings highlights a set of technical challenges and natural research directions:

• Incorporation of more subtle similarity metrics (e.g., sentence embedding cosine similarity) may further enhance reward signal quality for ambiguous references.
• Sequential or process-level reward modeling (e.g., process consistency filters) could be used to evaluate multi-step reasoning, especially where the reasoning path, not just the outcome, must be verified.
• Improving reward model calibration, especially in domains where expert references are noisy or incomplete, remains an open issue—there is a risk of reinforcing superficial patterns unless process-level verification is also enforced.
• There is ongoing work on harmonizing process and outcome rewards to better capture reasoning correctness at both the stepwise and outcome levels, as naive reward blending is susceptible to reward hacking.
• Further work will explore process-level, outcome-level, and dynamic reward models in settings such as chain-of-thought prompting, multimodal tasks, and open-ended dialogue.

A plausible implication is that RLVR, equipped with robust, cross-domain generative reward models and scalable learning algorithms, constitutes an effective and unified paradigm for realistic, scalable RL applications, where annotation is bottlenecked, references are noisy, and correctness is not always strictly defined.

6. Summary Table: Key Features of RLVR in Extension to Broad Domains

Feature	Structured Domains	Unstructured/Broad Domains
Reward Form	Binary, rule-based	Model-based (soft), generative
Reference Requirement	Explicit, clear answers	Expert-written, possibly noisy
Verifier Model	Rule or off-the-shelf LLM	Distilled LLM as reward model
Primary Challenge	Verification is trivial	Verification is ambiguous
Scalability	Usually robust	Scalable if using soft rewards
Reported SOTA Improvement	up to +8.0% over SFT/rule	Outperforms SOTA LLM baselines

7. Implications and Practical Relevance

The capacity to extend RLVR frameworks from strictly verifiable, structured tasks to broad, real-world domains—via generative, cross-domain reward models—is an important development, particularly for practical RL applications where high-quality annotation is infeasible and label correctness is ambiguous or only partially verifiable. RLVR’s robustness, scalability, and flexibility position it as a primary candidate for deploying RL systems in complex, realistic, and noisy-label scenarios spanning scientific domains, education, healthcare, and beyond (Su et al., 31 Mar 2025).