VeriRL: Verification-Based Reinforcement Learning

• VeriRL is an umbrella term for RL methods that integrate verification signals—such as rule-based rewards, testbench execution, or formal properties—to guide learning.
• It treats verifiability as a design element, using outputs from verifiers to shape reward functions and optimize policy updates in diverse applications.
• Innovations like Trace-back Rescore, Sample-balanced Weighting, and verifier-free curricula illustrate practical techniques to improve training stability and performance.

VeriRL is used in more than one sense in the recent literature. In one usage, it denotes verification-based reinforcement learning for LLM reasoning, where a verifier supplies the reward used by algorithms such as GRPO rather than human preference labels (Xu et al., 20 May 2025). In another, it appears as VERIRL, a domain-specific reinforcement-learning framework for Verilog code generation, where correctness is judged by compilation, simulation, and testbench execution (Teng et al., 25 Aug 2025). Closely related strands use reinforcement learning to improve verification workflows for digital designs (Shibu et al., 2021), and, in the opposite direction, use formal methods to verify deep reinforcement learning policies themselves; for that latter meaning, whiRL 2.0 is described as the closest match if “VeriRL” is intended as a generic label for DRL verification tooling (Amir et al., 2021). The term therefore denotes a family of verification-and-RL interactions rather than a single standardized method.

1. Terminological scope and conceptual boundaries

In the cited literature, “VeriRL” is best treated as an umbrella term spanning three technically distinct patterns. The first is verifier-produced reward learning, in which a rule-based or otherwise external verifier maps a prompt and a sampled response to a scalar reward, and reinforcement learning optimizes the policy under that signal. The second is RL for verification, in which an RL agent learns to generate stimuli, search queries, or other actions that improve coverage or evidence acquisition inside a verification workflow. The third is verification of RL systems, in which the policy and its environment are modeled as a transition system and subjected to bounded model checking, k-induction, abstraction, and invariant inference.

These directions differ in where verification enters the loop. In verification-based RL for LLM reasoning, verification is the reward function itself (Xu et al., 20 May 2025). In VERIRL for Verilog, execution-based checking generates rewards and also supports reward-model training (Teng et al., 25 Aug 2025). In VeRLPy, RL is not trained from a correctness verifier over generated text; instead, the design-under-test becomes the environment and coverage events define reward (Shibu et al., 2021). In whiRL 2.0, the object of study is the learned DRL controller, and the task is to prove or refute safety and liveness properties of its sequential behavior (Amir et al., 2021).

This taxonomy suggests that “VeriRL” should not be read as a single algorithmic recipe. It is more accurately a research area organized around different placements of verifiability: as reward, as search objective, or as formal specification.

2. VeriRL as verification-based reinforcement learning for LLM reasoning

In the LLM-reasoning usage, verification-based reinforcement learning treats the verifier as the source of task reward. TinyV gives the canonical formulation: a verifier $\psi$ maps a prompt-response pair to a binary signal,

$$r_i=\psi(\mathbf{x},\mathbf{y}_i,\mathbf{y}_{\text{ref}})\in\{0,1\},$$

with $r_i=1$ when the generated answer is judged equivalent to the reference answer, and the policy is trained under

$$\max_{\theta}\ \mathbb{E}_{\mathbf{y}\sim\pi_\theta(\cdot\mid \mathbf{x})}\left[r(\mathbf{x},\mathbf{y})-\beta D_{\mathrm{KL}}\!\left(\pi_\theta(\mathbf{y}\mid \mathbf{x})\Vert \pi_{\text{init}}(\mathbf{y}\mid \mathbf{x})\right)\right].$$

In this sense, the verifier is not an auxiliary evaluator but the effective reward model of the RL loop (Xu et al., 20 May 2025).

The same paper emphasizes that GRPO-style training is especially sensitive to whether prompts yield mixed outcomes across rollouts. It defines prompt efficiency

$$\eta_k=P_k(0<p(\mathbf{x})<1)=1-P_k(p(\mathbf{x})=0)-P_k(p(\mathbf{x})=1),$$

where $p(\mathbf{x})=\frac{1}{n}\sum_{i=1}^n r_i$ is the pass rate across $n$ rollouts. Prompts with $p(\mathbf{x})=0$ or $p(\mathbf{x})=1$ provide little useful gradient information, whereas prompts with $0<p(\mathbf{x})<1$ are informative for policy updates (Xu et al., 20 May 2025).

This framing has become increasingly broad. VideoRLVR adapts the same verifier-centered logic to video diffusion and flow-matching models by defining video reasoning as generation of a verifiable visual trajectory $$r_i=\psi(\mathbf{x},\mathbf{y}_i,\mathbf{y}_{\text{ref}})\in\{0,1\},$$0 conditioned on an initial image and instruction, with a task-specific verifier-derived reward $$r_i=\psi(\mathbf{x},\mathbf{y}_i,\mathbf{y}_{\text{ref}})\in\{0,1\},$$1 (Zhu et al., 14 May 2026). Veri-R1 applies online RL to claim verification, where the model learns a trajectory

$$r_i=\psi(\mathbf{x},\mathbf{y}_i,\mathbf{y}_{\text{ref}})\in\{0,1\},$$2

consisting of a plan, search actions, retrieved information, intermediate reasoning, and a final answer, and receives reward components for format, evidence overlap, label correctness, and validity weighting (He et al., 2 Oct 2025).

A plausible implication is that the unifying property of modern VeriRL systems is not any particular architecture or benchmark family, but the existence of a task-specific verifier whose outputs can be embedded into an RL objective. That verifier may check symbolic equality, unit tests, search-grounded evidence, or rule consistency in generated videos.

3. Verifier quality, noise, and security as central determinants of learning

A recurrent finding in this literature is that verifier quality is not a secondary engineering concern. TinyV reports that on Big-Math-RL-Verified, 87K / 226K seemingly incorrect prompt-response pairs were actually correct, corresponding to 38.5% false negatives, and that across 95K unique prompts, correct answers were missed for 40K prompts (42.1%); in some sources, especially CN_K12, false-negative rates exceed 50% (Xu et al., 20 May 2025). The paper argues that such false negatives damage credit assignment because correct answers receive reward $$r_i=\psi(\mathbf{x},\mathbf{y}_i,\mathbf{y}_{\text{ref}})\in\{0,1\},$$3, reduce the fraction of prompts with mixed outcomes, and slow convergence. During RL, the fraction of prompts that encounter at least one false-negative rollout grows to 46.7% for Qwen2.5-7B and 50.5% for Qwen2.5-Math-7B, with false-negative rollouts averaging about 20% (Xu et al., 20 May 2025).

TinyV addresses this by augmenting a rule-based verifier with a lightweight LLM-based fallback. Prime Verifier first checks the response; if it rejects the answer, TinyV is queried to determine whether the rejected answer is actually a false negative. TinyV is trained as a binary classifier on prompt, ground truth, and model answer, using 638,000 instances in total and a balanced 159,000-example subset for SFT, with Qwen2.5-1.5B-Instruct as the base model. The reported gains include up to 10% absolute gain on HardVerify-Math, faster convergence, and only about 6% computational overhead (Xu et al., 20 May 2025).

Rate or Fate? RLV$$r_i=\psi(\mathbf{x},\mathbf{y}_i,\mathbf{y}_{\text{ref}})\in\{0,1\},$$4R provides a complementary dynamical account of noisy verification. It models verifier noise through false negatives and false positives,

$$r_i=\psi(\mathbf{x},\mathbf{y}_i,\mathbf{y}_{\text{ref}})\in\{0,1\},$$5

and shows that the decisive scalar is Youden’s index

$$r_i=\psi(\mathbf{x},\mathbf{y}_i,\mathbf{y}_{\text{ref}})\in\{0,1\},$$6

Its central result is a phase transition: when $$r_i=\psi(\mathbf{x},\mathbf{y}_i,\mathbf{y}_{\text{ref}})\in\{0,1\},$$7, incorrect modes are driven toward extinction; when $$r_i=\psi(\mathbf{x},\mathbf{y}_i,\mathbf{y}_{\text{ref}})\in\{0,1\},$$8, training is neutral; and when $$r_i=\psi(\mathbf{x},\mathbf{y}_i,\mathbf{y}_{\text{ref}})\in\{0,1\},$$9, incorrect modes amplify and dominate. In the positive regime, noise changes the convergence rate rather than the asymptotic attractor, which the paper characterizes as “rate, not fate” (Rad et al., 7 Jan 2026).

The security literature shows that even a correct verifier does not eliminate training-time risk. “Backdoors in RLVR” demonstrates that RLVR can be backdoored without modifying the verifier by injecting only 200 poisoned samples, corresponding to less than 2% poisoning rate on average. The attack causes an average 73% safety degradation across jailbreak evaluations while producing only about a 0.55% decrease in benign task performance in the pure backdoor setting (Guo et al., 10 Apr 2026). This attack works by constructing poisoned prompts so that harmful continuations receive positive reward and refusals receive negative reward under the triggered condition, thereby exploiting the RL update rule rather than the verifier implementation itself.

Taken together, these results establish a general principle: in VeriRL, the verifier defines not merely correctness labels but the geometry of the learning signal. False negatives weaken gradient informativeness, false positives can reverse learning direction when $r_i=1$0, and poisoned data can exploit reward asymmetries even when the verifier code is untouched.

4. Methodological extensions: hidden-state control, verifier-free curricula, video reasoning, and competitive programming

Several papers generalize the VeriRL paradigm beyond direct verifier-to-reward pipelines. VERL argues that the familiar exploration-versus-exploitation narrative in RLVR is partly an artifact of token-level measurement. It shifts the analysis to hidden-state space and defines Effective Rank

$r_i=1$1

along with Effective Rank Velocity (ERV) and Effective Rank Acceleration (ERA) as first- and second-order temporal differences over hidden-state statistics. VERL uses ERA as a predictive meta-controller to shape the RL advantage through a dual-channel auxiliary signal over ER and ERV, and reports consistent gains across GRPO and PPO, including up to 21.4% absolute accuracy improvement on Gaokao 2024_I (Huang et al., 28 Sep 2025). The paper’s claim is not merely better performance but a reframing of RLVR optimization as control over semantic trajectories rather than token entropy.

VI-CuRL tackles a different problem: how to stabilize reasoning RL without external verifiers. It defines confidence as the complement of normalized token entropy,

$r_i=1$2

and uses a thresholded curriculum mask $r_i=1$3 with retention rate $r_i=1$4. The curriculum starts at about $r_i=1$5 and anneals toward $r_i=1$6. The paper proves asymptotic unbiasedness of the curriculum objective and gives an exact variance decomposition into action variance, problem variance, and masking variance. Empirically, VI-CuRL turns unstable verifier-free GRPO-like training into a stable method across six math benchmarks, often rivaling verifier-based curricula (Cai et al., 13 Feb 2026).

VideoRLVR extends RLVR to video models by introducing SDE-GRPO, dense decomposed rewards, and Early-Step Focus. In experiments it uses $r_i=1$7 denoising steps and restricts RL to the first $r_i=1$8 steps, reducing training latency by about 40% while maintaining nearly the same performance. On Maze, the full 20-step gradient yields F1 84.6, SR 72.3, time/step 156 s, whereas the early 10-step gradient yields F1 84.4, SR 72.2, time/step 93.5 s (Zhu et al., 14 May 2026). The important conceptual move is that the verifier acts on the decoded video trajectory, but the policy optimization acts on the denoising process that generates it.

In competitive-programming code generation, DRIVE argues that RLVR success depends as much on data curation, entropy expansion, and curriculum design as on the RL algorithm itself. Starting from Qwen2.5-32B-Instruct, it builds an SFT corpus from 1.27M open-source prompts, refines it through 5-round arena learning to 470K high-quality prompts, and then runs a two-stage GRPO pipeline: a broad Stage 1 over about 9K competitive programming prompts with 8 rollouts per prompt and a 24k response window, followed by Pre-GRPO on a hard-focus curriculum with 64 rollouts per prompt and continual retention of the hardest cases (Zhu et al., 9 Nov 2025). This suggests that, in code domains with executable rewards, VeriRL may depend critically on preserving difficult instances rather than averaging them away inside easier prompt distributions.

5. VERIRL for Verilog and RTL code generation

VERIRL, introduced for Verilog code generation, is a specialized reinforcement-learning framework for a domain in which correctness is governed by syntax, concurrency semantics, timing-sensitive behavior, and simulation outcomes rather than surface plausibility (Teng et al., 25 Aug 2025). The paper’s premise is that ordinary text-centric post-training fails because Verilog programs may appear locally reasonable yet still fail compilation or violate hardware behavior under testbenches.

The framework is organized as a four-stage iterative pipeline. A cold-start SFT phase trains a base model such as Qwen2.5-Coder-7B-Instruct on high-quality chain-of-thought-style Verilog examples formatted with explicit <REASON> and <SOLUTION> sections. This is followed by RL under a Reinforce++-style objective augmented with Sample-balanced Weighting (SbW). The improved policy is then used for rejection sampling to harvest stronger reasoning/code pairs for another SFT round, followed by a final RL stage. Policy and reward models co-evolve throughout this process (Teng et al., 25 Aug 2025).

A core enabler is Veribench-53K, curated from over 700K Verilog problems. Construction includes MinHash-based deduplication with a Jaccard threshold of 0.9, compile filtering, and quality screening. Retained problems are rewritten into a structured VerilogEval-human-style prompt format, and 5–10 diverse testbenches per problem are synthesized. The resulting resource supports roughly 240K preference pairs for reward-model training (Teng et al., 25 Aug 2025).

The paper’s two main technical additions are Trace-back based Rescore (TbR) and Sample-balanced Weighting (SbW). TbR samples $r_i=1$9 candidate designs per question, assigns pass-rate rewards $$\max_{\theta}\ \mathbb{E}_{\mathbf{y}\sim\pi_\theta(\cdot\mid \mathbf{x})}\left[r(\mathbf{x},\mathbf{y})-\beta D_{\mathrm{KL}}\!\left(\pi_\theta(\mathbf{y}\mid \mathbf{x})\Vert \pi_{\text{init}}(\mathbf{y}\mid \mathbf{x})\right)\right].$$0, flags solutions with $$\max_{\theta}\ \mathbb{E}_{\mathbf{y}\sim\pi_\theta(\cdot\mid \mathbf{x})}\left[r(\mathbf{x},\mathbf{y})-\beta D_{\mathrm{KL}}\!\left(\pi_\theta(\mathbf{y}\mid \mathbf{x})\Vert \pi_{\text{init}}(\mathbf{y}\mid \mathbf{x})\right)\right].$$1 for regeneration, and recursively refines them for up to two rounds of self-reflection using compiler and testbench feedback. Internal nodes are rescored by the average reward of their child nodes, and preference pairs are accepted only when the reward gap satisfies

$$\max_{\theta}\ \mathbb{E}_{\mathbf{y}\sim\pi_\theta(\cdot\mid \mathbf{x})}\left[r(\mathbf{x},\mathbf{y})-\beta D_{\mathrm{KL}}\!\left(\pi_\theta(\mathbf{y}\mid \mathbf{x})\Vert \pi_{\text{init}}(\mathbf{y}\mid \mathbf{x})\right)\right].$$2

SbW then adaptively reweights policy and KL terms using normalized generation probabilities and reward-model outputs, partitioning samples into high-quality, overfitting, high-capacity, noisy, and normal categories (Teng et al., 25 Aug 2025).

Experimentally, VERIRL-CodeQwen2.5 reaches 69.3% pass@1 and 78.1% pass@5 on VerilogEval-human, 58.2% / 66.0% on RTLLM v1.1, 67.2% / 76.1% on VerilogEval v2, and 63.3% / 70.3% on RTLLM v2 (Teng et al., 25 Aug 2025). The ablations are also specific: a Reinforce++ baseline without TbR or SbW gives 53.2% / 61.5% on VerilogEval v1; adding TbR alone yields 65.3% / 73.3%; adding SbW alone yields 64.7% / 69.2%; combining both produces the final 69.3% / 78.1%. In this setting, “VeriRL” refers not to formal verification of an RL policy, but to an RL pipeline whose reward signal is derived from executable verification infrastructure.

6. RL for verification and formal verification of RL systems

Another lineage places RL inside verification workflows themselves. VeRLPy is an open-source Python library for RL-driven verification of digital designs that treats the design-under-test as the environment and coverage-directed stimulus generation as the control problem (Shibu et al., 2021). Its architecture has three layers—hardware, cocotb, and RL—and the RL and cocotb processes communicate through a multiprocessing pipe. Rewards are defined over monitored events as

$$\max_{\theta}\ \mathbb{E}_{\mathbf{y}\sim\pi_\theta(\cdot\mid \mathbf{x})}\left[r(\mathbf{x},\mathbf{y})-\beta D_{\mathrm{KL}}\!\left(\pi_\theta(\mathbf{y}\mid \mathbf{x})\Vert \pi_{\text{init}}(\mathbf{y}\mid \mathbf{x})\right)\right].$$3

where $$\max_{\theta}\ \mathbb{E}_{\mathbf{y}\sim\pi_\theta(\cdot\mid \mathbf{x})}\left[r(\mathbf{x},\mathbf{y})-\beta D_{\mathrm{KL}}\!\left(\pi_\theta(\mathbf{y}\mid \mathbf{x})\Vert \pi_{\text{init}}(\mathbf{y}\mid \mathbf{x})\right)\right].$$4 is the number of occurrences of event $$\max_{\theta}\ \mathbb{E}_{\mathbf{y}\sim\pi_\theta(\cdot\mid \mathbf{x})}\left[r(\mathbf{x},\mathbf{y})-\beta D_{\mathrm{KL}}\!\left(\pi_\theta(\mathbf{y}\mid \mathbf{x})\Vert \pi_{\text{init}}(\mathbf{y}\mid \mathbf{x})\right)\right].$$5 and $$\max_{\theta}\ \mathbb{E}_{\mathbf{y}\sim\pi_\theta(\cdot\mid \mathbf{x})}\left[r(\mathbf{x},\mathbf{y})-\beta D_{\mathrm{KL}}\!\left(\pi_\theta(\mathbf{y}\mid \mathbf{x})\Vert \pi_{\text{init}}(\mathbf{y}\mid \mathbf{x})\right)\right].$$6 is a user-specified multiplier (Shibu et al., 2021).

The reported case studies are explicitly coverage-oriented rather than language-model-oriented. On an RLE Compressor, over 1,000 iterations, the target event $$\max_{\theta}\ \mathbb{E}_{\mathbf{y}\sim\pi_\theta(\cdot\mid \mathbf{x})}\left[r(\mathbf{x},\mathbf{y})-\beta D_{\mathrm{KL}}\!\left(\pi_\theta(\mathbf{y}\mid \mathbf{x})\Vert \pi_{\text{init}}(\mathbf{y}\mid \mathbf{x})\right)\right].$$7 occurs 61,018 times with RL versus 12,290 without RL, and the agent learns to prefer count_width values 6 and 7. On an AXI Crossbar, again over 1,000 iterations, the target FIFO-full event on the fifth slave occurs 49,772 times with RL versus 25,609 without RL (Shibu et al., 2021). Here verification is the environment objective, not the reward checker for generated text.

The inverse problem—formally verifying RL systems—appears in whiRL 2.0. That framework models a DRL system and its environment as a transition system

$$\max_{\theta}\ \mathbb{E}_{\mathbf{y}\sim\pi_\theta(\cdot\mid \mathbf{x})}\left[r(\mathbf{x},\mathbf{y})-\beta D_{\mathrm{KL}}\!\left(\pi_\theta(\mathbf{y}\mid \mathbf{x})\Vert \pi_{\text{init}}(\mathbf{y}\mid \mathbf{x})\right)\right].$$8

where $$\max_{\theta}\ \mathbb{E}_{\mathbf{y}\sim\pi_\theta(\cdot\mid \mathbf{x})}\left[r(\mathbf{x},\mathbf{y})-\beta D_{\mathrm{KL}}\!\left(\pi_\theta(\mathbf{y}\mid \mathbf{x})\Vert \pi_{\text{init}}(\mathbf{y}\mid \mathbf{x})\right)\right].$$9 is the state space, $$\eta_k=P_k(0<p(\mathbf{x})<1)=1-P_k(p(\mathbf{x})=0)-P_k(p(\mathbf{x})=1),$$0 the initial states, and $$\eta_k=P_k(0<p(\mathbf{x})<1)=1-P_k(p(\mathbf{x})=0)-P_k(p(\mathbf{x})=1),$$1 the transition relation. It combines BMC for counterexample search with k-induction for proof, plus semi-automated invariant inference and abstraction to scale to long-horizon or infinite behaviors (Amir et al., 2021). For Aurora, it proves two liveness properties that earlier work could not prove; the first is established with $$\eta_k=P_k(0<p(\mathbf{x})<1)=1-P_k(p(\mathbf{x})=0)-P_k(p(\mathbf{x})=1),$$2 in a few seconds, while the second requires $$\eta_k=P_k(0<p(\mathbf{x})<1)=1-P_k(p(\mathbf{x})=0)-P_k(p(\mathbf{x})=1),$$3 and about 4.5 hours. For Pensieve, a previously unresolved safety property is proved with $$\eta_k=P_k(0<p(\mathbf{x})<1)=1-P_k(p(\mathbf{x})=0)-P_k(p(\mathbf{x})=1),$$4 in about 20 minutes. For DeepRM, abstraction yields a stronger generalized property showing that, for any resource utilization level, when the queue is filled with identical jobs, the policy assigns higher probability to schedule$$\eta_k=P_k(0<p(\mathbf{x})<1)=1-P_k(p(\mathbf{x})=0)-P_k(p(\mathbf{x})=1),$$5 than to wait, thereby proving one earlier property and showing that two others do not hold (Amir et al., 2021).

These two directions clarify an important conceptual distinction. RL-for-verification systems such as VeRLPy use learning to drive test generation or coverage exploration. Formal-verification systems such as whiRL 2.0 treat the learned controller as the verification target. Both belong in the lexical neighborhood of “VeriRL,” but they solve fundamentally different problems.

7. Significance and recurring themes

Across these literatures, several themes recur. First, verifiability changes the optimization problem. Whether the verifier checks symbolic equivalence, unit tests, evidence overlap, or video rules, it converts difficult semantic goals into scalar rewards that can support policy improvement (Xu et al., 20 May 2025). Second, reward reliability is decisive. TinyV and RLV$$\eta_k=P_k(0<p(\mathbf{x})<1)=1-P_k(p(\mathbf{x})=0)-P_k(p(\mathbf{x})=1),$$6R show that false negatives, false positives, and the balance captured by $$\eta_k=P_k(0<p(\mathbf{x})<1)=1-P_k(p(\mathbf{x})=0)-P_k(p(\mathbf{x})=1),$$7 determine not only sample efficiency but, in some regimes, whether learning proceeds in the correct direction at all (Xu et al., 20 May 2025, Rad et al., 7 Jan 2026).

Third, curriculum and representation matter as much as the base RL algorithm. VI-CuRL stabilizes verifier-free training by selecting high-confidence prompts early (Cai et al., 13 Feb 2026); VERL reshapes the advantage using hidden-state geometry rather than token-level heuristics (Huang et al., 28 Sep 2025); DRIVE shows that hard-case retention and entropy-expansion stages are central to competitive-programming RLVR (Zhu et al., 9 Nov 2025); and VERIRL shows that rescoring partially correct solutions and balancing sample contributions can materially alter the reward landscape in HDL generation (Teng et al., 25 Aug 2025).

Fourth, verification can be either the means or the object. In RLVR, verification supplies reward. In VeRLPy, RL improves verification coverage. In whiRL 2.0, verification constrains and explains RL policies. This suggests that “VeriRL” is best understood as a family of research programs organized around the meeting point of formal criteria and reinforcement learning, rather than a single named architecture or benchmark.

A plausible implication is that future uses of the term will remain heterogeneous unless authors state explicitly which relation between RL and verification they mean: verification-guided RL, RL-guided verification, or verification of RL. The current literature supports all three, and their technical assumptions, failure modes, and evaluation criteria are materially different.