Verifiable & Adversarial RL: Concepts & Challenges
- VARL is a reinforcement learning framework that integrates externally verifiable reward signals with adversarial training to enhance robustness and mitigate reward exploitation.
- It employs precise verification methods—such as code execution tests, theorem checking, and structured metrics—to ensure outputs meet strict correctness criteria.
- Adversarial components like discriminator training and dynamic task generation work to counter reward hacking, thereby improving performance across diverse applications.
In "Right in the Right Way," VARL denotes Verifiable and Adversarial Reinforcement Learning: a training framework for LLMs in which RL with verifiable rewards is augmented by an adversarial generator-discriminator mechanism trained on human demonstrations (Damani et al., 1 Jul 2026). In the surrounding literature, the same two terms point to a broader pattern: reward is grounded in externally checkable criteria such as program execution, theorem checking, trajectory matching, or structured constraint evaluation, while robustness is pursued by hardening the verifier, generating harder tasks, correcting noisy reward channels, or co-training auxiliary components that resist reward hacking and weak supervision (Ruan et al., 13 Mar 2026). The resulting field is not methodologically uniform. Some works are verifiable RL papers first and only adversarial in a verifier-centric sense; others are explicitly adversarial through co-evolving generators and solvers; still others focus on formal verification of learned policies rather than adversarial training itself (Liu et al., 29 Mar 2026).
1. Definition and conceptual scope
The narrowest formalization in this literature comes from "Right in the Right Way," which starts from the RL with verifiable rewards objective
$\max_\theta \; \mathbb{E}_{(x, y^*) \sim \mathcal{D}, \; y \sim \pi_\theta(\cdot \mid x)} \big[ \mathbbm{1}_{y \equiv y^*} \big]$
and adds a second requirement: the policy should remain close to a human demonstration distribution in feature space (Damani et al., 1 Jul 2026). Its core VARL reward is verifier-gated: $R_{\mathrm{VARL}(x,y)=\mathbbm{1}_{y \equiv y^*}\cdot D_\eta(\phi(y),x),$ so incorrect outputs receive zero reward regardless of how human-like they appear, while correct outputs are further ranked by an adversarial discriminator trained to distinguish demonstrations from model outputs (Damani et al., 1 Jul 2026).
This definition captures a recurring tension across the literature. RL with verifiable rewards is attractive because it optimizes against signals that are externally checkable rather than preference-model scores or unconstrained judges. At the same time, multiple papers argue that verifiable reward alone is incomplete or brittle: code verifiers can be weak and static, multimodal reward models can be catastrophically exploitable, instruction verifiers can be shortcut-prone, and open-ended tasks often lack a single exact ground truth (Ruan et al., 13 Mar 2026). This suggests that VARL is best understood as an attempt to preserve the epistemic strength of externally checkable rewards while adding mechanisms that make optimization harder to exploit.
The scope of the term is therefore layered. At one end are strictly verifiable domains such as code generation, theorem proving, and symbolic reasoning, where success is grounded in execution, proof checking, or exact answers. At the other are partially verifiable domains, where structured rubrics, verifier-conditioned references, or tool outputs anchor only part of the reward. Between them sit domains such as video reasoning and negotiation, where success can be operationalized through objective task metrics even though the outputs themselves are high-dimensional and generative (Liu et al., 29 Mar 2026).
2. Verifiable reward channels
A defining property of this literature is that reward comes from mechanisms external to the policy. In code RL, "EvolveCoder" treats reward as execution-based on an evolved assert-based test suite , and its main claim is that the bottleneck is not the RL algorithm itself but verifier quality: weak, static, and redundant unit tests make reward only partially aligned with semantic correctness (Ruan et al., 13 Mar 2026). In formal theorem proving, "GAR" grounds reward in Lean4 verification and additionally penalizes statement modification, so the environment is verifiable because proof candidates are checked by a proof assistant rather than by a learned reward model (Wang et al., 13 Oct 2025). In video reasoning, "Wan-R1" replaces multimodal judges with objective task metrics, including exact-match, progress, maze-fidelity, and embedding-level rollout comparison, and reports that verifiable rewards are critical for stable training while multimodal reward models can lead to degenerate solutions (Liu et al., 29 Mar 2026).
The same principle appears in instruction following, negotiation, and general reasoning environments, but the verifier is no longer always a single exact checker. "IFDECORATOR" decomposes instructions into hard and soft constraints, using programmatic yes/no checks for hard constraints and checklist-based judgment for soft constraints, then filters data by empirical pass rate to retain only appropriately challenging instructions (Guo et al., 6 Aug 2025). "Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewards" uses a piecewise terminal reward based on buyer surplus, strict budget adherence, and deterministic action parsing; because , , and are stored in the environment, the reward is mechanically recoverable from the episode state and transcript (Liu et al., 10 Apr 2026). "Reasoning Gym" generalizes this infrastructure view by providing over 100 procedurally generated tasks whose generators also define exact verifiers, thereby exposing the data-generating process itself as the RL environment (Stojanovski et al., 30 May 2025).
Open-ended generation pushes verifiability further away from exact correctness and toward structured proxies. "Rubrics as Rewards" formalizes a prompt-specific weighted checklist
and explicitly states that classical RLVR is the special case with a single correctness criterion (Gunjal et al., 23 Jul 2025). "From Verifiable Dot to Reward Chain" similarly replaces a single verifiable endpoint with a reference-derived reward chain: content is scored by ordered keyword alignment via normalized LCS, and style is scored by executable Python checks produced offline from the reference (Jiang et al., 26 Jan 2026). These papers do not claim full formal verification of open-ended outputs; rather, they weaken the meaning of verifiability from exact-answer checking to transparent, structured, and executable subcriteria.
3. Adversariality in VARL
The adversarial component of VARL is not singular. In the surveyed literature, at least four distinct adversarial patterns appear.
First, adversariality can be verifier-centric. "EvolveCoder" strengthens RLVR by making the verifier both solution-conditioned and adversarial: test generation is conditioned on candidate programs and their pass/fail behavior, and new tests are explicitly constructed to challenge high-pass-rate programs, expose hidden corner cases, and separate solutions that the current verifier cannot distinguish (Ruan et al., 13 Mar 2026). Here the adversary is not an environment perturbing observations, but a procedure that iteratively evolves the reward channel itself.
Second, adversariality can be generator-discriminator training. In "Right in the Right Way," a discriminator is trained to distinguish human demonstrations from model generations in feature space, and the policy is rewarded only when it is both verifier-correct and discriminator-favored (Damani et al., 1 Jul 2026). The adversarial mechanism is therefore distribution matching among already-correct outputs. The paper’s theoretical analysis shows that, with , this corresponds to maximizing pass rate while minimizing the Vincze--Le Cam divergence between policy and human feature distributions (Damani et al., 1 Jul 2026).
Third, adversariality can be task-generation against the current policy frontier. "GAR" jointly trains a statement fuser and prover in Lean4. The fuser is rewarded by
0
which means it seeks theorems that are hard for the current prover but not universally unsolved, and it is penalized when proofs succeed only by modifying the statement (Wang et al., 13 Oct 2025). This is adversarial curriculum generation under exact verification.
Fourth, adversariality can be data flywheel and intent hardening. "IFDECORATOR" describes a cooperative-adversarial data flywheel that evolves instruction-verification pairs by empirical pass rate, augments them with programmatically verifiable constraints, and uses IntentCheck as a hard-gating bypass detector for cases where the response satisfies superficial constraints while violating user intent (Guo et al., 6 Aug 2025). The appendix does not formalize trip wires, but it documents placeholder substitution, semantic vacuity under structural compliance, repetitive token exploitation, and other reward-hacking examples collected from online RL without IntentCheck (Guo et al., 6 Aug 2025).
Not all verifiable RL papers are adversarial in the formal minimax sense. "Wan-R1" explicitly states that it is not an adversarial RL paper, yet its comparison with multimodal reward models shows a classic reward-hacking phenomenon: the generator learns visually plausible but logically wrong videos that fool a learned evaluator (Liu et al., 29 Mar 2026). This suggests that, in practice, adversariality often enters VARL through reward-channel exploitability even when no adversary is trained explicitly.
4. Representative domains and systems
The current VARL literature spans symbolic reasoning, code generation, video reasoning, instruction following, negotiation, and partially open-ended generation. The verifier and adversarial mechanism change by domain, but the common pattern is externally grounded reward plus a mechanism that resists weak supervision or proxy exploitation.
| Setting | Verifier | Adversarial or robustness mechanism |
|---|---|---|
| Code generation | Executable assert-based tests | Solution-conditioned adversarial test evolution |
| Formal theorem proving | Lean4 proof checking | Co-evolving statement fuser and prover |
| Instruction following | Hard checks + checklist-based soft checks | Cooperative-adversarial data flywheel + IntentCheck |
| Video reasoning | Objective trajectory and fidelity metrics | Reward-model failure analysis; verifier-grounded training |
| Open-ended generation | Verifier-gated correctness + discriminator | Human-demonstration discriminator over correct outputs |
In code generation, "EvolveCoder" constructs EvolveCoder-22k with 21,642 problems after multiple rounds of adversarial test evolution. The verifier becomes harder over rounds, with pass@1 decreasing from 43.80 to 31.22, and RL on Round 3 improves Qwen3-4B by 4.2 average points across four downstream benchmarks (Ruan et al., 13 Mar 2026). In theorem proving, "GAR" reports an average relative improvement in pass@32 of 4.20\% on MiniF2F-Test across two base provers, and DeepSeek-Prover-V2’s pass@32 on ProofNet-Test rises from 22.58\% to 25.81\% (Wang et al., 13 Oct 2025). In instruction following, "IFDECORATOR" reports that Qwen2.5-32B-Instruct-IFDecorator reaches 87.43\% on IFEval Prompt Strict and improves complex-following benchmarks while using a verifier-focused RLVR wrapper rather than preference optimization (Guo et al., 6 Aug 2025).
In video reasoning, "Wan-R1" adapts GRPO to flow-based video generation and shows that verifiable rewards improve generalization: on 3D Maze it reports a 29.1 percentage point exact-match gain relative to the SFT baseline, and on Trapfield a 51.4 point gain (Liu et al., 29 Mar 2026). In negotiation, RLVR with a structured dialogue/action protocol allows a 30B buyer model to learn a four-phase strategic evolution and to outperform larger baselines on buyer surplus while respecting strict budget constraints (Liu et al., 10 Apr 2026). In self-auditing math reasoning, "RISE" turns self-verification itself into an RL behavior and reports that verification accuracy rises dramatically—for example, on 3B models, average verification accuracy increases from 35.8 for Zero-RL to 74.3 for RISE while reasoning accuracy also improves from 32.5 to 33.5 (Liu et al., 19 May 2025).
Partially verifiable general-generation systems occupy a different point in the design space. "Rubrics as Rewards" shows up to a 28\% relative improvement on HealthBench-1k compared to simple Likert-based reward, while "RLVRR" reports that its reference-derived reward chain substantially outperforms SFT trained with ten times more data and advanced reward models on open-ended alignment benchmarks (Gunjal et al., 23 Jul 2025). These systems extend the verifier concept rather than preserving exact correctness in the strong math/code sense.
5. Failure modes, robustness, and formal guarantees
A central theme in VARL is that the verifier is part of the objective. If it is weak, noisy, or exploitable, RL optimizes the wrong thing. "Rate or Fate? RLV1R" makes this point explicit with a bandit-style analysis of noisy verifiers. With false negative rate 2, false positive rate 3, and Youden’s index
4
the paper derives a phase transition: when 5, incorrect modes are driven toward extinction; when 6, the process is neutral; and when 7, incorrect modes amplify until they dominate (Rad et al., 7 Jan 2026). In its terminology, noise then determines not just rate but fate.
A complementary treatment appears in "Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers," which models observed reward 8 as a stochastic channel over the clean binary reward 9: $R_{\mathrm{VARL}(x,y)=\mathbbm{1}_{y \equiv y^*}\cdot D_\eta(\phi(y),x),$0 It proposes a backward correction
$R_{\mathrm{VARL}(x,y)=\mathbbm{1}_{y \equiv y^*}\cdot D_\eta(\phi(y),x),$1
and a forward correction based only on $R_{\mathrm{VARL}(x,y)=\mathbbm{1}_{y \equiv y^*}\cdot D_\eta(\phi(y),x),$2, and reports that both improve RLVR under synthetic and real verifier noise, with the forward correction more stable under heavier noise (Cai et al., 1 Oct 2025). "RLAVR" addresses a different reward-reliability problem: when ground-truth labels are too expensive to obtain for every sample, pseudo-labels can cause collapse. Its Corrective Advantage Gap
$R_{\mathrm{VARL}(x,y)=\mathbbm{1}_{y \equiv y^*}\cdot D_\eta(\phi(y),x),$3
measures the supervision value of correcting a pseudo-label, and CARE uses this idea to decide which samples to label, which pseudo-labeled samples to keep, and which to drop (Wang et al., 25 May 2026).
Formal policy verification represents another robustness axis. "Verified Probabilistic Policies for Deep Reinforcement Learning" verifies stochastic policies by constructing an interval Markov decision process abstraction and proving
$R_{\mathrm{VARL}(x,y)=\mathbbm{1}_{y \equiv y^*}\cdot D_\eta(\phi(y),x),$4
so model checking yields a sound upper bound on finite-horizon failure probability (Bacci et al., 2022). "Verifiable Reinforcement Learning via Policy Extraction" takes a different route: it extracts compact decision trees from strong oracles using VIPER, then verifies local robustness, never-losing safety, and local stability with linear programming, SMT, and SOS techniques (Bastani et al., 2018). These works are not adversarial RL in the minimax sense, but they show that verifiability in RL can also mean formal post hoc certification of learned policies.
This literature repeatedly distinguishes exact correctness from proxy correctness. Reward hacking in coding can arise from weak test suites; video generators can fool multimodal judges with plausible but wrong trajectories; instruction followers can satisfy formatting while evading intent; and pseudo-labeling systems can reinforce incorrect beliefs. The common lesson is that verifier design, verifier noise, and verifier exploitation are first-order algorithmic concerns rather than implementation details.
6. Open problems, neighboring frameworks, and terminology
Several papers push beyond exact verifiers without abandoning the VARL agenda. "CRScore++" combines tool-based grounded signals from linters and smell detectors with LLM critique for code review comment generation, but it does not define a single explicit scalar verifier reward and remains only partially verifiable end to end (Kapadnis et al., 30 May 2025). "Rubrics as Rewards" extends RLVR to structured, prompt-specific criteria rather than a single exact target (Gunjal et al., 23 Jul 2025). "RLVRR" replaces a single verifiable dot with a reference-derived reward chain over content and style (Jiang et al., 26 Jan 2026). "VI-CuRL" goes further by asking whether verifier-independent curricula can stabilize reasoning RL when external verifiers are absent, and proves asymptotic unbiasedness for its confidence-guided curriculum objective even though the method is not adversarial RL (Cai et al., 13 Feb 2026). "Reasoning Gym" addresses yet another bottleneck: fixed datasets. By providing procedural generators with adjustable complexity and exact verifiers, it supplies a reusable substrate for RLVR and for future adversarial task-selection schemes (Stojanovski et al., 30 May 2025).
Across these works, several research directions recur explicitly. One is adaptive verifier design during RL, rather than only at dataset-construction time (Ruan et al., 13 Mar 2026). A second is a stronger minimax formulation for coupled policy-verifier or generator-solver systems (Wang et al., 13 Oct 2025). A third is verifier robustness analysis under noisy, biased, underspecified, or generator-targeted tests (Guo et al., 6 Aug 2025). A fourth is scaling beyond domains with exact executable semantics toward partial verification, theorem checking, simulators, and structured rubrics (Gunjal et al., 23 Jul 2025). A fifth is using online auditing, self-verification, or active label acquisition not as replacements for verifiers but as mechanisms that preserve reward fidelity when labels or evaluators are imperfect (Liu et al., 19 May 2025).
A final source of confusion is terminological. The acronym VARL is overloaded. In "Teaching RL Agents to Act Better," VARL means VLM as Action advisor for online Reinforcement Learning, not Verifiable and Adversarial Reinforcement Learning; that paper is about using a vision-LLM as a heuristic action advisor in SAC, and it explicitly has no formal verification or adversarial-training component (Wu et al., 25 Sep 2025). In the narrower sense used by "Right in the Right Way," however, VARL is a specific generator-discriminator extension of RLVR (Damani et al., 1 Jul 2026). A careful reading of recent work therefore benefits from distinguishing VARL as a named method from VARL as an umbrella for verifiable reward design plus adversarial or robustness-oriented training mechanisms.
Taken together, these papers support a consistent interpretation. Verifiable reward is a powerful supervision source when correctness can be externally checked, but it is rarely sufficient by itself. The adversarial half of VARL emerges wherever the training pipeline actively counters loopholes in that reward channel: by evolving stronger verifiers, co-generating harder tasks, matching human distributions among correct outputs, correcting noisy reward channels, or selectively acquiring oracle labels. In that sense, VARL is less a single algorithm than a design principle: if RL optimizes what the verifier measures, then both the verifier and the adversary against it must be part of the learning problem.