RLVR with Synthetic Errors

• RLVR with Synthetic Errors is a framework where reinforcement learning models receive binary rewards that are intentionally corrupted by synthetic false positives and negatives.
• The methodology introduces gradient corrections—backward and forward techniques—to counteract structured reward noise and improve policy optimization.
• Empirical and bandit analyses reveal that verifier discriminativity critically governs learning dynamics, with performance degrading when noise exceeds a positive signal threshold.

Searching arXiv for recent RLVR papers on noisy/verifiable rewards and related formulations. Reinforcement learning with verifiable rewards under synthetic errors denotes a class of RLVR settings in which the reward signal is produced by an automated verifier whose outputs are intentionally corrupted or modeled as imperfect, typically through synthetic false positives and false negatives. In this formulation, verifier unreliability is treated as structured reward noise rather than as ordinary stochasticity in policy optimization. Recent work studies this problem from complementary perspectives: one line formalizes the verifier as a stochastic reward channel and derives correction schemes for policy-gradient updates under asymmetric noise (Cai et al., 1 Oct 2025); a second line analyzes RLVR dynamics under noisy verification through a bandit and replicator framework, identifying a phase transition governed by verifier discriminativity (Rad et al., 7 Jan 2026); and a third line argues that, once noisy labels are rigorously re-verified to exclude accidental clean labels, synthetic noise remains materially destructive to RLVR performance and existing algorithmic variants fail to recover clean-data behavior (Zhu et al., 17 Mar 2026).

1. Problem formulation and synthetic-error regimes

RLVR formulates fine-tuning as a policy-optimization problem in which a model samples a trajectory or completion and receives a verifiable reward from a checker or execution procedure. In the notation used for RLVR, the objective is

$$J(\theta)=\mathbb{E}_{s\sim D,\tau\sim\pi_\theta(\cdot|s)}[r_v(s,\tau)],$$

with $r_v(s,\tau)\in\{0,1\}$ in the binary-verification setting (Zhu et al., 17 Mar 2026). GRPO is a common optimization backbone in this literature, using groupwise sampled rollouts and normalized advantages (Zhu et al., 17 Mar 2026).

Synthetic errors are introduced to study the effect of imperfect verification under controlled conditions. One construction begins with a clean dataset and replaces a fraction $p$ of gold answers with model-generated incorrect answers, with $p\in\{0\%,10\%,20\%,50\%,100\%\}$ (Zhu et al., 17 Mar 2026). The same study also considers two extreme synthetic noise models: random annotations, where the noisy answer is drawn uniformly from the answer space, and format-only reward, where the verifier returns 1 when a surface-form constraint is met, such as containing \boxed{·} (Zhu et al., 17 Mar 2026). These interventions emulate rule-based or model-based reward corruption in deployed RLVR pipelines.

A distinct but closely related formalization models synthetic verifier errors at the reward level rather than the label level. In this view, the clean reward $R^*(x,y)\in\{0,1\}$ is passed through a Bernoulli-to-Bernoulli noise channel that produces an observed reward $\tilde R(x,y)\in\{0,1\}$ with false-positive rate $\rho_0$ and false-negative rate $\rho_1$ (Cai et al., 1 Oct 2025). This abstraction isolates verifier imperfection from the underlying task distribution and permits closed-form de-biasing under class-conditional, instance-independent noise assumptions.

A broader dynamical account abstracts prompt-level RLVR into a $(K+M)$-armed bandit over “reasoning modes,” partitioned into correct and incorrect subsets, with noisy verification parameterized by $\mathrm{FNR}=\delta_{\rm FN}$ and $r_v(s,\tau)\in\{0,1\}$0 (Rad et al., 7 Jan 2026). This view treats synthetic verifier errors as perturbations to the selection dynamics over mode probabilities rather than solely as corrupted scalar rewards.

2. Stochastic reward-channel model

In the stochastic-channel formulation, verifier error is defined by

$r_v(s,\tau)\in\{0,1\}$1

with $r_v(s,\tau)\in\{0,1\}$2 (Cai et al., 1 Oct 2025). Here $r_v(s,\tau)\in\{0,1\}$3 is the false-positive rate and $r_v(s,\tau)\in\{0,1\}$4 is the false-negative rate. Under this model, the noisy reward satisfies

$r_v(s,\tau)\in\{0,1\}$5

so the observed reward is an affine transformation of the true reward in expectation (Cai et al., 1 Oct 2025).

The core assumptions are that the noise rates are class-conditional and instance-independent, and that conditioned on $r_v(s,\tau)\in\{0,1\}$6, the noisy reward is independent of the policy’s sampling randomness, including score-function terms (Cai et al., 1 Oct 2025). These are standard class-conditional-noise assumptions and are the basis for the paper’s closed-form backward and forward corrections.

This reward-channel account is technically narrower than the error processes emphasized in work arguing that noisy data is destructive to RLVR. That study explicitly attributes failure of existing methods in part to violation of i.i.d. noise assumptions, observing that real annotation errors are question-dependent and non-uniform (Zhu et al., 17 Mar 2026). This suggests that the usefulness of channel-based correction depends materially on whether verifier errors approximate uniform class-conditional flips. A plausible implication is that the channel model is best interpreted as an analytically tractable and algorithmically actionable approximation rather than a universal description of real-world noise.

The bandit analysis of noisy RLVR introduces an alternative summary statistic, Youden’s index

$r_v(s,\tau)\in\{0,1\}$7

which measures verifier discriminativity (Rad et al., 7 Jan 2026). While expressed differently, it captures the same asymmetry between accepting incorrect outputs and rejecting correct outputs, and it becomes central to the dynamical phase structure of learning under synthetic errors.

3. Gradient correction under noisy rewards

Two lightweight corrections have been proposed for RLVR under imperfect verifiers, both implemented as hooks in a GRPO pipeline (Cai et al., 1 Oct 2025).

The first is a backward correction that inverts the affine channel:

$r_v(s,\tau)\in\{0,1\}$8

Substituting $r_v(s,\tau)\in\{0,1\}$9 for $p$0 in a REINFORCE-style gradient estimator yields an unbiased policy-gradient estimator in expectation, provided the channel assumptions hold and the noise rates are accurately estimated (Cai et al., 1 Oct 2025). The paper emphasizes that other RL components, including advantage normalization and KL penalties, can remain unchanged, with $p$1 simply replacing $p$2 (Cai et al., 1 Oct 2025).

The second is a forward correction based on reweighting score-function terms rather than de-biasing the reward directly. With

$p$3

the corrected update is

$p$4

and its expectation satisfies

$p$5

Because the multiplicative factor is positive and can be absorbed into the learning rate, the expected update has the same direction as the clean policy gradient (Cai et al., 1 Oct 2025). Notably, this forward correction depends only on the false-negative rate $p$6 for the per-trajectory weights; $p$7 appears only in the global scaling factor (Cai et al., 1 Oct 2025).

The paper characterizes the two methods through a bias-variance trade-off. Backward correction is unbiased by construction but can suffer large variance when $p$8 is small or when $p$9 are mis-estimated. Forward correction is directionally correct in expectation and more variance-robust under heavier noise, with mild mis-specification of $p\in\{0\%,10\%,20\%,50\%,100\%\}$0 perturbing the weights without causing blow-up (Cai et al., 1 Oct 2025).

These conclusions are contested by later results on rigorously re-verified noisy datasets. In synthetic-noise experiments using Qwen2.5-Math-7B and multiple RLVR algorithm variants, PGFC, DAPO, SAPO, TIS, and Dr. GRPO do not outperform GRPO by more than 1 percentage point under 50% noise, and all remain 3–7 percentage points behind clean-data GRPO (Zhu et al., 17 Mar 2026). The divergence between these findings reflects a central methodological issue: whether the synthetic noise process conforms closely enough to the assumptions that justify correction.

4. Dynamical analysis and phase transition

A separate line of work analyzes RLVR with noisy verification through a multi-armed-bandit abstraction in which full-sequence completions are clustered into recurring reasoning modes $p\in\{0\%,10\%,20\%,50\%,100\%\}$1, partitioned into correct modes $p\in\{0\%,10\%,20\%,50\%,100\%\}$2 and incorrect modes $p\in\{0\%,10\%,20\%,50\%,100\%\}$3 (Rad et al., 7 Jan 2026). For a fixed prompt, the policy over modes is parameterized by a softmax distribution $p\in\{0\%,10\%,20\%,50\%,100\%\}$4, and the total bad-mode mass is

$p\in\{0\%,10\%,20\%,50\%,100\%\}$5

with $p\in\{0\%,10\%,20\%,50\%,100\%\}$6 denoting the mass on correct modes (Rad et al., 7 Jan 2026).

Under GRPO, the expected logit update induces a replicator flow on the probability simplex:

$p\in\{0\%,10\%,20\%,50\%,100\%\}$7

In a block-symmetric two-class reduction, the one-dimensional evolution of the bad mass obeys

$p\in\{0\%,10\%,20\%,50\%,100\%\}$8

where $p\in\{0\%,10\%,20\%,50\%,100\%\}$9 and $R^*(x,y)\in\{0,1\}$0 (Rad et al., 7 Jan 2026). The sign of $R^*(x,y)\in\{0,1\}$1 therefore determines the direction of drift.

This yields a sharp phase transition at $R^*(x,y)\in\{0,1\}$2 (Rad et al., 7 Jan 2026). If $R^*(x,y)\in\{0,1\}$3, then $R^*(x,y)\in\{0,1\}$4 and incorrect mass is driven toward extinction, corresponding to successful learning. If $R^*(x,y)\in\{0,1\}$5, the process is neutral, with no systematic progress. If $R^*(x,y)\in\{0,1\}$6, then $R^*(x,y)\in\{0,1\}$7, so incorrect modes amplify until they dominate, leading to anti-learning and collapse (Rad et al., 7 Jan 2026). The paper’s phrase “rate, not fate” refers specifically to the $R^*(x,y)\in\{0,1\}$8 regime, where noise rescales convergence time rather than changing the asymptotic outcome (Rad et al., 7 Jan 2026).

Controlled experiments on Python code generation with Qwen-2.5-3B and synthetic noise injected through a noisy wrapper around an oracle checker reproduce the predicted $R^*(x,y)\in\{0,1\}$9 boundary (Rad et al., 7 Jan 2026). For $\tilde R(x,y)\in\{0,1\}$0, pass@1 rises monotonically; at $\tilde R(x,y)\in\{0,1\}$1, it remains flat; and for $\tilde R(x,y)\in\{0,1\}$2, it declines (Rad et al., 7 Jan 2026). The same study reports asymmetric noise effects: at fixed $\tilde R(x,y)\in\{0,1\}$3, high FPR slows learning more than high FNR (Rad et al., 7 Jan 2026).

This dynamical framework offers a conceptual distinction absent from pure reward-correction analyses. It identifies a regime in which no amount of RL compute can guarantee improvement if verifier discriminativity is too poor, namely when $\tilde R(x,y)\in\{0,1\}$4 (Rad et al., 7 Jan 2026). A plausible implication is that correction methods and hyperparameter tuning can only be effective when they operate within a verifier regime that already preserves positive signal polarity.

5. Empirical findings on synthetic errors

The reward-correction study evaluates synthetic verifier noise with $\tilde R(x,y)\in\{0,1\}$5 and $\tilde R(x,y)\in\{0,1\}$6 on math reasoning, using Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, Llama-3.2-3B, and Qwen2.5-Math-7B (Cai et al., 1 Oct 2025). It reports Pass@1 and Pass@8, using 16 samples and 5 seeds, on AIME2024/5, AMC2023, MATH500, Minerva MATH, and OlympiadBench (Cai et al., 1 Oct 2025). The compared conditions are Base, Oracle, Noise, Noise+BC, and Noise+FC, and the stated result is that both corrections recover nearly all of the Oracle gap, with forward correction slightly superior under heavier noise and more stable (Cai et al., 1 Oct 2025). The same paper finds that backward correction degrades sharply if $\tilde R(x,y)\in\{0,1\}$7, whereas forward correction remains flat around the true $\tilde R(x,y)\in\{0,1\}$8 (Cai et al., 1 Oct 2025).

The destructive-noise study reaches a markedly different conclusion after constructing what it describes as truly noisy datasets through a four-stage re-verification pipeline involving GPT-5 Pro annotation, symbolic equivalence checking, LLM-as-Judge filtering, and manual investigation with a Clopper–Pearson upper-bound error analysis (Zhu et al., 17 Mar 2026). It reports that prior “100% noise” datasets were contaminated, with roughly 16% of purportedly incorrect labels actually valid (Zhu et al., 17 Mar 2026). After re-verification, synthetic 100% noisy training data yields substantial degradation.

Its quantitative synthetic-noise results for Qwen2.5-Math-7B with GRPO are summarized below (Zhu et al., 17 Mar 2026).

Benchmark	$\tilde R(x,y)\in\{0,1\}$9	$\rho_0$0
MATH-500	55.2%	46.2%
AIME avg.	44.8%	34.8%
AMC avg.	62.3%	53.8%

For the same setting, the reported degradation at $\rho_0$1 is 9.0 percentage points on MATH-500, 10.0 percentage points on AIME average, and 8.5 percentage points on AMC average (Zhu et al., 17 Mar 2026). As $\rho_0$2 increases from 0% to 100%, accuracy degrades nearly linearly, with $\rho_0$3 at $\rho_0$4 and $\rho_0$5 at $\rho_0$6 (Zhu et al., 17 Mar 2026). The pass@k curves show that models trained on 100% noise fall below the base model for all $\rho_0$7, and average generation length drops monotonically with noise, by –5.2% at $\rho_0$8 and –23.9% at $\rho_0$9 (Zhu et al., 17 Mar 2026).

The code-domain bandit study provides a third empirical perspective. On OpenR1 programming tasks with Qwen-2.5-3B under synthetic $\rho_1$0 noise, it reports validation pass@1 after 2 epochs of 0.16% for $\rho_1$1 at $\rho_1$2, 13.4% for $\rho_1$3 at $\rho_1$4, 16.0% for $\rho_1$5 at $\rho_1$6, 14.6% for $\rho_1$7 at $\rho_1$8, 18.6% for $\rho_1$9 at $(K+M)$0, and 20.8% for $(K+M)$1 at $(K+M)$2 (Rad et al., 7 Jan 2026). This pattern supports the predicted phase boundary and the claim that high FPR is especially damaging.

Taken together, these results do not yield a single uniform verdict. They instead delineate conditions under which synthetic errors are tractable and conditions under which they remain destructive. This suggests that empirical outcomes are highly sensitive to how noise is generated, whether synthetic corruption is reward-level or label-level, how faithfully noisy labels exclude clean cases, and whether the assumptions underlying rate correction are actually satisfied.

6. Online FN-rate estimation and appeals

In practical RLVR systems, rule-based checkers may have negligible false positives but high false negatives (Cai et al., 1 Oct 2025). To address this asymmetry, an appeals mechanism has been proposed for online estimation of the false-negative rate.

At each update step, the method collects the rule-based negatives $(K+M)$3 and positives $(K+M)$4, samples each item in $(K+M)$5 with probability $(K+M)$6, and re-verifies sampled negatives with a lightweight LLM verifier called TinyV (Cai et al., 1 Oct 2025). Let $(K+M)$7 denote the sampled negatives that flip positive under the LLM verifier. The total false-negative count is then estimated by the Horvitz–Thompson estimator

$(K+M)$8

and the false-negative rate is estimated using a Beta prior $(K+M)$9:

$\mathrm{FNR}=\delta_{\rm FN}$0

An EMA smoothing step updates $\mathrm{FNR}=\delta_{\rm FN}$1 via

$\mathrm{FNR}=\delta_{\rm FN}$2

(Cai et al., 1 Oct 2025).

In real-world noise experiments combining a rule-checker with TinyV appeals, forward correction using the online $\mathrm{FNR}=\delta_{\rm FN}$3 outperforms pure rule-based RL, direct LLM-judge rewards, and appeals without gradient correction, with gains of several percentage points in Pass@1/8 across all backbones and benchmarks (Cai et al., 1 Oct 2025). The additional cost is approximately $\mathrm{FNR}=\delta_{\rm FN}$4 extra LLM calls per update and is described as typically a small fraction, for example $\mathrm{FNR}=\delta_{\rm FN}$5 (Cai et al., 1 Oct 2025).

This appeals pipeline is significant because it estimates only $\mathrm{FNR}=\delta_{\rm FN}$6, matching the parameter requirement of forward correction. It therefore provides an operational pathway for deploying correction in settings where $\mathrm{FNR}=\delta_{\rm FN}$7 is difficult to estimate reliably. A plausible implication is that the combination of sparse secondary verification and one-parameter correction is better aligned with production RLVR constraints than full channel inversion.

7. Interpretation, limitations, and contested points

A recurring misconception is that binarized verifiable rewards eliminate reward hacking by making the learning signal effectively clean. The literature instead emphasizes that imperfect verifiers inevitably introduce false negatives and false positives even when rewards are reduced to $\mathrm{FNR}=\delta_{\rm FN}$8 (Cai et al., 1 Oct 2025). Synthetic-error studies exist precisely because verifier imperfections survive binarization.

Another misconception is that algorithmic modifications alone can compensate for arbitrary label corruption. One set of results shows that both backward and forward corrections improve RLVR under synthetic and real verifier noise, with forward correction more stable under heavier noise (Cai et al., 1 Oct 2025). Another shows that existing RLVR algorithm improvements, including PGFC, fail to mitigate the impact of truly incorrect annotations and perform similarly to basic GRPO once contamination is removed from noisy datasets (Zhu et al., 17 Mar 2026). The disagreement is not merely empirical; it reflects distinct assumptions about the noise process. Correction methods presuppose class-conditional, instance-independent noise and reliable rate estimation (Cai et al., 1 Oct 2025), whereas the destructive-noise account argues that real annotation errors are question-dependent and non-uniform, violating those assumptions (Zhu et al., 17 Mar 2026).

The bandit analysis refines this dispute by separating two questions: whether verifier noise changes asymptotic learning fate and whether it slows the rate of convergence. Its answer is conditional. When $\mathrm{FNR}=\delta_{\rm FN}$9, noise primarily rescales learning time; when $r_v(s,\tau)\in\{0,1\}$00, learning can stall or reverse regardless of additional compute (Rad et al., 7 Jan 2026). This implies that correction schemes may be effective only within a verifier regime that retains positive discriminative power.

Practical recommendations in the literature are correspondingly conditional. If both $r_v(s,\tau)\in\{0,1\}$01 and $r_v(s,\tau)\in\{0,1\}$02 can be estimated reliably, backward correction may be used for unbiasedness; if $r_v(s,\tau)\in\{0,1\}$03 is hard to estimate or total noise is large, forward correction using only $r_v(s,\tau)\in\{0,1\}$04 is preferred (Cai et al., 1 Oct 2025). At the same time, rigorous data re-verification pipelines, including symbolic checks, LLM judging, and manual audits, are recommended because high-quality data remains essential for effective RLVR (Zhu et al., 17 Mar 2026). The bandit analysis adds a verifier-centric design principle: improve discriminativity, especially by reducing false positives, since high FPR is especially damaging and can push the system toward anti-learning (Rad et al., 7 Jan 2026).

In aggregate, RLVR with synthetic errors has emerged as a focal setting for understanding the interaction between policy optimization and reward imperfection. The area now contains both constructive and cautionary results: lightweight gradient corrections can be derived and can improve performance under controlled noisy-verifier assumptions (Cai et al., 1 Oct 2025); learning dynamics under noise exhibit a sharp phase transition governed by verifier quality (Rad et al., 7 Jan 2026); and truly noisy labels, once separated from accidental clean contamination, remain a substantial obstacle that present RLVR methods do not reliably overcome (Zhu et al., 17 Mar 2026).