An Imperfect Verifier is Good Enough: Learning with Noisy Rewards

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become a prominent method for post-training LLMs. However, verifiers are rarely error-free; even deterministic checks can be inaccurate, and the growing dependence on model-based judges exacerbates the issue. The extent to which RLVR is robust to such noise and the verifier accuracy required for effective training remain unresolved questions. We investigate these questions in the domains of code generation and scientific reasoning by introducing noise into RL training. Noise rates up to 15% yield peak validation accuracy within 2 percentage points of the clean baseline. These findings are consistent across controlled and model-based noise types, three model families (Qwen3, GLM4, Llama 3.1), and model sizes from 4B to 9B. Overall, the results indicate that imperfect verification does not constitute a fundamental barrier to RLVR. Furthermore, our findings suggest that practitioners should prioritize moderate accuracy with high precision over perfect verification.

• The paper demonstrates that RLVR retains near-baseline validation accuracy within two percentage points under noise levels up to 15%.
• The methodology involves systematic experiments using controlled noise injection and model-based verifiers across code generation and scientific reasoning tasks.
• The results reveal that high precision in verifiers prevents reward hacking, while moderate noise acts as a beneficial regularizer for improved exploration.

Robustness of RLVR to Noisy Rewards: An Empirical Study

Introduction

This work investigates the resilience of Reinforcement Learning with Verifiable Rewards (RLVR) to noise in the reward evaluation process during post-training of LLMs. While RLVR—especially via protocols such as Group Relative Policy Optimization (GRPO)—has demonstrated scalable post-training effectiveness for LLMs in domains like mathematics and code, the paradigm increasingly relies on imperfect verifiers, either due to inherent task ambiguity or the adoption of model-based judges. The central question addressed is: To what extent must verifiers be accurate for RLVR to remain effective, and does verifier noise fundamentally constrain post-training outcomes?

Methods and Experimental Design

The study conducts a systematic empirical evaluation across:

• Domains: Primarily code generation (MBPP), with generalization checks on scientific reasoning (GPQA)
• Models: Qwen3 (4B, 8B), GLM4 (9B), Llama 3.1 (8B)
• Reward Noises:
  • Controlled stochastic corruption—bit flips at varying structural levels (cell, row, column, matrix within group-by-unit test reward matrices) and rates ($p \in [0.01, 0.50]$)
  • Model-based verifier noise—a learned judge substitutes for deterministic unit test execution, spanning different judge model capacities for realism

The evaluation metric is mean validation pass rate, using ground-truth verifiers, quantitatively measuring both best and final post-training model performance under various noisy reward conditions.

Main Results

Robustness Across Noise Types and Rates

The empirical findings demonstrate that RLVR exhibits remarkable robustness to verifier noise for rates up to 15%. Across all considered noise modalities, including structured and model-based, both best and final validation accuracies remain within two percentage points of the perfect-verifier baseline for $p \leq 0.15$. Significant performance degradation only sets in at much higher noise levels.

(Figure 1)

Figure 1: Best validation reward across noise levels for group rollout noise; RLVR retains baseline performance up to 15% noise.

Similarly, the analysis reveals that the exact structure of the injected noise (group-level vs. sample-level) is less crucial than the overall noise magnitude. Group-level and sample-level noise types yield similar tolerances, with slight preference toward robustness for group-structured perturbations.

Noisy Rewards as Implicit Regularization

Surprisingly, moderate reward noise can slightly improve generalization, hypothesized to be due to regularization: low-frequency gradient inversion induced by group-level noise aids escape from sharp, potentially overfit basins, analogous to noise-induced exploration benefits in SGD for non-convex landscapes. This is substantiated with a controlled optimization experiment on the Ackley function, where moderate noise dislodges trajectories from local minima, whereas noiseless training converges prematurely.

(Figure 2)

Figure 2: Optimization trajectories on the Ackley function. Moderate reward noise facilitates escape from local attractors and improves final outcomes.

Generalization to Model-Based Verifier and Domain Transfer

When a model-based judge replaces the deterministic verifier, RLVR performance scales as a monotonic function of the verifier’s precision and accuracy, not recall, with strong models (e.g., Qwen3-30B verifier, $>$85% precision) maintaining post-training quality close to the unit-test baseline. Weak verifiers (e.g., Qwen3-4B) with lower precision severely impair learning.

The results extend to the GPQA scientific reasoning dataset. Even at noise rates as high as $p=0.30$, no significant accuracy drop occurs over noise-free RLVR, suggesting broad applicability.

Precision-Recall Bias in Reward Models

The study convincingly demonstrates that precision (low false-positive rate) is far more critical for verifying rewards than recall. Because high false positives induce reward hacking, the model exploits erroneous reward assignments; in contrast, false negatives merely encourage additional exploration without leading to overoptimization. This implicates that, in semi-verifiable or model-judged domains, practitioners should favor high precision verifiers, accepting some misses on correct cases to avoid runaway reward ambiguity.

(Figure 3)

Figure 3: Training reward, precision, and recall under 4B (poor) vs 30B (good) model-based verifier. High precision is critical for post-training efficacy.

Theoretical Implications

This work empirically substantiates the hypothesis that noise-induced reward corruption does not pose a fundamental barrier to RLVR for practical noise rates (<15–20%). Moderate amounts of noise may, in fact, act as a beneficial regularizer, facilitating policy robustness by disrupting overfitting to dataset artifacts and reward hacking, particularly under exploration-heavy RL protocols like GRPO.

These findings dovetail with classical theory on noisy optimization (e.g., SGD, entropy-based minimization, sharpness-aware training), supporting the conjecture that RLVR’s invariance to verifier imperfections is a consequence of gradient stochasticity and landscape smoothing.

Practical Implications

• Verifier Engineering: For RL-based post-training of LLMs, perfectionist efforts on verifier recall are unwarranted when high precision is attainable. Model-based judges with ~85% precision are already “good enough” for most post-training use cases.
• Domain Extension: Results lower the barrier for extending RLVR to semi-verifiable or subjective domains (e.g., law, finance), where deterministic labels are rare, and model-based or rubric-driven reward assignment is mandatory.
• RLVR Adoption: Institutions and practitioners can streamline RLVR pipelines by focusing on systematic evaluation of verifier precision, aligning resources toward the most impactful error mitigation axis.

Future Directions

Open avenues include systematic evaluation for much larger models, tasks with persistent/correlated noise rather than resampled i.i.d. corruption, and domains where reward ambiguity exceeds 20–30%. There is explicit motivation for theory expansion regarding asymmetric noise impacts (precision vs. recall) and overoptimization dynamics in RL with non-stationary reward channels.

Conclusion

The empirical analysis provides strong evidence that RLVR is robust to moderate levels of verifier noise—contrary to the presumption that nearly perfect reward supervision is essential. Practically, an imperfect verifier with moderate accuracy and especially high precision suffices, opening the door for scalable RLVR in challenging, real-world, and semi-verifiable environments.