When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR

Abstract: The test suites used as RLVR rewards for code have natural false positives: per-task, persistent, asymmetric errors that accept the same wrong programs every time they appear, unlike the symmetric or resampled noise assumed by existing noise-robustness analyses. We run a preregistered two-arm causal contrast on a deployed suite: GRPO on identical MBPP tasks, seeds, and compute, rewarded by the original MBPP tests (leaky) versus the MBPP+ extra tests (hardened). Two further families replicate the design under a preregistration frozen before their data existed. [C] The average held-out effect is bounded: non-inferior under a preregistered 1.5-pt margin (gap 0.20 pt, one-sided 95% upper bound 0.75 pt). [C] Rewarded false-positive mass tracks a cheap static leakiness audit computed before training (Spearman 0.80), and the registered train-side test puts the leak-stratum FP share +43.8 pt above clean tasks. [E] Auditing every rewarded FP under signed, human-adjudicated rules finds a large residual of verified genuinely wrong code: 47.57% record-weighted; both replication families reproduce a large share. The reward paid for real bugs, not merely suite artifacts. [E] Mechanism evidence is consistent with selection of pre-existing error modes rather than learned exploitation: FP incidence does not grow within our horizon, and untrained base models already produce the same wrong outputs under the leaky filter. We then turn the same instrument on the frontier judges themselves: on their own false positives they self-assess only weakly, a same-author test is unresolved, and even the highest-scoring reader we probe stays far below its score on a weaker policy's errors -- two subjects on MBPP, licensing nothing about frontier models in general. A cheap static audit locates exposure before training; hardening the reward removes the measurement inflation, though here it buys little capability.

• The paper presents a preregistered causal study showing that leaky test suites yield persistent false positives without causing aggregate capability collapse.
• It employs a dual-arm RL experiment to correlate static leakiness scores with misrewarded outputs, demonstrating selection of pre-existing errors.
• The findings highlight that pre-training static audits can accurately predict reward leakage, guiding targeted suite hardening in code RL.

Preregistered Causal Measurement of Natural Verifier False Positives in RLVR

Introduction and Motivation

The paper "When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR" (2607.11022) offers a rigorous, preregistered causal measurement of how persistent, per-task false positives in standard test suites affect RL-based code model training. Unlike the established RLVR noise literature (which assumes symmetric, resampled label noise that diminishes with averaging), this study targets natural suite errors: programmatic faults where a weak suite persistently rewards the same incorrect code variant on repeated rollouts.

With current suite audits finding that approximately 25–28.5% of RL training tasks for code accept at least one incorrect solution (Rajan, 14 Jun 2026), but without causal evidence of the resulting impact, this work fills the gap. The main experimental axis probes whether such "leakiness" actively distorts learned policies, the extent to which models collect reward from genuinely incorrect code, and whether these effects can be statically predicted and pre-empted.

Experimental Design

The core contrast is a dual-arm RL experiment. For each MBPP code-generation task (Austin et al., 2021), models are trained with identical data, seed, and RL protocol, varying only the reward function: the "leaky" arm uses the standard MBPP tests; the "hardened" arm uses the MBPP+ (EvalPlus) extra test suite, which is significantly more thorough [liu2023evalplus]. The study is cross-validated on three model families: Qwen2.5-Coder-1.5B-Instruct, deepseek-coder-1.3b-instruct, and Llama-3.2-1B-Instruct, each evaluated with 5 seeds $\times$ 400 steps.

Figure 1: Overview of the design. Each MBPP task has original (leaky) and MBPP+ (hardened) test suites; a static audit pre-computes per-task leakiness; two GRPO RL arms differ only by reward suite; the static audit predicts where FP mass appears; all held-out evaluations score with MBPP+.

Before RL, a static audit ("leakiness score") is computed by generating base model completions and identifying the fraction that pass the leaky tests but fail the hardened tests. During RL, every rewarded output is separately evaluated under both suites: rollouts rewarded under the leaky arm but failing the hardened suite (rewarded FPs) directly measure exploitation of suite errors.

A preregistered protocol locks all evaluation axes and includes both confirmatory tests and exploratory follow-ups.

Main Empirical Findings

Aggregate Effect of Leaky Rewards

On held-out tasks with the hardened (MBPP+) suite as oracle, the difference between arms is statistically bounded (gap of 0.20 points, one-sided 95% upper bound of 0.75, below the preregistered 1.5 point margin). Both arms yield similar improvements over the base model. Thus, leakiness does not lead to detectable aggregate capability collapse within the measured horizon and scale.

Static Predictability of Leakiness

Rewarded FP mass per task during training correlates tightly with the static audit's leakiness score (Spearman 0.80 raw, 0.79 after controlling for task difficulty). The static audit provides a low-cost predictive screen for future reward leak and exposure.

Figure 2: Rewarded false-positive mass tracks static leakiness in all three families: for each task, the fraction of rewarded rollouts that are FPs (y-axis) is strongly correlated with pre-training static leakiness (x-axis).

Composition of Rewarded FPs

A comprehensive, human-adjudicated audit of all rewarded FPs in family A shows that about half the instances for which the leaky suite pays reward are genuinely wrong code (47.6% with 95% bootstrap CI [36.4, …]). The remaining FPs are split among extra-test false negatives, contract violations, canonical-convention mismatches, and resource artifacts. Both replication families exhibit large residuals of genuinely wrong code (45–62%).

Mechanistic Axis: Selection vs. Learning

Multiple strands of evidence—static FP rates established early and flat across training, the presence of exploited error modes in untrained base models, and absence of within-horizon growth—indicate selection of pre-existing errors rather than gradient-driven learning of hacks. All major persistent channels are present in base policies.

Nuances and Cross-Model Generality

• Train-task exposure is concentrated: FP mass is highly non-uniform, with a small set of tasks responsible for the bulk of reward leakage and corresponding bugs.
• Hardening removes measurement artifacts but brings little held-out capability change.
• Statically leaky held-out tasks see less improvement under leaky training only in one model family, and with qualification (specification search correction, concentration in a small set of tasks, and non-replication in other families).
• Results on selection rather than exploitation persist out to 800 steps, with no emergent behavioral growth in reward FPs.

Implications

Practical

• Pre-training static audits are essential: Low-cost static leakiness probes can accurately localize which tasks/suites will distort reward streams.
• Reward accounting must be performed on held-out tasks: Train-task reward inflation does not robustly indicate capability or learned hacking.
• Suite hardening primarily improves measurement, not model generalization at this scale.
• Repair effort is efficiently targeted: Fixing the top leaking tasks would eliminate most of the reward leakage.

Theoretical

This work demonstrates that most persistent reward exploitation under leaky test suites in code RL is determined by static properties of the model and suite—not by RL-induced search for reward hacks. This decouples persistent suite errors from the phase transition behavior seen for symmetric noise, aligning with theoretical predictions for persistent FP reward in RLVR (Plesner et al., 9 Apr 2026, Rad et al., 7 Jan 2026).

Methodological

The approach shows that external, interface-level audits (static and dynamic) provide a more actionable metric than lower-layer interpretability or per-sample confidence signals, the latter of which can remain under-specified with binary reward.

The study offers a precise causal map from task-level leakiness to empirical FP incidence and reward distribution, supporting prior observations that aggregate metrics can mask error channels paying for real semantic bugs (Rajan, 14 Jun 2026, Ma et al., 9 Jul 2025).

Oversight and Self-Evaluation

Exploratory meta-audits on frontier LLM code judges reveal that, even when presented with their own prior FPs, self-evaluation discriminates weakly between correct and FP outputs, and much of the discriminability is driven by task-level structure rather than solution error. This raises further methodological questions about the scalability and reliability of learned evaluators and underlines the importance of exogenous audit protocols.

Limitations and Open Directions

Findings strictly apply at the measured scale (1–1.5B params, 400–800 RL steps, MBPP/MBPP+ evaluation). The absence of detected learning-based exploitation does not rule out different outcomes for larger models, longer horizons, or richer RL protocols. The evaluation is limited to automatically generated tests and a specific code-generation setting; transfer to long-horizon or agentic RL tasks warrants additional investigation.

The meta-judging experiments are exploratory and item-pool confounded, with no causal claims permitted. Practical hardness of error detection for stronger models and tasks remains to be empirically characterized as generator capabilities increase.

Conclusion

This work provides an authoritative, preregistered causal estimate of the impact of leaky test suites on RL-based code model training. Natural suite-based false positives pay for real, persistent code bugs but do not, at this scale and horizon, induce observable reward-hacking or generalization deficits. Static audits accurately localize suite-induced leakage and can be used to guide repair and evaluation. Clean aggregate reward is not evidence of a clean interface, and reward measurement must be performed out-of-distribution and post-hoc to avoid masked exploit channels.

These findings contribute a robust measurement scaffold for RLVR reward QA and motivate rigorous, interface-level evaluation as a necessary complement to architectural or interpretability-based reconciliation of reward hacking in code agents.

---

References:

• "When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR" (2607.11022)
• "Auditing Reward Hackability in Code RL Training Environments" (Rajan, 14 Jun 2026)
• "An Imperfect Verifier is Good Enough: Learning with Noisy Rewards" (Plesner et al., 9 Apr 2026)
• "Rate or Fate? RLVεR: Reinforcement Learning with Verifiable Noisy Rewards" (Rad et al., 7 Jan 2026)
• "Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of LLMs for Code Generation" [liu2023evalplus]
• "Program Synthesis with LLMs" (Austin et al., 2021)