- The paper introduces a freeze-then-audit protocol showing that single-reference evaluation inflates apparent evolution gains by 9.46–14.85 percentage points across cohorts and models.
- Independent resampling outperforms mutation-based evolution by 6.01–18.83 points after auditing, indicating that reported mutation benefits largely reflect oracle errors and unequal search opportunities.
- A density-matched placebo reproduces most observed feedback effects, leaving the causal value of behavior-aligned execution feedback inconclusive and underscoring the need for stronger oracles.
This paper presents a systematic audit of feedback-driven test generation with LLMs, demonstrating that a widely used evaluation shortcut—treating the output of a single accepted program as ground truth—manufactures large apparent gains that do not survive scrutiny (2608.19626). The work is diagnostic rather than constructive: it contributes an audit-and-placebo protocol for attributing gains in self-evolving test generators, not a new generator.
The oracle problem in a feedback loop
The setting is natural-language-to-test generation for competitive-programming tasks: a model receives a problem statement and up to three public examples and emits complete input–output tests. Executable tests are attractive as verifiable feedback for code models, and recent systems rank programs with generated tests or co-train coder and tester models. The paper observes that execution verifies a generated test only if the input is permitted by the specification and the expected output is correct. The tempting shortcut of executing a generated input on one accepted program and treating its output as ground truth fails on both counts: the program may accept out-of-domain inputs, mishandle valid corner cases, or implement undefined behavior. In a self-evolution loop, optimization can repeatedly select tests that disagree with faulty submissions for the wrong reason.
The paper formalizes three oracle notions. The direct model oracle compares the model's expected output against the consensus of an accepted-program panel. The single-reference oracle replaces the model's output with the output of one hash-selected accepted program, removing output hallucination but not input validity. The multi-implementation audit executes each selected test on all remaining accepted programs after the archive is frozen, removing disconfirmed entries without replacement. Gain inflation is defined as the difference between the apparent evolution gain under a single reference and the gain of the same frozen archive after audit.
Study design
Three cohorts are analyzed separately: 142 development tasks, a procedure-locked 114-task external cohort from TestCase-Eval, and a 138-task held-out cohort that is outcome-blind but qualification-amended (the original frozen gate failed before any model calls, and a separately frozen amendment reduced the fault threshold from 20 to 18). Two Qwen-family models (Qwen2.5-Coder-7B Instruct and Qwen3.5-9B) and three seeds are used, with real historical faulty submissions split into balanced cross-fitted folds. The primary outcome is task-macro held-out constrained kill; tasks, not tests or executions, are the unit of inference. External executions run in bubblewrap namespaces, and artifacts are linked by SHA-256 locks.
The feedback attribution experiment compares a genuine three-round loop (Real) against a density-matched placebo that preserves prompt structure, history size, detected-set cardinality, and kill-count density while randomizing the input–fault correspondence. A pre-specified ±2 smallest-effect-of-interest rule governs equivalence and superiority decisions on the held-out cohort.
Oracle reliability and gain inflation
Even on inputs where all available accepted implementations agree, model-proposed expected outputs match the panel for only 44.16% (development) and 27.79% (external) of admissible inputs under Qwen2.5, and 58.90% and 50.12% under Qwen3.5. Fully panel-confirmed suites occur in only 2.53–5.07% of external calls. This is a strong quantitative statement: even a substantially better model remains too unreliable to serve as a reward oracle.
| Cohort |
Model |
Matches panel (%) |
Panel-confirmed suite (%) |
| Development |
Qwen2.5 |
44.16 |
4.77 |
| Development |
Qwen3.5 |
58.90 |
11.50 |
| External |
Qwen2.5 |
27.79 |
2.53 |
| External |
Qwen3.5 |
50.12 |
5.07 |
The central result is the inflation measurement. Apparent evolution gains of 10.58–13.38 points on development shrink to −1.47–−0.02 after audit; apparent external gains of 11.58–12.57 shrink to 2.12–2.18. Estimated gain inflation ranges from 9.46 to 14.85 percentage points, with all 95% intervals strictly above zero. Adding a second accepted implementation already removes most of the inflation, reducing apparent gains to 3.60–4.32 points.
Auditing reverses the method ranking
Under equal candidate budgets (three model calls, nine slots), independent resampling (Matched) exceeds mutation-based evolution (Evolve) by 18.83 and 9.72 points on development and 10.91 and 6.01 points externally, with all four paired intervals strictly negative. The modest audited Evolve-over-One-shot gain of about 2 points externally is metric-sensitive: held-out task detection changes by −1.02 and +0.15 with zero-crossing intervals, and audited task coverage falls. The implication is that the reported advantage of mutation-based evolution is attributable to the oracle artifact and additional search opportunity, not to the mutation operator itself. A robustness analysis confirms that all 12 model–cohort–seed gain-inflation intervals have positive lower bounds and all 12 audited Evolve-minus-Matched intervals have negative upper bounds.
Mechanism evidence
Decomposition of 2,836 real faults shows that public examples and official exact-output anchors expose 41.61% and 27.79% of faults respectively, while a comparable 11.28% family is exposed only through public mutations that the panel disconfirms—92.45% (Qwen2.5) and 84.52% (Qwen3.5) of detections in that family are audit-disconfirmed. Unsafe mutation families concentrate spurious credit, and no fault subgroup reverses the Evolve–Matched ranking.
A blinded human audit of 300 panel-disconfirmed inputs by two doctoral reviewers yields five-class agreement of 91.67% (κ = 0.823) and design-weighted estimates of 94.41% jointly invalid and 3.60% jointly valid. This supports invalid inputs as the dominant mechanism while establishing that panel disagreement is informative evidence, not semantic proof: a small nonzero fraction of disconfirmed inputs is genuinely valid, and one jointly valid case supports an auditor-side output.
Feedback attribution remains unresolved
The placebo results are the paper's most cautious finding. On the exposed external cohort, Real feedback appears to hurt Qwen2.5 (−6.11 vs. plain resampling) and help Qwen3.5 (+2.73), but the placebo reproduces both directions almost exactly (−6.24 and +3.23), leaving Real–Placebo at +0.13 and −0.50 points with zero-crossing intervals. The model-dependent shift is therefore attributable to iterative scaffolding rather than behavior-aligned feedback content. On the qualification-amended held-out cohort, Real–Placebo is +1.99 [0.08, 3.88] and +0.28 [−1.41, 2.03]; the first misses the frozen +2 superiority threshold and the second misses the equivalence boundary by 0.03, so both decisions are inconclusive. A post-hoc replay matching observed reference-valid candidate yield changes the contrasts to −0.30 and +0.78, all zero-crossing, indicating that equal nominal slots do not guarantee equal usable opportunity (Qwen2.5 yields 0.837 vs. 0.794 confirmed candidates per round for Real vs. Placebo). The paper is explicit that E15 is post hoc and E16-B is qualification-amended, so RQ6 provides triangulated diagnostic evidence, not confirmation that aligned feedback has zero effect.
Limitations and open questions
The paper concedes several boundaries. Accepted-program agreement is a benchmark-specific empirical oracle, not specification conformance; accepted programs may share latent defects, and the study restricts to exact-output Python tasks, excluding special checkers. The human audit is structured judgment, not ground truth—reviewers may share interpretation biases, and a 67-case adjudication packet remains unresolved. The placebo removes fine-grained attribution but retains coarse progress signals, so Real–Placebo estimates only one component of feedback value. E16-B's qualification amendment admits tasks with more exact-compatible accepted programs than excluded tasks, and no outcomes exist for excluded tasks, so selection bias cannot be ruled out; profile analyses of nested fault thresholds show non-monotone point estimates. Both models are from the Qwen family, so cross-family generality and scale effects are untested; a size-matched model from an independent family is identified as the highest-priority replication. The yield-matched replay conditions on a post-treatment mediator and demonstrates sensitivity, not a causal decomposition.
Conclusion
The paper establishes three quantified findings: single-reference evaluation inflates apparent test-evolution gains by 9.46–14.85 percentage points; after independent audit, equal-slot resampling outperforms mutation-based evolution by 6.01–18.83 points in every condition; and a three-round execution-feedback loop's model-dependent benefit is reproduced by a density-matched placebo, with held-out attribution contrasts inconclusive under pre-specified rules. The contribution is an evaluation protocol—freeze-then-audit, candidate-budget matching, cross-fitted real faults, and interaction-matched placebos—that separates verifier artifacts, interaction scaffolding, and grounded feedback credit, and that should precede any claim of learning in feedback-driven test generation.
