- The paper demonstrates that specification-grounded tests achieve 100% final correctness on eight core tasks and detect 30 of 30 buggy drafts, compared with 60% correctness and 18 of 30 detections for edge-prompted ungrounded tests.
- The paper finds that explicit specification content drives most of the improvement, while enumerating each rule adds a smaller benefit by ensuring every requirement becomes a testable case.
- The paper shows that grounding improves both bug detection and precision across models and external oracles, often outperforming larger models, extra test budgets, property-based testing, and agentic repair workflows at minimal added cost.
The question: why do AI-written tests help?
LLMs generate code that typically handles the common case while failing on unstated edge conditions — a missing input check, an unhandled boundary. The standard remedy is to have the model write its own tests and repair until they pass, a pattern instantiated by CodeT, AlphaCodium, and related flows (&&&2the union of the rules.2&&&, &&&2IIII2&&&). Prior work establishes that such tests help; it does not establish why. Two hypotheses imply opposite practitioner advice: quantity (more tests catch more bugs) versus grounding (tests derived from an explicit specification target the behaviours that matter). This paper isolates that single factor through a controlled design and concludes decisively for grounding.
Experimental design
The setup holds everything constant except one line of the test-writer's prompt. A code model writes one implementation from a prose ticket (no planted bugs). A fixed tester (Claude Sonnet 4.6 throughout the main study) then writes PRESERVED_PLACEHOLDER_2the union of the rules.2^ tests under one of two prompts: spec receives the specification enumerated into PRESERVED_PLACEHOLDER_2IIII2^ checkable rules and writes one test per rule; free+ sees only the ticket but is explicitly told to test invalid inputs and edge cases. Including free+ is what makes the comparison fair — it already knows to probe edges, so any remaining spec advantage reflects grounding rather than a naive reminder. Additional controls include free (no edge prompt), free2k (doubled budget), blind self-refine, and oneshot. Correctness is judged by an independent, hand-written gold suite no model ever sees, deliberately broader than the rule list so passing cannot be "2the union of the rules.2" The benchmark comprises 2IIII28 specification-completeness tasks (8 core, 4 held-out, 6 scale-up) plus 8 logic tasks as a scope control. Significance is assessed conservatively at the task level via sign tests, with pre-registered go/no-go criteria committed before results were examined.
Primary results
On the eight core tasks across 48 instances (two code-model tiers × three seeds), final correctness was 38% for oneshot and free alike, 42% for free2k, 62the union of the rules.2% for free+, and 2IIII2the union of the rules.2the union of the rules.2% for spec, with detection of 32the union of the rules.2/32the union of the rules.2^ buggy drafts versus 2IIII28/32the union of the rules.2^ for free+. Doubling the ungrounded budget barely helps, and unioning eight independent free+ suites plateaus at 52% detection on the real-world slice while a single grounded draw reaches 2IIII2the union of the rules.2the union of the rules.2%. The task-level sign test over all 2IIII28 specification-completeness tasks gives p=0.002, with spec strictly better on every task showing any difference.
The gap replicates broadly:
| Setting |
free+ |
spec |
Gap |
| Core, 8 tasks |
62the union of the rules.2% |
2IIII2the union of the rules.2the union of the rules.2% |
+42the union of the rules.2^ |
| Three Claude tiers |
62% |
2IIII2the union of the rules.2the union of the rules.2% |
+38 |
| Held-out, 4 tasks |
25% |
62IIII2% |
+36 |
| Scale-up, 6 tasks |
78% |
96% |
+2IIII28 |
| GPT-5.3-codex (full-stack) |
72% |
2IIII2the union of the rules.2the union of the rules.2% |
+28 |
| Gemini 3.5 Flash (full-stack) |
72% |
92% |
+2IIII29 |
A capability-substitution result follows: the smallest model with grounded tests reaches 2IIII2the union of the rules.2the union of the rules.2%, while the largest run plain reaches only 54%; closing that gap with model scale alone costs roughly 2.2× the tokens, whereas grounding adds almost nothing over the fair baseline (~$6–7 per 2IIII2k tasks).
Isolating the mechanism
An ablation separates the three components bundled in spec — content, enumeration, and coverage planning. Given only a chain-of-thought decomposition plan with no specification content (decomp), the tester catches 2/32the union of the rules.2^ bugs, indistinguishable from free. Given the same specification content as a plain paragraph (prose), it catches 27/32the union of the rules.2. Full enumeration recovers the last three (32the union of the rules.2/32the union of the rules.2), all on split_money where prose had the right content but failed to convert every rule into a triggering test. Specification content is therefore the dominant cause; enumeration adds only a small margin against this rule-to-test gap.
Stronger automated baselines do not overturn the finding. A property-based generator in the QuickCheck/Hypothesis tradition detects 28/32the union of the rules.2^ but invents out-of-spec assertions — two of fifteen batteries wrongly reject the trusted reference by asserting roman_to_int("2IIII2 must raise. An AlphaCodium-style agentic flow on GPT-5.3-codex reaches 72%, exactly matching the codex free+ baseline and well short of codex spec at 2IIII2the union of the rules.2the union of the rules.2%; without the enumerated spec, its self-generated tests miss non-obvious edges and dual-fixing can rewrite correct tests to match buggy code.
Grounding also improves precision, not just sensitivity. On trickier behavioural specs, spec raises 2the union of the rules.2% false alarms versus 33% for free+, whose testers systematically invented expectations (all three draws assumed excel_column(2the union of the rules.2) should raise when the reference returns ""). Against real production oracles the effect sharpens: reimplementations judged against CPython show free+ falsely rejecting correct code 68% of the time, and against PyPA packaging, 2IIII2the union of the rules.2the union of the rules.2% — turning a suite that rejects everything into one that classifies perfectly once grounded. Grounded test-pass fraction is also a far better selective-prediction signal (risk–coverage AURC 2the union of the rules.2.257 vs. 2the union of the rules.2.577).
Scope and robustness
The claim is bounded to specification-completeness defects. On eight logic tasks there were zero natural bugs among 45 valid submissions — strong models get well-specified algorithms right in one pass, leaving nothing to catch. A port to HumanEval+ under EvalPlus's own oracle confirms this null publicly: pooled detection is comparable (2IIII23 vs. 2IIII24 of 22 bugs), because well-specified algorithmic benchmarks rarely elicit the targeted defect class. Robustness checks include six tasks whose edges are fixed by external standards (RFC 792IIII2, CSS colour grammar, ISBN-2IIII2the union of the rules.2), where the gap widens to +68 (spec 2IIII2the union of the rules.2the union of the rules.2% vs. free+ 32%); stateful multi-step tasks; and deliberately corrupted specs, which degrade gracefully along separable axes — completeness governs detection, correctness governs precision, neither fails silently.
Limitations and open questions
The study is a controlled isolation, not a leaderboard. Tasks are small functions (with a modest stateful extension); repository-scale benchmarks like SWE-bench are out of scope. Bugs come from the models themselves, so difficulty is uncontrolled, though the independent oracle mitigates this. The fixed tester is deliberately strong (99–2IIII2the union of the rules.2the union of the rules.2% accuracy); weaker testers retain a precision floor even when grounded (GPT-4o-mini keeps 35–67% false alarms), and the tester sweep shows grounding helps only once the tester can apply a rule correctly. Auto-derived specifications recover only 47% of hand-written detection because they restate the stated happy path and omit unstated error conditions; prompting for edges restores detection but invents wrong error semantics, raising false alarms to 33%. The binding cost is therefore not writing rules (~5 per task) but knowing correct edge semantics — measuring that elicitation cost remains open, as does a public benchmark of under-specified tickets. Mutation score, notably, favours free+ slightly (95.9% vs. 93.9%), demonstrating that mutation strength rewards sensitivity regardless of expectation correctness and cannot substitute for end-to-end correctness judging.
Conclusion
This paper identifies specification grounding as the principal mechanism behind test-driven repair for LLM code, demonstrated with a fair, edge-prompted baseline, an independent oracle, pre-registered criteria, and replication across three model families and multiple external oracles. Grounding improves both sides of the confusion matrix simultaneously — more real bugs caught and fewer false alarms — because both failures stem from the same missing information: the intended behaviour at unstated edges. The practical prescription is inexpensive and specific: write down the edge rules and turn each into a check before buying a larger model or more tests. The open problem the paper leaves is obtaining accurate specifications at scale, since the hard part is knowing the correct edge behaviour in the first place.