Grounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test Generation

Abstract: LLM-based agents are increasingly used for coding tasks, where they have outperformed many classical approaches and scaled to repository-level tasks, such as test generation. However, when directly prompted to generate tests, these agents can fail to reason about the code and its underlying contracts, thereby missing edge cases and behavioral boundaries that affect test quality. To address this limitation, we propose Spec-Driven Test Generation, where we instruct an agent to first reason about -- and explicitly document -- code pre-conditions, post-conditions, and undefined behaviors. This intermediate semi-formal specification acts as a cognitive scaffold to guide subsequent test generation. Our evaluation on production bugs from Google shows that the spec-driven agent can deliver a 9.8 percentage points ($p = 0.0352$) improvement in bug detection rate and a 2.5 percentage point ($p = 0.0034$) improvement in branch coverage, compared to a traditional test generation agent baseline. Using LLM-as-a-Judge, we further show that test suites generated by the spec-driven agent are superior to the baseline and human-authored tests in 77.8% and 56.7% of the cases, respectively, and demonstrated improvements on following best practices, readability, and edge-case coverage.

• The paper introduces a two-phase workflow that first infers semi-formal contracts and then uses them to guide unit-test generation, improving five-run bug detection from 53.4% to 63.2% across 90 Google production bugs.
• The spec-driven agent increased branch coverage from 46.4% to 48.9% and produced stronger edge-case tests, while maintaining nearly identical test pass rates and leaving mean line coverage essentially unchanged.
• The approach’s effectiveness depends on recovering the violated contract—raising conditional detection from 19.4% to 54.9%—but it required 38% more tokens, highlighting a quality-versus-cost trade-off.

The paper presents Spec-Driven Test Generation, a two-phase agentic workflow intended to improve the semantic effectiveness of LLM-generated unit tests. Its central claim is that direct test synthesis leaves the agent without an explicit behavioral oracle, encouraging superficial path exploration, missed boundary conditions, and weak assertions. The proposed solution is to require the agent to first infer and document a semi-formal contract for the target code, then use that artifact to guide test construction. Evaluated on 90 historical production bugs from Google, the approach improves fault detection and branch coverage relative to an otherwise identical direct-generation baseline, while incurring a substantial token overhead (2608.17177).

Problem formulation and conceptual basis

The work adapts Design by Contract to repository-level agentic test generation. Traditional contracts are typically written by developers before or alongside implementation, whereas the setting considered here involves existing code whose intended behavior is only partially explicit. The agent therefore performs retroactive contract inference using source code, documentation, comments, existing usage patterns, and other repository context. The objective is not formal verification or proof of functional correctness. Rather, the specification is a structured natural-language artifact that provides a defect-oriented testing oracle.

For each code unit, the specification contains four elements: a behavioral description, preconditions, postconditions, and local test suggestions. Conditions are additionally marked as either already tested or untested. An untested condition is intended to yield a concrete test obligation. This representation combines the reviewability and expressive flexibility of natural language with the organizational discipline of contract-based reasoning. It is consequently less rigorous than Hoare logic, JML, Dafny, or SMT-based specifications, but more directly usable in heterogeneous, multilingual codebases.

The approach also differs from dynamic invariant miners such as Daikon. Dynamic miners treat observed executions as evidence of intended behavior, which can encode faulty behavior as an invariant. The proposed agent instead attempts to infer developer intent from multiple semantic sources. That distinction is important in a historical-bug setting: the target is not merely to summarize what the buggy or fixed implementation does, but to identify the behavioral constraint that the defect violates.

Spec-driven generation framework

The framework separates specification extraction from test synthesis. In Phase I, the agent aggregates contextual information, infers behavioral boundaries, compares inferred conditions with available tests, and produces suggestions for uncovered conditions. The paper does not programmatically enforce that every API or condition is represented; completeness is encouraged through prompting and agent behavior. The artifact can optionally be reviewed and amended by a developer before synthesis, although the empirical evaluation deliberately omits this human-in-the-loop stage.

Figure 1: Overview of spec-driven test generation.

In Phase II, the specification and its test suggestions are supplied to the same agent architecture used for the baseline. The agent implements the suggestions, modifies only the permitted test and build files, executes the tests, and iterates based on compilation or runtime feedback. The specification therefore functions both as a planning document and as an oracle-oriented context scaffold. Its intended effect is not simply to increase the volume of generated tests, but to focus generation on input constraints, error behavior, state transitions, side effects, and branch boundaries.

The experimental comparison isolates this intermediate reasoning step. Both configurations use Gemini 3 Flash, identical inference parameters, the same repository-navigation tools, and the same agentic harness. The baseline receives source code and is directly instructed to produce a comprehensive test suite. The spec-driven agent must first generate the contract artifact and then synthesize tests from it. This control is methodologically valuable because it reduces the likelihood that differences arise from model choice, tool access, or execution infrastructure.

Industrial evaluation protocol

The evaluation uses 90 reproducible, human-filed production bugs from Google’s internal issue-tracking system. Each bug is represented by a buggy implementation and its corresponding fixed implementation. The dataset spans C++, Java, Python, and Go, but is restricted to fixes affecting a single production source file, with a corresponding test file and optional build configuration changes. This restriction makes fail-to-pass evaluation tractable, although it limits the scope of the conclusions for multi-file defects and broader integration failures.

The study adopts a greenfield setup. Existing tests are removed before agent execution, and the agent is given the fixed source without access to the issue description, commit message, or fix diff. Generated tests must compile and pass on the fixed implementation. The workspace is then reverse-patched to the buggy version, and a bug is counted as detected if at least one generated test fails. Build failures do not count as detections. Post-execution audits discard runs in which the agent modifies the source under test or unrelated pre-existing files.

The primary quantitative measure is detect@k, where $k$ independent stochastic generations are attempted for each bug. The paper reports results for $k$ from 1 through 5, using bug-level bootstrap confidence intervals and McNemar’s test for paired detection outcomes. Structural coverage is measured using line and branch coverage on the fixed implementation, with paired comparisons evaluated using the Wilcoxon signed-rank test. Qualitative test quality is assessed by Gemini 3.1 Pro, which compares anonymized suites across best-practice adherence, readability, edge-case coverage, and overall superiority. Five judge invocations are aggregated by majority vote.

Fault detection and structural coverage

Both agents produce valid test suites at nearly identical rates. At one attempt, pass@1 is 94.4% for the baseline and 94.2% for the spec-driven agent; at five attempts, both reach 98.9%. This result is important because it rules out a simple explanation in which the proposed method improves bug detection merely by producing more compilable tests. The difference is semantic rather than syntactic.

Metric	Baseline	Spec-driven	Difference	Significance
detect@1	36.9%	41.1%	+4.2 pp	$p = 0.3075$
detect@3	49.4%	56.6%	+7.2 pp	$p = 0.0574$
detect@5	53.4%	63.2%	+9.8 pp	$p = 0.0352$
Line coverage	74.8%	74.4%	-0.4 pp	$p = 0.3659$
Branch coverage	46.4%	48.9%	+2.5 pp	$p = 0.0034$

The spec-driven agent outperforms the baseline at every execution budget. At $k=5$, it detects 63.2% of historical bugs compared with 53.4% for the baseline, an absolute improvement of 9.8 percentage points that is statistically significant under McNemar’s test. The performance gap increases with additional samples: from 4.2 points at $k=1$ to 9.8 points at $k=5$. The implication is that independent sampling is more productive when each generation is constrained by an explicit behavioral scaffold, although the study does not establish whether the improvement results from greater diversity, better prioritization, or both.

The overlap analysis further supports a meaningful qualitative difference. The two agents detect 45 bugs in common; the spec-driven agent uniquely detects 12 bugs missed by the baseline, whereas the baseline uniquely detects only 3 bugs missed by the spec-driven agent. Thus, the improvement is not attributable solely to a uniform increase in detection probability. The intermediate contract appears to expose some defect classes that direct prompting systematically overlooks.

The coverage results are more selective than the detection results. Mean line coverage is essentially unchanged, with a nonsignificant decrease from 74.8% to 74.4%. Branch coverage, however, increases from 46.4% to 48.9%, and the 2.5-point difference is statistically significant. This divergence is technically consequential: line coverage can be increased by exercising straight-line code without testing alternative outcomes, whereas branch coverage more directly reflects exploration of conditional behavior, exception paths, and boundary decisions. The result supports the paper’s claim that the specification encourages deeper control-flow exploration rather than merely increasing execution volume.

The data also reveal substantial per-bug heterogeneity. At $k$0, 65.6% of baseline bugs fall at either 0% or 100% detection across runs, compared with 57.8% for the spec-driven agent. The proposed method shifts some bugs from complete nondetection into an intermediate regime in which detection is conditional. This reduces, but does not eliminate, the strongly bimodal difficulty distribution. Consequently, the reported confidence intervals remain wide, and the aggregate mean should not be interpreted as uniform effectiveness across defects.

Qualitative rigor of generated tests

The qualitative evaluation compares 83 successfully generated spec-driven suites with baseline suites and with the original developer-written suites. The judge exhibits at least 90% self-agreement across criteria and a tie rate no greater than 5.6%, although these reliability measures concern consistency of the evaluator rather than agreement with human experts.

Figure 2: Pairwise LLM-as-a-Judge comparison of baseline-generated and spec-driven test suites.

Against the baseline, the spec-driven suites are judged overall superior in 77.8% of cases. They are preferred for testing best practices in 65.6% of cases, readability in 68.9%, and edge-case coverage in 83.3%. The strongest result is therefore not general stylistic preference but boundary-oriented behavior. The judge identifies more explicit validation of exceptional inputs, missing optional fields, boundary values, and negative paths. It also reports more localized assertions, stronger naming conventions, and more idiomatic use of test-framework facilities.

These qualitative outcomes are consistent with the branch-coverage and fault-detection results. A contract organized around preconditions and postconditions naturally directs the agent toward behaviorally distinct states rather than only method invocation. Nevertheless, the LLM-as-a-Judge design introduces construct-validity concerns. The evaluator may favor verbosity, stylistic regularity, or outputs resembling its own training distribution. The paper mitigates this with anonymization, randomized ordering, a stronger judging model, and repeated voting, but does not provide human adjudication or inter-rater comparison.

Figure 3: Pairwise comparison between developer-written suites and spec-driven suites, showing competitive qualitative quality.

Compared with developer-authored tests, the spec-driven suites are rated overall superior in 56.7% of cases, which the paper interprets as approximate parity with human engineering rigor. The agent’s strengths include narrow assertions, reduced boilerplate, explicit negative-path testing, and robust concurrency constructs such as absl::Notification. Human-written suites remain stronger on intricate algorithmic scenarios, complex state transitions, and timezone-sensitive behavior. The agent also occasionally violates public API encapsulation through reflection-based access to private methods. The comparison therefore does not establish general superiority over human testing; it indicates that, under this evaluation protocol, contract-guided generation can produce suites competitive with developer-authored artifacts on several observable quality dimensions.

Contract coverage as an explanatory metric

The paper introduces ContractCoverage@k to separate specification quality from downstream test-generation quality. The metric asks whether the generated specification explicitly captures the behavioral contract violated by the historical bug. It is deliberately defect-oriented rather than a measure of complete semantic coverage. A specification can therefore receive positive Contract Coverage while omitting many valid behaviors unrelated to the selected defect.

ContractCoverage@1 is 61.1%, increasing to 69.7% at $k$1 and 78.9% at $k$2. More importantly, contract coverage strongly predicts fault detection. When the specification captures the violated contract, the resulting suite detects the bug in 54.9% of runs, compared with 19.4% when the contract is absent. Fisher’s exact test yields $k$3, with a moderate Phi association of $k$4.

This conditional analysis clarifies the mechanism behind the aggregate improvement. Specification generation is not merely an extra textual planning step: successful contract recovery substantially increases the probability that synthesis will produce a fault-revealing test. However, the 54.9% conditional detection rate also shows that a correct specification is insufficient. In 110 runs, the agent identified the relevant contract but still failed to detect the bug. Failures include inadequate input data, omitted scenarios, weak assertions, flawed test logic, and invalid test code. Conversely, 34 runs detect bugs without Contract Coverage, likely through generic structural execution or broad crash assertions. The specification is therefore a useful but imperfect oracle.

The paper categorizes specification-generation failures into omitted methods, missing error handling, omitted data transformations, abstraction of constants, and missing execution constraints. These categories identify a central bottleneck: the agent may produce a coherent-looking contract while omitting exactly the low-level value mapping, ordering constraint, or exceptional behavior that distinguishes correct from incorrect execution. The artifact’s semi-formal nature improves accessibility but does not provide machine-checkable guarantees against omission or hallucination.

Cost and efficiency trade-offs

The method requires materially more inference. Across five runs of the 90-bug dataset, the baseline consumes 243.9 million tokens, whereas the spec-driven configuration consumes 336.7 million, a 38.0% increase. Input consumption rises by 36.2%, primarily because specifications are added to the synthesis context, while output consumption rises by 59.1%, reflecting both specification generation and larger test suites.

The spec-driven agent detects 57 unique bugs compared with 48 for the baseline, an 18.8% increase in unique bugs. However, tokens per unique bug detected increase from 5.1 million to 5.9 million, or 16.2%. The approach therefore improves absolute yield but not token-normalized efficiency. This distinction matters for deployment decisions: the method is advantageous when additional inference cost is acceptable and fault-finding quality is prioritized, but the evidence does not support the claim that specification grounding is computationally cheaper.

Limitations and open questions

The empirical scope is limited to 90 bugs from one organization and to defects whose fixes modify a single production file. Google’s monorepo, internal conventions, documentation density, build infrastructure, and testing practices may differ substantially from open-source repositories or other industrial environments. The study also evaluates only Gemini 3 Flash as the generation model and Gemini 3.1 Pro as the judge. Absolute rates may change with model family, prompting strategy, context-window behavior, or tool implementation.

The greenfield protocol provides experimental control but removes existing tests that would normally supply naming conventions, fixtures, behavioral examples, and implicit contracts. Results therefore characterize specification-guided generation under deliberately sparse evidence, not necessarily incremental augmentation of a mature test suite. Similarly, historical bug detection measures regression-test recovery rather than discovery of previously unknown defects.

The LLM-as-a-Judge is the principal threat to qualitative and specification-validity claims. Repeated majority voting improves consistency but cannot establish semantic correctness. ContractCoverage itself depends on a known fix and evaluates only the contract relevant to that fix. It does not penalize over-specification, contradictory conditions, hallucinated requirements, or omissions outside the selected defect. The paper also does not include an ablation separating the effects of structured contract extraction from simpler planning artifacts such as pseudocode, explicit test plans, execution traces, or chain-of-thought-like decomposition. Finally, the optional HITL stage is not evaluated, leaving open whether expert curation improves detection enough to justify its review cost and whether it corrects rather than reinforces agent-generated misconceptions.

Conclusion

The paper provides controlled empirical evidence that inserting semi-formal precondition and postcondition extraction before test synthesis improves LLM-agent test generation on real industrial defects. At five runs per bug, the approach raises historical bug detection from 53.4% to 63.2% and branch coverage from 46.4% to 48.9%, while maintaining essentially identical test-suite pass rates. Its strongest explanatory result is the association between Contract Coverage and detection: capturing the violated contract increases conditional detection from 19.4% to 54.9%. The gains come with a 38.0% token overhead and do not establish superiority across organizations, models, or nonhistorical testing tasks. Within the evaluated setting, however, the findings support contract extraction as a substantive semantic intervention rather than merely an additional planning format (2608.17177).