POSTCONDBENCH: Benchmarking Correctness and Completeness in Formal Postcondition Inference

Abstract: Formal postconditions precisely characterize program behavior and support debugging, testing, and verification, but writing them requires substantial expertise and effort. This has motivated recent work on automatically generating postconditions from code and natural-language artifacts using LLMs. However, evaluation remains a key bottleneck. Existing benchmarks primarily emphasize correctness under limited evaluation settings, often relying on surface-form matching or manual assessment on small or synthetic datasets. We introduce POSTCONDBENCH, a multilingual benchmark for evaluating method-level postcondition generation from real-world software. POSTCONDBENCH comprises 420 Python and Java tasks drawn from 121 open-source projects, each paired with a high-quality ground-truth postcondition set constructed with expert involvement. To enable automatic evaluation, POSTCONDBENCH provides a runnable execution environment and operationalizes completeness via defect discrimination: a postcondition set is more complete if it is violated by more defective implementations while remaining satisfied on correct executions. Using POSTCONDBENCH, we formulate three generation settings and evaluate five SOTA LLMs. Our results reveal a substantial gap between correctness and completeness, and show that repository-level dependencies and method complexity exacerbate this gap.

• The paper introduces PostcondBench, a multilingual benchmark of 420 Python and Java tasks with repository context, tests, mutants, and expert-refined postconditions evaluated through JML and icontract.
• The study finds a persistent correctness–completeness gap: at one generation, completeness trails correctness by 0.063–0.423, and fewer than half of most models’ correct outputs are complete.
• The results show that specification-language misuse, under-specified return behavior, external dependencies, and longer methods are major failure sources, while mutation-based completeness remains stable across substantial ablations.

Postcondition inference—generating formal specifications that constrain program state after method execution—has attracted renewed attention with the application of LLMs, yet reliable evaluation remains a bottleneck. Existing benchmarks either restrict correctness assessment to surface-form matching against a single ground truth or rely on manual judgment of specification quality on small, synthetic datasets. The paper introduces PostcondBench, a multilingual benchmark that addresses this gap by providing 420 method-level postcondition generation tasks in Python and Java, drawn from 121 real open-source repositories, together with an automatic evaluation platform that measures both correctness and completeness (2605.03356).

Benchmark design

Each benchmark instance is a tuple containing the method signature, natural-language (NL) description, implementation, a test suite $\mathcal{T}$, a set of implementation mutants $\mathcal{M}$, and a ground-truth postcondition set $\mathcal{P}$. Postconditions are expressed in mature specification frameworks—JML for Java and icontract for Python—rather than the native assertion statements or custom formats used in earlier work. This allows the benchmark to capture intra- and inter-class dependencies and pre-state information via constructs such as \old(·), and to support full logical connectives (implication, equivalence, inequivalence), which the authors note are insufficiently supported by prior formats.

The construction pipeline proceeds in several stages. Repositories are mined from GitHub (top-200 tags, permissive license, top 5,000 by stars per language), and an LLM-driven environment setup loop—iteratively querying gpt-o4-mini to repair configuration files—successfully builds runnable environments for 941 of 10,000 repositories (9.4%), slightly better than SWE-agent. Methods are filtered by comment quality (more than 15 English words), complexity (at least 15 lines of code or cyclomatic complexity ≥ 3), and test coverage (≥ 90% line coverage), yielding 1,030 Python and 1,476 Java candidate methods.

Mutants are generated through two complementary mechanisms: operator-based mutation (reusing Mutmut operators for Python and reproducing PIT operators for Java) and LLM-based mutation targeting conditionals, loop headers, and method calls. Mutants that pass the original test suite or throw exceptions under normal execution are discarded, and methods must retain at least five valid mutants, leaving 999 Python and 1,431 Java methods with an average of 24.8 mutants each.

Ground-truth postconditions are drafted by gpt-5-mini with reasoning enabled, prompted with the method, repository context (via a code-interpreter tool), and per-mutant diffs. A postcondition set is correct if the original implementation satisfies it across the developer test suite, and bug-complete if it is correct and additionally kills every generated mutant. When the LLM-drafted set is not bug-complete, two domain experts (each with over five years of programming experience and over one year of formal-specification experience) refine it until completeness is reached. Final task selection uses farthest-first traversal over CodeBERT embeddings of method headers, producing 210 methods per language with an average line coverage of 98.8–99.3%, roughly 30 lines of code, cyclomatic complexity around 5, and 3.6–5.5 ground-truth postconditions per method.

Evaluation setup

The authors evaluate five state-of-the-art LLMs—GPT-5, Claude-4.5 (Sonnet), LLaMA-4 (Maverick), Qwen3-32B, and Gemma-3-27B—under three input settings: code-only (C2P), NL-only (N2P), and code plus NL (F2P). For each method, five candidate postcondition sets are independently sampled, and results are reported as Corr@k (probability that at least one of $k$ generations is test-correct) and Comp@k (probability that at least one is complete).

The correctness–completeness gap

The central empirical finding is a large, systematic gap between correctness and completeness. At @1, aggregated across languages and input settings, Claude-4.5 achieves the highest correctness (0.629) while GPT-5 achieves the strongest completeness (0.255); the absolute gap $\Delta@1$ ranges from 0.063 to 0.423, and the conditional completeness rate $\rho@1 = \text{Comp@1}/\text{Corr@1}$ falls below 0.5 for most models—fewer than half of test-correct outputs are complete. Notably, higher correctness does not imply a smaller gap: Claude-4.5, the most correct model, exhibits the largest gap (0.423). Increasing the sample budget to @5 improves both metrics (GPT-5 reaches 0.802 Corr / 0.446 Comp; Claude-4.5 reaches 0.822 / 0.292) but does not close the gap. Method-level gap distributions are broad and right-skewed, with median gaps around 0.2 and upper quartiles reaching 0.6–0.8, indicating the gap is widespread rather than driven by outliers. The gap is most severe under N2P, while C2P and F2P are similar, suggesting code provides most of the constraining signal for completeness.

The authors also test whether the gap is an artifact of prompt-level specification-language unfamiliarity. Adding a brief grammar-guidance block (summarized from icontract and JML documentation) to the F2P prompt improves both metrics—Claude-4.5 Corr@1 rises from 0.629 to 0.814 and Comp@1 from 0.207 to 0.283—but the correctness–completeness gap persists, supporting the claim that completeness is a substantive modeling deficiency rather than a formatting issue.

Failure modes and missed behaviors

Manual analysis of 50 incorrect postcondition sets from GPT-5 and Claude-4.5 attributes 54% of failures to specification-language or API misuse. Many of these are not superficial: more than half of the Java mistakes in this category misuse post-state variables inside \old(·), indicating an insufficient understanding of the pre-/post-state boundary rather than mere syntax errors. Semantic overreach—formalizing behaviors unsupported by the implementation or documentation—accounts for 34%, with low-frequency syntactic mistakes and unguarded null dereferences making up the remaining 12%.

For correct-but-incomplete postconditions, inspection of 50 (postcondition set, unkilled mutant) pairs shows that 78% of incompleteness stems from under-specified return-value behavior, where constraints cover only special cases, types, or coarse ranges. Over one third of missed cases involve simple scalar returns, implying that incompleteness frequently arises from weak semantic constraints rather than complex output structures. Unconstrained object state accounts for 16%, and missing defensive guards for 6%.

Effects of method characteristics

Two method-level factors systematically degrade performance. Dependency complexity: standalone methods (runnable with only built-ins) outperform dependency-heavy ones in 51 of 60 paired comparisons (85%); average Comp@1 rises from 0.035 to 0.090 in Python (2.6×) and from 0.188 to 0.316 in Java (1.7×) when moving from non-standalone to standalone methods, indicating that external dependencies make sufficiently constraining specifications harder to produce. Method length: performance decreases monotonically with lines of code, with the steeper drop on completeness—for example, Claude-4.5 under C2P falls from Comp@1 of 0.388 (short) to 0.206 (medium) to 0.076 (long).

Stability of the completeness metric

Because the completeness metric depends on the mutant set, the authors conduct three ablations. Removing any single mutation operator changes Comp@1 by at most 0.004 in Python and ≤ 0.001 for most Java operators. Randomly removing 5 of 11 operators or 50% of LLM-based mutants (50 trials each) shifts mean Comp@1 by less than 4% relative to the full-mutant baseline, with standard deviations ≤ 0.004. Sampling mutant subsets shows clear saturation at ≥ 80% of the mutant budget: GPT-5's mean Comp@1 varies by only ≈ 0.003 between the 80% and 100% subsets. These results indicate the metric is neither dominated by a particular mutation family nor overly sensitive to operator choice, though the authors are explicit that mutation-based completeness is an operational proxy for behavioral coverage, not a definitive measure of semantic completeness.

Limitations

The paper concedes several restrictions on scope. Methods or mutants requiring specification constructs unsupported by JML or icontract—most exclusions arising from the representational limits of these frameworks—or remaining unkillable under available tests are removed, which may bias evaluation of behaviors such as iterator or concurrency semantics; exceptional postconditions are out of scope entirely. The execution-based metrics also restrict the benchmark to methods with high line coverage, and the authors acknowledge that high coverage does not guarantee strong test assertions, which can produce optimistic correctness estimates. Finally, the environment-setup success rate of 9.4% means the benchmark samples from the subset of repositories amenable to automated build and test.

Conclusion

PostcondBench provides a multilingual, repository-level benchmark with expert-refined, bug-complete ground-truth postconditions and an automatic evaluation platform that operationalizes completeness through defect discrimination. Evaluation of five frontier LLMs shows that test-correct postconditions are frequently incomplete—completeness lags correctness by 0.06–0.42 at @1 and remains below 0.5 of correctness for most models—and that repository-level dependencies and method length widen this gap. The benchmark's central open question is how to generate postconditions that are not merely consistent with observed executions but behaviorally discriminating, particularly for dependency-heavy and longer methods.