Teaching Code LLMs to Reason with Intermediate Formal Specifications

Abstract: Unlike natural-language specifications, executable formal specifications provide machine-checkable constraints for verifying, debugging, and repairing code. However, writing such specifications is labor-intensive, and existing LLM-based methods mainly infer whole-program pre/postconditions, missing the intermediate semantic commitments that programmers rely on when reasoning about an algorithm. Our study further shows that prompting current CodeLLMs often produces executable assertions that are syntactically invalid, trivial, or too weak to reject behavior-changing faults. In this paper, we study executable checkpoint specification generation, where assertions are inserted at meaningful internal program points to describe expected intermediate states. We introduce SpecCoder, a verification-guided CodeLLM training framework that learns from validated reference programs, behavior-changing mutants, and multi-turn specification-refinement traces. SpecCoder selects specifications that hold on correct executions while rejecting faulty executions, turning specifications from passive annotations into executable evidence. To evaluate this setting, we introduce HumanExec, a benchmark built from recent Codeforces competitive programming problems with test suites, reference solutions, and human buggy submissions, supporting three tasks: specification generation, program correctness checking, and program repair. Experiments on HumanExec show that SpecCoder substantially improves checkpoint-specification quality over base CodeLLMs. Across Qwen2.5-Coder models, SpecCoder improves inline-specification correctness by up to 55.8%, completeness by up to 358.1%, and executable assertion validity by up to 26.6%. These gains further translate to downstream correctness reasoning and repair, showing that executable checkpoints provide fine-grained evidence for reliable verification.

• The paper presents SpecCoder, a verification-guided training framework that enables LLMs to generate semantically rich, executable checkpoint assertions.
• It employs multi-turn, execution-guided refinement with correct and mutant code examples to significantly improve assertion correctness and discriminative completeness.
• Empirical evaluation on the HumanExec benchmark demonstrates enhanced correctness checking and repair, advancing AI-driven code verification.

Teaching Code LLMs to Reason with Intermediate Formal Specifications

Introduction

"Teaching Code LLMs to Reason with Intermediate Formal Specifications" (2607.04232) addresses a longstanding obstacle in software engineering automation: enabling LLMs for code to reason about program correctness through machine-checkable, executable intermediate specifications rather than relying solely on natural language or coarse pre/postconditions. The work introduces SpecCoder, a verification-guided training framework for code LLMs targeting the generation of inline checkpoint specifications, and presents HumanExec, a benchmark for evaluating specification generation, correctness checking, and program repair in the context of real-world competitive programming problems.

Motivation and Problem Formulation

Classical formal methods leverage executable assertions (e.g., invariants, preconditions, postconditions) as oracles for verification and diagnosis. However, LLM-based specification inference has been dominated by end-to-end, coarse-grained statements or natural language comments ill-suited for fine-grained semantic reasoning or automated verification. Manual authoring of high-quality, intermediate assertions remains prohibitively expensive for practical deployment at scale.

The paper identifies and systematically demonstrates two key limitations:

• Granularity: Existing LLM methods (NL2Postcond, SpecMind, SpecRover) focus on pre/postconditions and miss the stepwise, semantically meaningful checkpoints leveraged by human developers for compositional reasoning.
• Executability: Natural-language specifications are non-executable and cannot be automatically validated or used for runtime fault localization.

Empirical analysis in the paper shows that, even when LLMs are prompted to generate checkpoint assertions, the outputs are frequently trivial, non-executable, or semantically weak, resulting in low coverage of relevant faults and poor discriminative power.

SpecCoder: Verification-Guided Specification Generation

SpecCoder introduces a systematic, feedback-driven data synthesis and training framework for checkpoint specifications in code LLMs. The key idea is to replace purely textual generation with a process grounded in execution-based validation utilizing both correct and mutant (faulty) versions of code.

Pipeline Overview

Figure 1: Pipeline for mutant-guided checkpoint specification data construction.

The pipeline comprises:

1. Reference Solution Mining: Select reference implementations from LeetCode-style submissions that pass extensive test suites, establishing trusted ground truth for correct runtime behavior.
2. Mutant Synthesis: Generate one-line mutant versions (i.e., plausible faults) via targeted LLM prompting, retaining only non-trivial variants that pass some but not all tests.
3. Multi-Turn, Execution-Guided Refinement: Employ a teacher LLM to annotate checkpoint assertions, iteratively refining based on execution outcomes over both reference and mutant programs. Assertions must meet strict correctness (on reference) and completeness (bug-finding) thresholds before being accepted.
4. Supervised Data Construction: Aggregate accepted annotated code as fine-tuning supervision for two tasks: (a) enriching given code with checkpoint specifications, and (b) generating complete code with specifications directly from problem descriptions.

This approach operationalizes the dual objectives that a useful checkpoint assertion must be both correct for the intended computation and discriminative (able to reject behavior-changing mutants).

Benchmarking with HumanExec

To evaluate generalization and effectiveness under realistic developer errors, the authors introduce HumanExec, a benchmark derived from recent Codeforces problems. Each instance consists of a problem, solution, real human buggy submissions, and an executable test suite. Three tasks are supported: (i) checkpoint specification generation, (ii) program correctness checking, and (iii) program repair. Buggy programs in HumanExec reflect real-world developer errors instead of synthetic manipulations, enabling robust, realistic assessment of the impact of intermediate executable specifications.

Empirical Results

Specification Generation

SpecCoder is evaluated against vanilla, zero-shot Qwen2.5-Coder models ranging from 7B to 32B parameters. The quality of generated checkpoint assertions is measured by (a) validity (syntax/executability), (b) correctness (holds on reference executions), and (c) completeness (rejects faulty mutants).

Figure 2: Quality of generated checkpoint assertions.

Across all scales, Qwen2.5-Coder models fine-tuned with SpecCoder exhibit dramatic improvements in both assertion correctness and discriminative completeness. For Qwen2.5-Coder-14B, SpecCoder increases checkpoint correctness from 0.52 to 0.65 and completeness from 0.20 to 0.77, with 100% assertion validity. For the largest backbone (32B), correctness jumps to 0.81 and completeness to 0.68.

These results confirm that mere scaling or prompting is inadequate for robust, semantically meaningful checkpoint assertion synthesis; verification-guided fine-tuning is required to produce nontrivial, fault-discriminating specifications.

Checkpoint usage analysis further reveals that the rate of nontrivial assertions—those rejecting at least one mutant execution—increases from under 40% in the base models to over 80% post SpecCoder tuning.

Model Internalization of Assertion Syntax and Placement

To assess whether improvements reflect genuine model capacity or only data memorization, per-assertion perplexity is evaluated for held-out specifications.

Figure 3: ECDF of per-assertion perplexity on held-out inline checkpoint specifications. Each panel compares a base Qwen2.5-Coder model with its SpecCoder-tuned counterpart on the same assertion tokens.

Lower perplexity in SpecCoder-fine-tuned models over assertion tokens indicates that the model internalizes both the syntax and preferred placements of checkpoint specifications, suggesting systematic learning rather than prompt-based surface patterning.

Qualitative Comparison

Concrete outputs confirm that vanilla LLMs tend to insert generic or natural-language comments in place of executable predicates, even when instructed otherwise, while SpecCoder yields precisely scoped, in-scope Boolean predicates relevant to the program state.

Figure 4: Qualitative comparison of specification generation.

Correctness Checking and Repair

By propagating the improved checkpoint annotations into correctness-checking and repair scenarios, the impact of SpecCoder extends beyond specification generation. On HumanExec, SpecCoder-tuned models improve balanced accuracy and Matthews Correlation Coefficient (MCC) in classifying programs as correct or buggy compared to vanilla and chain-of-thought or natural-language specification-based approaches. In program repair, executable checkpoint violation signals drive superior Pass@1 and average patch rates, outperforming natural-language-guided repair pipelines such as SpecRover.

Related Work and Distinction

Prior work in both dynamic invariant inference (e.g., Daikon) and static specification mining has established executable properties as crucial for advanced verification, but integration with deep learning for code has so far emphasized static code structure or high-level pre/postcondition generation [nl2postcond, le-etal-2026-specmind, ruan2025specrover]. Recent LLM-based repair systems (e.g., SWE-bench, FixEval) and agents (SpecRover) utilize natural-language annotations but lack executable evidence at fine granularity.

TRACED [ding2024traced] introduced execution-aware pretraining for static/dynamic property learning, but does not address direct assertion generation or refinement. NExT [ni2024next] targets code execution reasoning but does not formalize the checkpoint specification task or its integration for repair/verification. SpecCoder thus occupies a distinct functional niche, operationalizing the generation and utilization of executable, intermediate specifications for automated reasoning and repair with LLMs for code.

Theoretical and Practical Implications

The results demonstrate that verification-guided fine-tuning using both correct and mutant executions is effective at instilling LLMs with robust stepwise reasoning abilities, enabling substantial improvements not only in specification quality, but also in downstream correctness checking and repair. This paradigm operationalizes and automates desiderata long articulated in classical verification—compositionality, stepwise reasoning, and dynamic evidence for verification—at LLM scale.

From a theoretical standpoint, the approach demonstrates the limits of scaling and instruction tuning for compositional code reasoning, highlighting the necessity of grounded, execution-based feedback. Practically, the findings suggest that executable intermediate specifications can render code generation and repair by LLMs far more transparent, verifiable, and auditable at fine granularity in real-world developer settings.

Future Directions

The work opens several directions in AI for code:

• Scalable Data Synthesis: Expansion to broader programming domains, other languages, and richer fault models.
• Interleaved Search and Verification: Hybrid agentic architectures wherein LLMs self-propose, validate, and refine checkpoints in a continual loop, potentially guided by counterexample generation.
• Formalism-Driven LLM Supervision: Integration of symbolic logic or static analysis as part of specification generation/validation feedback.
• Human-AI Collaboration: Leveraging high-quality checkpoints as understandable evidence supporting developer code review and co-verification workflows.
• Execution-Guided Pre-Training: Pre-training LLMs over instrumented code with dynamic assertions and counterfactual traces to achieve better generalization for semantic reasoning.

Conclusion

"Teaching Code LLMs to Reason with Intermediate Formal Specifications" demonstrates that verification-guided, execution-validated fine-tuning yields substantive improvements in code LLMs' ability to generate, utilize, and reason about executable checkpoint specifications. The method systematically surpasses baseline prompting and natural-language annotations, proving essential for discriminative, compositional program reasoning, correctness checking, and agentic code repair. As code LLM deployment expands, automated, stepwise, and executable specification reasoning will be foundational for next-generation, verifiable AI programming systems.