SpecCoder: Verification-Guided CodeLLM Training

• SpecCoder is a verification-guided CodeLLM training framework that produces executable checkpoint assertions for validating intermediate program states.
• It leverages validated reference programs, behavior-changing mutants, and multi-turn specification refinement to differentiate correct from faulty executions.
• Empirical evaluations on Python competitive programming problems demonstrate substantial improvements in correctness, completeness, and assertion validity.

SpecCoder is a verification-guided CodeLLM training framework for executable checkpoint specification generation, a setting in which assertions are inserted at meaningful internal program points to describe expected intermediate states. It is designed around the observation that executable formal specifications provide machine-checkable constraints for verifying, debugging, and repairing code, whereas existing LLM-based methods mainly infer whole-program preconditions and postconditions and therefore miss the intermediate semantic commitments programmers rely on when reasoning about an algorithm. SpecCoder learns from validated reference programs, behavior-changing mutants, and multi-turn specification-refinement traces, and it selects specifications that hold on correct executions while rejecting faulty executions. The framework is introduced together with HumanExec, a benchmark built from recent Codeforces competitive programming problems with test suites, reference solutions, and human buggy submissions (Le-Anh et al., 5 Jul 2026).

1. Motivation and problem setting

The motivating distinction in SpecCoder is between whole-program pre/postconditions and intermediate checkpoint assertions. Whole-program preconditions summarize what must hold at entry, and postconditions summarize what must hold at exit. By contrast, checkpoint assertions are Boolean predicates placed at internal program points; they can be executed on concrete traces and can reject faults that slip past end-to-end tests. The paper argues that this internal form of specification better matches step-by-step algorithmic reasoning, such as checking that an array is sorted before merging or that an invariant holds at each loop iteration (Le-Anh et al., 5 Jul 2026).

A central empirical premise is that prompting current CodeLLMs often produces executable assertions that are syntactically invalid, trivial, or too weak to reject behavior-changing faults. SpecCoder is therefore framed not simply as a prompting strategy but as a training framework that turns specifications from passive annotations into executable evidence. In this formulation, the target is not merely descriptive commentary inside code; it is an assertion set whose execution behavior distinguishes correct from faulty internal states.

The interval-merge example in the paper illustrates the intended semantics. A natural-language checkpoint such as “intervals are sorted by start point” is converted into an executable assertion:

$y_i$5

This assertion rejects a bug in which intervals are incorrectly sorted by end point instead of start point. The example is important because it shows the paper’s notion of a checkpoint: it is local, executable, and fault-discriminative, rather than a global summary of input/output behavior.

2. Verification-guided training pipeline

SpecCoder is built on top of an existing CodeLLM backbone, specifically Qwen2.5-Coder instruction-tuned models at 7B, 14B, and 32B scales, with no architectural modifications. The framework relies on supervised fine-tuning over newly constructed instruction-to-annotated-program examples rather than on a new decoding architecture or an auxiliary verifier integrated into the model itself (Le-Anh et al., 5 Jul 2026).

Its data construction process has three principal components. First, it uses reference programs $y_i$, defined as human-written solutions that pass all available tests $T_i$. Second, it constructs behavior-changing mutants $\mathcal{M}_i$, defined as one-line syntactic perturbations of $y_i$ that pass at least one test and fail at least one test. Third, it creates specification-refinement traces, which are multi-turn “Explore/Submit” logs in which a teacher model proposes assertions and a verifier executes them on both $y_i$ and $\mathcal{M}_i$, returning feedback including reachability, correctness, and mutant-rejection counts.

The pipeline described in the paper proceeds as follows. Problems are first filtered to retain only those with sufficiently large test suites, specifically $\lvert T_i \rvert > 30$. Mutants are generated for each reference program, and only non-trivial ones are retained. Then, for up to $K$ turns, the teacher model proposes checkpoint assertions and the verifier reports syntax or unreachable errors, per-assertion correctness $\mathrm{Corr}_{ij}$, and per-assertion completeness $\mathrm{Comp}_{ij}$. Finally, only annotated programs are accepted in which every assertion satisfies

$T_i$0

with default thresholds $T_i$1 and $T_i$2.

This construction places the quality control in the training data rather than in a separate objective term. A plausible implication is that SpecCoder’s behavior depends heavily on the selectivity of this filtering stage, since the model is trained only on assertions that have already survived execution-based validation against both correct and faulty runs.

3. Formal criteria for checkpoint selection

The paper formalizes checkpoint evaluation through reachability, correctness, and completeness. Reachability asks whether a test $T_i$3 executes the source location of a checkpoint $T_i$4. Correctness measures whether the checkpoint holds on correct executions, and completeness measures whether it rejects eligible faulty executions induced by mutants (Le-Anh et al., 5 Jul 2026).

Correctness is defined as

$T_i$5

Completeness is defined with respect to the eligible mutant set $T_i$6, where the eligible mutants are those that modify code before the checkpoint and actually reach that location:

$T_i$7

The acceptance rule keeps only those annotated programs in which every checkpoint satisfies $T_i$8 and, when $T_i$9, also satisfies $\mathcal{M}_i$0, with at least one non-vacuous mutant set. The paper’s pseudocode sketch further rejects assertions that are unreachable on the reference program.

SpecCoder is trained on two supervised tasks. In specification generation, the input is $\mathcal{M}_i$1, where $\mathcal{M}_i$2 is either the reference program $\mathcal{M}_i$3 or a mutant $\mathcal{M}_i$4, and the target is the annotated program $\mathcal{M}_i$5. In code + spec generation, the input is the problem statement $\mathcal{M}_i$6 alone, and the target is the annotated solution $\mathcal{M}_i$7. Optimization uses the standard autoregressive language-modeling loss

$\mathcal{M}_i$8

where $\mathcal{M}_i$9 is the instruction and $y_i$0 is the annotated solution. The paper explicitly states that SpecCoder does not optimize separate validity or discriminability losses.

4. HumanExec benchmark

HumanExec is introduced to evaluate generalization beyond LeetCode-style data. It is an execution-based benchmark sourced from Codeforces and contains 150 programming problems. Each problem includes a natural-language problem statement with I/O specification, a reference solution that passes its test suite, a set of human-written buggy submissions that each pass at least one test and fail at least one test, and an executable test suite ranging from 48 to 279 tests per problem, with mean $y_i$1 (Le-Anh et al., 5 Jul 2026).

The benchmark supports three evaluation tasks:

Task	Input	Evaluation
Specification generation	Problem statement + reference solution	Syntax validity, correctness, completeness
Program correctness checking	Statement + candidate solution	MCC, balanced accuracy, false positive/negative rates
Program repair	Statement + buggy solution	Pass@$y_i$2, average repair rate

For specification generation, completeness is computed against held-out mutants generated only for evaluation. For correctness checking, the paper reports Matthews correlation coefficient together with balanced accuracy and error rates. For repair, the evaluation uses Pass@$y_i$3 and average repair rate (APR).

HumanExec is significant because it aligns the training and evaluation story around executable evidence. The benchmark does not ask only whether a model can generate plausible annotations; it asks whether those annotations are executable, semantically correct on correct runs, and useful for downstream reasoning about candidate solutions and repairs.

5. Empirical results

The paper reports three specification-quality metrics: executable validity, correctness, and completeness. Executable validity is the fraction of generated assertions that compile and refer to in-scope variables. Correctness measures whether assertions hold on correct executions, and completeness measures whether they reject faulty behaviors (Le-Anh et al., 5 Jul 2026).

For inline checkpoint specifications on Qwen2.5-Coder-7B, the paper reports the following changes under SpecCoder:

Metric	Base	+SpecCoder
Correctness	0.279	0.435
Completeness	0.171	0.782
Validity	0.827	0.903

These correspond to +55.8\% correctness, +358.1\% completeness, and +9.1\% validity. The abstract additionally states that, across Qwen2.5-Coder models, SpecCoder improves inline-specification correctness by up to 55.8\%, completeness by up to 358.1\%, and executable assertion validity by up to 26.6\%.

The paper also reports overall average gains across the model family: correctness rises from 0.478 to 0.631, completeness from 0.211 to 0.746, and validity from 0.839 to 0.967. Similar large gains are said to hold across the 14B and 32B backbones.

The downstream evaluations are consistent with the specification results. In program correctness checking, using SpecCoder’s executable assertions as evidence in an AssertChecker classifier improves MCC by 25–36\% over the base model and outperforms chain-of-thought and natural-language (HoarePrompt) baselines. In program repair, the AssertRepair agent, which uses violated checkpoint signals, achieves the highest Pass@1 across all three model sizes and improves APR relative to vanilla and natural-language specification methods. The paper interprets these results as evidence that executable checkpoints provide fine-grained evidence for reliable verification.

6. Qualitative behavior, limitations, and open questions

The qualitative examples emphasize how checkpoint assertions expose semantic deviation before final outputs are computed. In the interval-merge example, the assertion over interval start points fails immediately when a buggy variant sorts by end point instead of by start point. In a second example involving code computing $y_i$4, SpecCoder inserts assertions such as the following (Le-Anh et al., 5 Jul 2026):

$y_i$6

According to the paper, when a buggy variant fails to update s or computes t incorrectly, one of these checkpoints fails and provides immediate evidence of the semantic deviation. This failure signal guided the repair agent to localize and fix the bug.

The paper also delineates several limitations. SpecCoder relies on existing test suites with sufficient coverage, so unchecked paths yield unreachable assertions. Its mutant-guided filtering depends on the quality and diversity of generated mutants. The reported experiments are focused on Python competitive-programming code, and generalization to large codebases, interactive applications, or other languages remains an open empirical question.

The proposed extensions remain explicitly tentative. The paper lists integrating symbolic or static-analysis reachability checks, extending the method to infer and refine loop invariants or data-structure shape properties, and combining dynamic checkpoint learning with language-model-driven or reinforcement-learning fine-tuning for end-to-end program synthesis with built-in verification. It also identifies open research questions on checkpoint granularity and placement, reusable libraries of verified algorithmic patterns, joint use of natural-language documentation and executable checkpoints, and interaction with larger models or multi-modal reasoning over code and execution traces.

Taken together, these points position SpecCoder as a framework for learning intermediate executable specifications rather than only boundary contracts. Its central claim is that code reasoning can be strengthened by training CodeLLMs to produce internal assertions that are both valid on correct executions and discriminative against behavior-changing mutants, thereby linking specification generation, correctness checking, and repair within a single execution-grounded formulation.