SpecGen: LLM-Driven Formal Specification

• SpecGen is a system that automates the generation of formal Java/JML specifications using LLM-driven conversational prompting combined with mutation-based refinement.
• It utilizes a two-phase architecture where initial candidate specifications are iteratively verified and repaired through systematic mutation and heuristic selection.
• The approach outperforms traditional methods by enhancing verifiability and semantic richness, and has inspired successors that integrate adaptive prompting and knowledge augmentation.

Searching arXiv for the core "SpecGen" paper and closely related follow-up work. SpecGen most commonly denotes the 2024 large-language-model system for automated formal program specification generation in Java/JML, introduced in "SpecGen: Automated Generation of Formal Program Specifications via LLMs" (Ma et al., 2024). In that sense, SpecGen is a hybrid generation-and-verification technique: it first uses conversational prompting to synthesize requires, ensures, and maintaining clauses, then applies mutation-based refinement when the initial specification fails verification. In later literature, the term also names the SpecGen task in benchmarks such as Verina, where models must synthesize a precondition and postcondition aligned with natural-language intent, and it serves as a central baseline for subsequent work on adaptive prompting, benchmark construction, knowledge augmentation, and traceable repair (Ye et al., 12 Apr 2026).

1. Origins and problem formulation

The original SpecGen paper addresses the long-standing difficulty of writing formal program specifications by hand. In Java/JML, the target artifacts are precise behavioral annotations such as preconditions (requires), postconditions (ensures), and loop invariants (maintaining), which are checked by OpenJML (Ma et al., 2024). The paper frames manual specification authoring as both time-consuming and labor-intensive, especially when a specification must correctly and comprehensively describe the semantics of complex real-world programs.

SpecGen is motivated by a contrast between traditional automated tools and LLM-based synthesis. The paper characterizes prior tools such as Houdini and Daikon as relying on predefined templates, grammar-driven generation, dynamic traces, or simple candidate instantiation, and therefore often producing trivial or overly simplistic constraints such as null checks or basic range restrictions rather than semantically rich contracts (Ma et al., 2024). Its central claim is that LLMs bring stronger code-comprehension capability, but that raw prompting alone is insufficiently reliable for verifiable formal synthesis.

Later survey work places SpecGen squarely in the requirements-engineering and specification-generation strata of LLM-based software agents. In that framing, SpecGen exemplifies a broader pattern in software engineering agents: generation is coupled to external tools, iterative feedback, and refinement rather than treated as a one-shot text production problem (Liu et al., 2024). This positioning matters because it makes SpecGen less a prompt template than a verification-centered workflow.

2. Core architecture and formal mechanisms

SpecGen has two phases: conversation-driven specification generation and mutation-based specification generation (Ma et al., 2024). In the first phase, the initial prompt contains a system role, few-shot examples, and the queried program. The experiments use 4 few-shot examples. Generation then proceeds conversationally: the LLM proposes candidate JML specifications, OpenJML verifies them, verifier error messages are fed back into the next prompt if verification fails, and the loop continues until the specification verifies or a maximum number of rounds is reached. The experimental setup uses 10 as the maximum number of conversation rounds.

The second phase is a fallback for cases in which conversational prompting still fails. The paper introduces a mutation-and-selection procedure over failed template specifications. Let $E_t$ denote template specifications generated by the LLM that failed verification, $M$ the mutation operators, $E_{mutated}$ all mutated candidates, $E_{selected}$ the current subset selected for verification, $E_{refuted}$ the rejected subset, and $E$ the final verified specifications. The paper summarizes the process as

$E_{mutated} = \mathrm{SpecMutation}(E_t, M)$

followed by

$E = \mathrm{SpecSelection}(E_{mutated}, E_t, M).$

Verification is iterative: current selected specifications are checked, refuted ones are identified, each refuted specification is replaced by a mutated variant from the same family, and the process continues until all selected specifications verify or no candidates remain.

SpecGen defines four mutation operators, aligned with JML operator classes. Predicative mutation swaps quantifiers such as $\forall \leftrightarrow \exists$. Logical mutation changes connectors such as $\land$, $M$0, $M$1, $M$2, and $M$3. Comparative mutation changes comparison operators such as $M$4, $M$5, and $M$6, and also permits boundary-shifting variants such as $M$7 and $M$8. Arithmetic mutation swaps $M$9 (Ma et al., 2024).

To avoid exhaustive verification of the mutation space, the method uses a heuristic selection rule: $E_{mutated}$0 Here $E_{mutated}$1 is the mutation family derived from a template $E_{mutated}$2, $E_{mutated}$3 counts how often mutation $E_{mutated}$4 was applied, and $E_{mutated}$5 is the mutation weight. The implementation uses negative weights to prefer fewer mutation occurrences: comparative $E_{mutated}$6, logical $E_{mutated}$7, arithmetic $E_{mutated}$8, and predicative $E_{mutated}$9. Comparative mutations are treated as most likely to help.

Two common misconceptions are corrected by this design. First, SpecGen is not merely few-shot prompting; the mutation phase is an explicit second-stage repair mechanism. Second, verification success is not obtained by unconstrained self-revision alone; it is mediated by formal checking and a structured mutation family.

3. Datasets, experimental protocol, and reported performance

The original evaluation uses two datasets (Ma et al., 2024). The first is the SV-COMP Java benchmark, starting from 265 class definitions in the Java category of SV-COMP. Programs with expected result FALSE were adapted so they can exit properly, and competition-specific calls such as Verifier.nondetInt() were replaced with equivalent Java library calls. The paper notes that 88.7% of SV-COMP programs are loop-free. The second dataset is SpecGenBench, a new expert-annotated benchmark of 120 Java programs: 20 from Nilizadeh et al. and 100 from LeetCode. Its category counts are 26 sequential, 23 branched, 24 single-path loop, 26 multi-path loop, and 21 nested loop programs.

The implementation uses gpt-3.5-turbo-1106, temperature 0.4, 4 few-shot examples, OpenJML as verifier, and a 30 minutes timeout per verification. The hardware is an 8-core Intel Core i7-12700H, 32GB RAM, Ubuntu 22.04.3 LTS. OpenJDK 1.8.0_371 is used generally, while Houdini requires OpenJDK 1.6.0_45 (Ma et al., 2024).

The headline result is that SpecGen generates verifiable specifications for 279 out of 385 programs, outperforming the conversational LLM-only baseline (218), Houdini (98), and Daikon (72) (Ma et al., 2024). On the 265 SV-COMP programs, the paper reports the following numbers of passed programs: Daikon 51, Houdini 56, 0-shot GPT 81, 2-shot GPT 83, 4-shot GPT 94, conversational GPT 146, and SpecGen 179. On SpecGenBench, SpecGen handles 24 / 26 sequential, 20 / 23 branched, 23 / 24 single-path loop, 20 / 26 multi-path loop, and 13 / 21 nested loop problems.

The paper further defines success probability for LLM-based methods as

$E_{selected}$0

with 10 attempts in the experiments. On this metric, conversational LLM-only achieves 35.95%, while SpecGen reaches 59.97% (Ma et al., 2024). The system also records 14 “unique wins” not solved by other baselines: 6 nested-loop programs, 3 multi-path loop programs, and 5 single-path loop programs.

Efficiency is measured partly through verifier calls. Random candidate selection requires 18.44 verifier calls on average overall, while the heuristic selection requires 15.51. On loop-heavy SpecGenBench categories, the averages are 36.20 for random selection and 28.51 for the heuristic (Ma et al., 2024). Semantic quality is assessed by 15 Ph.D. students on a Likert scale from 1 to 5. The reported scores are 4.83 for oracle specifications, 4.54 for SpecGen, and 2.32 for both Houdini and Daikon.

These results are often cited as evidence that verifier-guided LLM synthesis can generate specifications that are both verifiable and semantically richer than classical template- or trace-based tools. At the same time, the paper explicitly notes limitations: LLMs still struggle with complex programs, especially nested loops and quantifier-heavy invariants; OpenJML may reject some correct specifications due to incompleteness; outputs are prompt-sensitive; and opaque pretraining raises possible data-leakage concerns (Ma et al., 2024).

4. Benchmark semantics, evaluation uncertainty, and later critiques

Subsequent papers treat SpecGen as an important baseline but argue that its evaluation setting can overestimate capability. The most direct benchmark critique appears in "VeriScale: Adversarial Test-Suite Scaling for Verifiable Code Generation" (Bai et al., 21 May 2026). That paper diagnoses a core weakness in SpecGen-style evaluation in Verina: test suites are too small and too weak, especially on the negative side. It distinguishes sparse positive/expected input-output pairs from two kinds of missing negative evidence: unexpected inputs, which should be rejected by the precondition, and unexpected outputs, which differ from the reference implementation but are still accepted by a flawed generated postcondition. The claim is that sparse negative cases allow an underspecified or unsound specification to appear correct.

VeriScale addresses this with test-suite expansion and test-suite reduction. Expansion constructs expected input-output pairs, unexpected inputs, and adversarial unexpected outputs using LLM seed generation, type-aware mutation, precondition-guided input classification, and adversarial synthesis. Reduction combines boundary-preserving selection with adversary-killing reduction, in which expected cases are chosen greedily for their ability to kill adversarial implementations (Bai et al., 21 May 2026). Type-aware mutation is defined as

$E_{selected}$1

with mutated values

$E_{selected}$2

and the type-preservation guarantee

$E_{selected}$3

Instantiated on Verina, VeriScale yields VerinaPlus and VerinaLite. The paper reports that VerinaPlus expands the original benchmark by over 83×, while VerinaLite is a 14× variant. Average test volumes per task change as follows: Verina has Expected IO 5.89, Unexpected output 12.69, Unexpected input 0.65; VerinaPlus has Expected IO 370.07 (×62.83), Unexpected output 1114.01 (×87.79), Unexpected input 119.00 (×183.08); VerinaLite has Expected IO 52.34 (×8.89), Unexpected output 202.35 (×15.95), Unexpected input 15.80 (×24.31) (Bai et al., 21 May 2026). For the strongest reported model, GPT-5.5, the SpecGen score drops from 68.78% to 44.44% on VerinaPlus; CodeGen drops from 96.83% to 86.24%. The paper also emphasizes reduced ambiguity: for Claude-Opus-4.7, the baseline upper/lower bound gap is 13.23% (77.78% vs. 64.55%), while on VerinaPlus it shrinks to 4.76% (50.26% vs. 45.50%). This is used to argue that stronger negative coverage makes SpecGen evaluation more decisive.

A second critique appears in "AutoReSpec: A Framework for Generating Specification using LLMs" (Ayon et al., 4 Apr 2026). That paper explicitly characterizes SpecGen as using few-shot conversational prompting and mutation-based validation, but criticizes it as a static or fixed prompting and single-model strategy with no mechanism for adaptive prompting, dynamic model selection, or reliable recovery when the first specification is unverifiable. It reports a concrete failure case, TokenTest02, in which SpecGen produces JML syntax errors. Quantitatively, on SpecGenBench in RQ1, AutoReSpec achieves 119 passes out of 120, 69.32% success probability, and 60.33% completeness, whereas SpecGen achieves 100 passes. On a combined benchmark of 72 programs, AutoReSpec succeeds on 67 of 72, SpecGen on 58 of 72, and FormalBench on 13 of 72. The paper also reports average success probability of 58.2% for AutoReSpec versus 51.4% for SpecGen, and describes SpecGen as particularly vulnerable to Postcondition and LoopInvariantBeforeLoop errors.

Taken together, these critiques do not negate the significance of SpecGen. They instead sharpen its interpretation: the method demonstrated the viability of LLM-plus-verifier synthesis, but later work argues that both the inference procedure and the benchmark methodology needed stronger adaptivity, stronger negative cases, and better recovery signals.

5. Successors and reformulations in formal specification synthesis

Several later systems treat SpecGen as the baseline from which more adaptive or better-instrumented methods depart.

KBSpec augments LLM specification synthesis with a dual-source, self-evolving knowledge base consisting of external knowledge from official documentation and internal knowledge distilled from verifier feedback, successful generation trajectories, and successful repair trajectories (Wang et al., 19 Jun 2026). Each knowledge item is represented as

$E_{selected}$4

with raw content, LLM-generated summary, and LLM-generated keywords. Retrieval is driven by

$E_{selected}$5

and knowledge items are scored for helpfulness, harmfulness, and neutrality according to verification outcomes. On JML/OpenJML evaluation with GPT-5.2, GPT-5-mini, and DeepSeek-v3.2, KBSpec is reported to improve verification pass rates by 10-25% over state-of-the-art LLM-based approaches. Against SpecGen, the reported pass-rate gains are +15.59 for GPT-5.2, +25.61 for GPT-5-mini, and +10.59 for DeepSeek-v3.2. The paper also states that KBSpec produces the largest number of high-completeness specifications.

VeriSpecGen reformulates the problem as intent-aligned formal specification synthesis in Lean, using requirement decomposition, requirement-targeted tests, explicit traceability, and localized repair (Ye et al., 12 Apr 2026). In this setting, the SpecGen task is to synthesize

$E_{selected}$6

where $E_{selected}$7 rejects invalid inputs and $E_{selected}$8 accepts only correct outputs for valid inputs. The framework decomposes natural language into atomic requirements

$E_{selected}$9

builds a traceability map

$E_{refuted}$0

and validates candidate specifications against positive, negative-input, and negative-output tests using Lean propositions such as

$E_{refuted}$1

The failing tests are collected as

$E_{refuted}$2

and implicated requirements as

$E_{refuted}$3

On VERINA SpecGen, VeriSpecGen reports 86.6% pass@1 with Claude Opus 4.5, improving over baselines by up to 31.8 points across model families. It also produces 343,827 trajectory-derived supervised examples and reports 62–106% relative improvement when training on these trajectories.

These successors preserve the core SpecGen intuition that generation must be paired with verifier-grounded repair, but they alter the repair substrate. KBSpec introduces persistent memory and retrieval over accumulated verifier-grounded knowledge; VeriSpecGen replaces global failure signals with requirement-level attribution. A plausible implication is that later progress has focused less on raw specification generation than on the granularity and persistence of feedback.

6. Terminological scope and cross-domain reuse

Within software verification literature, “SpecGen” most often refers either to the 2024 Java/JML system or to the formal-specification-generation task used in Verina-style evaluation (Ma et al., 2024, Ye et al., 12 Apr 2026). The broader agent survey likewise describes SpecGen as a two-stage system for generating JML requirement specifications, validating them with OpenJML, integrating verification feedback into prompting, and then mutating invalid specifications to improve diversity and quality (Liu et al., 2024).

The name has also been reused in other research areas, with different technical meanings.

Usage of “SpecGen”	Domain	Source
Automated formal program specifications via LLMs	Java/JML verification	(Ma et al., 2024)
PDE-grounded intent verification and refinement for generated simulation code	Multiphysics simulation in MOOSE	(Song et al., 10 May 2026)
Neural spectral BRDF generation via spectral-spatial tri-plane aggregation	Spectral rendering	(Jin et al., 24 Aug 2025)
Speculative generation for agentic GPU kernel optimization	Systems and kernel tuning	(Guo et al., 16 Jun 2026)

In the multiphysics simulation setting, the term is used for a verification layer that reconstructs the PDE encoded by generated MOOSE input files and compares it against an intended physics contract using the Intent Fidelity Score (IFS) (Song et al., 10 May 2026). That work argues that execution success and physical correctness are separable, reporting that 39–40% of cases under execution-only repair remain runnable but solve the wrong physics. In rendering, the 2025 SpecGen paper denotes a model that generates spectral BRDFs from a single RGB image of a sphere via Spectral-Spatial Tri-plane Aggregation, reporting 35.22 PSNR / 0.9175 SSIM against 26.41 / 0.7801 for HD-Net and 27.03 / 0.8007 for MST++ (Jin et al., 24 Aug 2025). In systems research, the 2026 SpecGen paper denotes an iteration-level speculative-generation framework for agentic kernel optimization, reporting 1.68× end-to-end time reduction over CudaForge, 1.81× over AlphaEvolve, and 1.82× over KernelAgent (Guo et al., 16 Jun 2026).

This reuse has made “SpecGen” a polysemous research label rather than a single stable concept. In software-engineering contexts, however, the dominant meaning remains the formal specification generation line initiated in 2024 and subsequently extended through stronger benchmarks, adaptive repair, evolving knowledge bases, and traceable refinement.