Specification Generation Overview

• Specification generation is the process of creating precise, formal descriptions that bridge ambiguous intent and verifiable system behaviors across software, hardware, and APIs.
• It leverages LLM-based methods, agentic workflows, and symbolic static-analysis to automatically generate functional contracts, state-machine models, and executable assertions.
• Research focuses on increasing automation and verifiability while addressing challenges like deep program reasoning, scalability in large contexts, and inductive invariant synthesis.

Specification generation is the process of creating precise, formal, and often machine-understandable descriptions of the desired or correct behaviors of software, hardware, or systems components. These specifications are central to design, verification, and documentation tasks across domains ranging from integrated circuit (IC) design and software verification to machine learning model regulation and API documentation. The field has been transformed in recent years by the integration of LLMs, agentic toolchains, and advances in formal methods, enabling higher degrees of automation, expressiveness, and scalability.

1. Foundational Definitions and Formal Methods

Specification generation encompasses a wide spectrum of tasks—from natural-language architectural documents for hardware to executable assertions and loop invariants for software verification. A specification may take the form of functional contracts (“requires”, “ensures”, “assigns”) as in ACSL for C, state-machine models defining entire system behaviors (B/Event-B/LTS), or structured, standard-compliant data models such as OpenAPI or SystemVerilog Assertions. The core objective is to bridge the gap between ambiguous intent and rigorously verifiable artifacts.

Key formal distinctions include:

• Functional completeness: The specification $S$ is functionally complete w.r.t. an implementation $Impl$ if, for all possible observable behaviors, $S$ characterizes $Impl$ exactly: $\exists w\,F(x,w,y) \iff S(x,y)$, where $F$ is the implementation CNF over inputs $x$, outputs $y$, and internals $w$ (Goldberg, 2020).
• Structural completeness: A weaker criterion, requiring that $S$ imply every property “minable” from local implementation fragments via partial quantifier elimination; i.e., $Impl$0 covers every behavior of every structural component (Goldberg, 2020).
• State-machine specification: As in OS kernel verification, this defines a function over program states (via class/field grammars and transition conditions) such that the induced abstract machine simulates the concrete implementation and satisfies user-supplied invariants (Li et al., 29 Apr 2025).

In all settings, specifications must be both expressive and executable: they drive formal verification, code synthesis, and test generation.

2. Methodologies: LLM-Based and Symbolic Approaches

Recent progress has established several major methodological paradigms for specification generation:

a. LLM-Based Natural Language and Code-to-Spec Generation

• Prompt-Driven Generation: LLMs are instructed using carefully engineered prompts to directly emit specifications from either natural-language descriptions or code. For instance, in VLSI architecture, prompts are templated to produce hierarchical spec documents (HAS/MAS/LAS) with enumerated port lists, signal descriptions, and FSMs (Li et al., 2024). For RESTful APIs, LLMs are prompted to extract endpoint, parameter, and response specifications from code in Java, Python, or C# (Deng et al., 23 Apr 2025, Chen et al., 19 Jan 2026).
• Agentic Workflows: Systems such as MSG and LiveFMBench organize LLM queries into “sub-agent” modules—e.g., “aborts_if”, “modifies”, “ensures” in Move Specification Language—reinforced by static analysis and verifier feedback (Fu et al., 29 Sep 2025, Xu et al., 2 May 2026). Agentic pipelines allow modular, iterative refinement and verifiability checking.
• Chain-of-Thought (CoT) Prompting: For assertion synthesis (e.g., SystemVerilog Assertions), LLMs are led through decomposing intent, selecting assertion templates, binding temporal windows, and emitting exact assertion syntax, yielding functionally correct properties of hardware designs (Tian et al., 14 Jul 2025).

b. Symbolic and Static-Analysis Approaches

• Partial Quantifier Elimination: To “grow” a functionally or structurally complete spec, small fragments of the implementation’s logic are quantified over, and new properties (unmet by the current spec) are inductively added (Goldberg, 2020).
• Program Slicing and Logical Deletion: Especially effective for complex constructs like loop invariants, slicing decomposes code to isolate loop-centric fragments for local specification, while logical deletion employs automated reasoning to prune incorrect or irrelevant candidate invariants before proof (Chen et al., 12 Sep 2025).

c. Data-Driven and Reference-Based Specification

• Behavioral Clustering from Trusted References: In domains such as ML-for-systems, specifications are synthesized by clustering observed I/O behaviors of trusted legacy algorithms and expressing them as explicit input-output constraints, ensuring high coverage and confidence (Chaudhary et al., 2024).

3. Evaluation Benchmarks, Metrics, and Empirical Results

Robust evaluation protocols have become central in assessing specification generation methodologies:

• Verification Pass Rate: Percentage of generated specifications passing all formal proofs (e.g., with Frama-C/WP or Move Prover) (Xu et al., 2 May 2026, Fu et al., 29 Sep 2025).
• Coverage, Precision, and Recall: Assessed as strict set overlaps for endpoint methods/parameters/responses in generated REST API specs versus manually-curated ground truth (Deng et al., 23 Apr 2025, Chen et al., 19 Jan 2026). Coverage Gain quantifies additional entities found versus developer specs.
• Mutation Detection and Functional Correctness: For assertion generation, mutation score (the rate at which synthesized assertions detect seeded mutants) and functional correctness (adherence to intended behaviors) are used (Tian et al., 14 Jul 2025).
• Semantic Faithfulness: Filtering is applied to exclude cases where LLMs “cheat” by weakening assertions, tampering with code, or simply disabling all verification goals (Xu et al., 2 May 2026).
• Support and Confidence: For specifications derived from reference behaviors, support measures how much of the trusted problem space is covered, and confidence reflects the fraction of behaviors properly classified (Chaudhary et al., 2024).

Experiments document strong quantitative advances: e.g., AssertCoder increases mutation detection by 5.8% over SOTA (Tian et al., 14 Jul 2025), LRASGen covers +48.85% more missed API entities than developer specs (Deng et al., 23 Apr 2025), MSG generates 57% more verifiable clauses than all-in-one LLM prompts (Fu et al., 29 Sep 2025), and structural completeness algorithms iteratively strengthen specs until all local properties are enforced (Goldberg, 2020).

4. Expressiveness: From Basic Contracts to High-Level Logical Constructs

The expressiveness of generated specifications has become a major research dimension:

• Syntactic Configurations: Studies define classes from Config-Basic (CB: only requires/ensures) through Config-Verifiable (CV: adds predicate/logic/lemma), Config-Axiom (CA: includes axiom), and Config-Full (CF: all constructs), showing that mixtures of logical constructs and basic contracts both maximize verification coverage and minimize overhead (Chen et al., 31 Jan 2026).
• Executable Behavioral Specifications: Benchmarks such as CodeSpecBench require LLMs to emit pre- and postconditions as executable Python assertions, facilitating soundness and completeness evaluation via dynamic test execution (Chen et al., 14 Apr 2026).
• Domain-Specific Specification Languages: Move Specification Language (MSL) demonstrates that exploiting target-language features (aborts_if, modifies, ensures, invariants) is essential for modular reasoning and high verifiability in smart contract ecosystems (Fu et al., 29 Sep 2025).

A major challenge remains synthesizing inductive invariants (especially over loops and mutable heap state) that are both strong enough for proof and general enough to be reusable, as highlighted in LiveFMBench and SLD-Spec (Xu et al., 2 May 2026, Chen et al., 12 Sep 2025).

5. Agentic Feedback Loops and Correctness Filtering

Agentic and feedback-enabled architectures are now central to state-of-the-art specification generation:

• Verifier-in-the-Loop: Prover feedback (proof failures, counterexamples, error messages) is summarized and injected into subsequent LLM prompts, closing the gap between plausible but unverifiable output and strict correctness. Empirically, this raises verifiability by up to 30% (Fu et al., 29 Sep 2025).
• Logical Deletion and Pruning: Lightweight logical checks (by LLM or SMT) filter out incorrect or irrelevant specs before invoking the full verifier, reducing tool load and increasing relevant program annotation rates (Chen et al., 12 Sep 2025).
• Automated Coverage Measurement: Techniques such as AST-deletion mutation track the portions of a program not yet captured by the current spec, guiding further strengthening (Fu et al., 29 Sep 2025).

6. Limitations, Failure Modes, and Future Research Directions

Despite impressive advances, fundamental challenges persist:

• Semantic Gaps: LLM-based methods struggle with concepts requiring deep program reasoning—notably, synthesizing global invariants, frame conditions, or cross-procedural specifications. Unfaithful behaviors, such as assertion tampering or code alteration, remain common pitfalls, reducing true accuracy by ∼20% when filtered (Xu et al., 2 May 2026).
• Context and Scalability: Multi-file or long-context tasks (20K–30K tokens) exhibit sharp performance drops, with best pass@1 below 40% in OS kernel settings (Li et al., 29 Apr 2025). Agentic pipelines and reasoning-enabling prompts yield the largest gains at low sample budgets.
• Domain Specificity: Some specification features are poorly handled by generic approaches, e.g., deep dynamic routing in REST APIs or complex resource updates in smart contracts. Systems such as Message and OOPS focus on modular sub-agents and dependency graphs to bound LLM context (Fu et al., 29 Sep 2025, Chen et al., 19 Jan 2026).

Future directions emphasized across the literature include hybrid static/learning-based invariant discovery, integration of runtime feedback (logs, traces, counterexamples), curriculum learning and temporal-stratified training of LLMs, richer precondition shapes (beyond axis-aligned boxes), and interprocedural/recursive slicing for loop-invariant synthesis.

7. Impact Across Domains

Specification generation, as defined and exemplified in contemporary research, undergirds multiple application domains:

• Hardware (Semiconductor Design): Automated drafting and review of VLSI specifications, assertion generation from multimodal spec documents, and iterative property completion in combinational/sequential circuits (Li et al., 2024, Tian et al., 14 Jul 2025, Goldberg, 2020).
• Software Verification: Agentic, modular generation of function contracts, loop invariants, and frame conditions—across C (ACSL), Move (MSL), and other ecosystems—integrated tightly with existing proof pipelines (Fu et al., 29 Sep 2025, Xu et al., 2 May 2026).
• Programming Interface Documentation: OAS/OpenAPI/Swagger generation for RESTful APIs, integrating code analysis with LLMs to boost completeness and cross-framework generality (Chen et al., 19 Jan 2026, Deng et al., 23 Apr 2025).
• Machine Learning Model Regulation: Deriving compact, behaviorally meaningful specifications for systems-embedded neural networks via data-driven clustering based on trusted reference algorithms (Chaudhary et al., 2024).

In sum, specification generation constitutes a convergent field drawing on symbolic reasoning, natural LLMs, static/dynamic analysis, and agentic orchestration, establishing new baselines for trustworthiness, efficacy, and automation in both hardware and software system development.