LLM-Based Specification Generation Framework

• The paper introduces a structured methodology that integrates program slicing, advanced LLM prompting, and formal verification to generate semantically valid specifications.
• The framework employs a multi-stage pipeline that isolates loop structures, applies chain-of-thought reasoning for logical deletion, and refines invariants via SMT-based verification.
• Empirical evaluations show improved specification correctness and reduced runtime, demonstrating its effectiveness for complex, loop-rich code analysis.

A LLM-Based Specification Generation Framework is a structured methodology that utilizes LLMs, often in combination with symbolic, static, or verification-driven methods, for the automated synthesis of formal specifications from code, design artifacts, or natural language descriptions. These frameworks aim to automate, accelerate, and enhance the quality of specification construction, which is pivotal for software and hardware verification, synthesis, and maintenance. They are architected as multi-stage pipelines that integrate code and specification analysis, advanced LLM prompting, heuristic or formal refinement, and verification-in-the-loop to yield specifications that are not only syntactically correct but also semantically verifiable and relevant to complex program structures such as deep loop nests.

1. Core Architecture: Pipeline, Components, and Interaction

LLM-based specification generation frameworks, exemplified by SLD-Spec, consist of sequential, modular phases designed to maximize specification correctness and verifiability, particularly in the context of complex loop-rich code (Chen et al., 12 Sep 2025). The canonical pipeline in these frameworks comprises:

• Program Slicing: Static decomposition of the input source (e.g., C code) through dependency-guided backwards slicing. Each slice contains at most a single loop structure, minimizing inter-loop interference and context explosion during LLM prompt construction. Slicing criteria are defined as point-variable pairs, and redundant or overlapping slices are eliminated via a greedy set cover algorithm.
• Specification Generation (“Guessing”): Each slice is independently presented to an LLM—such as GPT-3.5-turbo—along with tightly scoped prompts and structure markers (e.g., INFILL/END) to elicit candidate formal specifications in the target annotation language (e.g., ACSL). The LLM is tasked with emitting loop invariants, assigns clauses, variants, and function contracts at the slice level.
• Logical Deletion: A critical refinement stage in which candidate specifications are subjected to LLM-based chain-of-thought reasoning, not just automated verification. This process involves four steps: (i) exclusion of irrelevant specs by variable presence, (ii) extraction of informal requirements from the spec, (iii) reasoning over requirement-code fidelity, and (iv) Boolean acceptance or rejection of each candidate. Only specifications that are deemed both relevant and semantically plausible by the LLM survive for aggregation.
• Formal Verification: Remaining slice-level specifications are composed into a function-level contract and passed to a deductive verifier, such as Frama-C/WP+SMT. Verification failures trigger iterative deletion of non-verified specs until full verification is reached or no more candidate invariants remain.

2. Formal Specification Targets and Languages

The generation framework outputs formal specifications in annotation-rich languages appropriate to the domain. For instance, SLD-Spec targets the ACSL annotation language for C programs (Chen et al., 12 Sep 2025). ACSL supports:

• Function contracts: /*@ requires P; ensures Q; assigns [X](https://www.emergentmind.com/topics/stability-index-x); */
• Loop annotations: Placed as ghost annotations inside loop bodies, commonly:
  • loop invariant I;
  • loop assigns A;
  • loop variant V;

An example fragment generated for a loop body:

/*@ loop invariant 0 ≤ a ≤ x; loop variant x - a; loop assigns *x_res; */ while(a < x) { ... }

This formalization aligns with toolchains for deductive program verification, enabling subsequent semantic checking and proof discharge.

3. Slicing and Logical Deletion: Algorithms and Rationale

Program slicing is realized through an automated backward data dependency analysis. Given a function $F$, the set of slicing criteria $S_{sc}$ is constructed from all local variables not internal to loops, considered at the function’s end. Each criterion leads to a backward slice; a greedy cover ensures the selected slices partition the function so that each contains at most one loop (Chen et al., 12 Sep 2025).

Pseudocode for automatic slicing:

def AutoSlicing(F): if F has no caller: GenStubCaller(F) S_sc = set() for v in Var(F) \ LoopVars: S_sc.add((end_of_F, v)) S_fs = set() for sc in S_sc: fs = DG.Slicing(F, sc) if fs: S_fs.add(fs) return SimplifySlicing(S_fs)

Logical deletion addresses the inadequacy of rely-only-on-verification approaches for pruning spurious or ambiguous LLM-generated specifications. The process applies four step LLM-based filtering: variable-based exclusion, requirement translation, semantic reasoning, and decision. This phase is necessary because verification tools like Frama-C may not identify all cases of semantic irrelevance or misalignment in suggested specs.

4. Empirical Evaluation: Metrics, Benchmarks, and Results

Frameworks such as SLD-Spec are rigorously evaluated across both simple and complex program datasets. Key metrics include:

• PCRSAV: Number of correct and relevant specifications after final verification.
• NAV: Number of driving code assertions that pass under generated specs.
• NPP: Number of programs fully verified (i.e., all assertions pass).
• RT: Average end-to-end runtime for verified programs.

Results on the Frama-C-Problems dataset show that SLD-Spec verified 32/51 C programs, five more than the previous best, while reducing runtime by nearly 24% (Chen et al., 12 Sep 2025). On custom datasets featuring parallel, multi-path, and nested loops, SLD-Spec exhibited near-complete PCRSAV and high NAV/NPP, in contrast to zero successful end-to-end runs for AutoSpec. Ablation studies confirm that both slicing and logical deletion contribute critically: skipping logical deletion reduced specification correctness and relevance, while omitting slicing led to missed invariants and overall incomplete specs.

Summary Table: Ablation Study Results

Configuration	PCRSAV (%)	NAV (/49)	NPP (/11 programs)
w/o PS+LD	~90	21.4	2
w/o LD	~99	27.8	8
SLD-Spec (full)	~100	46.6	10

5. Error Taxonomy in LLM-Generated Specifications

A comprehensive error taxonomy for LLM-generated loop specifications was established:

• Incorrect boundaries/invariants: e.g., off-by-one on loop variable range.
• Pattern summarization mistakes: e.g., failing to model iteration-dependent values.
• Specification misalignment: applying the wrong invariant to a given loop context.
• Incorrect assigns clause: marking variables as modified when they are not.
• Multiple loop variants: introducing syntactic errors in annotation.
• Non-monotonic loop variants: violating the requirements for progress proofs.

Logical deletion via LLM chain-of-thought reasoning addresses such classes, particularly those unlikely to be caught directly by SMT-based verification.

6. Limitations, Future Work, and Outlook

Principal limitations include residual non-determinism in LLM behavior, occasional over-pruning where correct specifications may be deleted, and potential inadequacy in handling pointer-intensive, higher-order, or concurrent code constructs. Future enhancements are projected in ensemble LLM reasoning, richer dependency analyses such as semantic slicing, tighter integration with proof assistants (Coq, Why3), and improved feedback loops combining LLM synthesis with proof search (Chen et al., 12 Sep 2025).

7. Significance and Impact

LLM-based specification generation frameworks such as SLD-Spec represent a substantial advance in the automation of formal program verification. By orchestrating static program analysis, advanced LLM reasoning, and iterative verification-driven refinement, these frameworks bridge the gap between code, formal contract, and tool-driven proof. They enable robust handling of complex program structures, particularly those involving multiple, interacting loops, that have previously stymied both template-based and vanilla LLM approaches. Above all, these frameworks delineate a path towards automated formal methods pipelines that couple code, specification, LLM, and verifier in a synergistic loop (Chen et al., 12 Sep 2025).