SpecVerify: Automated C Verification

• SpecVerify is an end-to-end framework that converts free-form natural language requirements into concrete C assertions using LLM-based formalization and ESBMC verification.
• It automatically maps requirement semantics to C variables, supporting properties beyond classical LTL and handling IEEE-754 floating-point behavior.
• Evaluated on aerospace cyber-physical systems, SpecVerify achieved a 46.5% verification rate while reducing false positives compared to traditional workflows.

SpecVerify is an end-to-end framework that uses a LLM, specifically Claude 3.5 Sonnet, together with the ESBMC bounded model checker to automatically verify C implementations of cyber-physical systems against natural language requirements. It was introduced to address the difficulty of deriving verifiable formal properties from free-form requirements, and it conceptually replaces NASA’s FRET–CoCoSim pipeline with LLM-based requirement formalization, LLM-based property generation, and ESBMC-based verification on the generated C code. Evaluated on nine Lockheed Martin cyber-physical-system tasks, it achieved a 46.5% verification rate, comparable to CoCoSim, while reporting lower false positives and supporting requirements that go beyond the expressive power of classical LTL (Wang et al., 7 Jul 2025).

1. Definition and aims

SpecVerify is defined around three stated goals: automating the transformation from free-form requirements to verifiable properties, removing manual variable mapping and FRETish/LTL encoding, and supporting requirements that go beyond classical LTL expressiveness by allowing arbitrary C assertions and auxiliary state (Wang et al., 7 Jul 2025). Its immediate target is C code generated from Simulink for aerospace cyber-physical systems, with ESBMC checking the resulting instrumented program under precise C and IEEE-754 semantics.

The framework’s significance lies in the point at which it intervenes. Classical requirements-verification workflows often require an intermediate formal language such as FRETish and then a temporal-logic or Lustre encoding before model checking. SpecVerify instead tries to move directly from natural-language requirements to code-level verification artifacts. This makes the central technical problem not merely model checking, but semantic interpretation: preconditions, signals, internal states, timing relations, and code-variable bindings must all be inferred from the requirement text and the generated C program (Wang et al., 7 Jul 2025).

A second defining feature is that SpecVerify operates at implementation level rather than through a manually reconstructed high-level model. This is directly relevant for generated C that includes floating-point behavior and code-specific variable encodings. The paper treats this as a practical route to requirements verification when manual variable mapping and LTL formalization are bottlenecks (Wang et al., 7 Jul 2025).

2. Workflow and internal representation

The workflow comprises requirement formalization, assertion generation, and ESBMC-based verification. In the first stage, the input is the original requirement text, optionally accompanied by subsystem hints or a high-level description. The LLM interprets requirement semantics, identifies preconditions, variables and signals, and expected behavior, and then maps textual terms to concrete C variables in the Simulink-generated code (Wang et al., 7 Jul 2025).

The intermediate product is a semi-formal, human-readable specification, often presented in a Hoare triple-like form,

$$\{ P \}\ \text{(code fragment / function)}\ \{ Q \},$$

where $P$ is a precondition over concrete C variables and $Q$ is a postcondition or invariant, with auxiliary definitions when needed. The paper’s example contrasts a Lustre-style description using FC, mid_value, and set_val with an LLM-generated description that maps the “no-fail state” to rtDW.Delay1_DSTATE[2] == 0 and the “mid-value” computation to an internal variable sel_val. The stated advantage is proximity to the actual implementation and easier inspection by developers familiar with the code (Wang et al., 7 Jul 2025).

In the second stage, the LLM translates the semi-formal specification into embedded assert(...) statements and auxiliary C code. This may include extra state such as prev_x, prev_mode, counters for event occurrences, or more complex numeric logic such as median calculations or threshold-based control (Wang et al., 7 Jul 2025). Assertion placement is also synthesized: the framework relies on the model’s understanding of control-step boundaries, loop locations, and computation blocks to decide where checks should be inserted.

The third stage is ESBMC verification. ESBMC performs SMT-based bounded model checking by unwinding loops and recursion up to a bound $k$, translating the bounded program into constraints over integers, bit-vectors, arrays, floating-point numbers under IEEE-754 semantics, and Booleans, and checking whether there exists an execution up to depth $k$ on which an inserted assertion fails. If such a path exists, ESBMC returns a counterexample trace; otherwise the property is reported as verified up to bound $k$ (Wang et al., 7 Jul 2025).

3. Expressiveness beyond classical LTL

A central claim of SpecVerify is that it is not restricted to pure LTL formulas. Instead, it can express any property encodable as C assertions over finite traces with auxiliary state, which the paper organizes into safety properties, bounded liveness or response properties, historical or past-time properties, numerical invariants, and bounded timing constraints (Wang et al., 7 Jul 2025).

Safety properties correspond to assertions that never fail up to the ESBMC bound. Typical examples are variable bounds such as

$\forall t\; (\text{altitude}(t) \ge 0)$

or mode consistency constraints such as $$G(\text{mode} = \text{FAILSAFE} \Rightarrow \text{output} = 0)$$. In SpecVerify these appear simply as code-level assertions guarded by the relevant preconditions.

Bounded liveness and response properties are expressed by synthesizing finite-state instrumentation rather than by relying on LTL operators alone. The paper gives the temporal form

$G\big(\phi \Rightarrow F_{\leq N}\psi\big),$

but the implementation mechanism is imperative: counters and control variables track whether the required response occurs within the specified number of steps (Wang et al., 7 Jul 2025). Historical properties are handled similarly, for example by storing previous modes or counting consecutive threshold violations.

The framework’s “beyond LTL” claim is especially concrete for properties that mention previous values, consecutive-step streaks, or counts of events. These are described as past-time properties that classical future-time LTL and FRET often struggle to express, but that are straightforward as auxiliary C state plus assertions (Wang et al., 7 Jul 2025). The paper also emphasizes arithmetic-intensive requirements, including median selection, saturation behavior, and floating-point corner cases.

The floating-point median example illustrates this in detail. For

$$a = 1.813356\times 10^{24},\quad b = 2.328307\times 10^{-10},\quad c = 1.999512,$$

the intended median in real arithmetic is $P$0, since $P$1. But with IEEE-754 floating-point arithmetic,

$P$2

so the code’s mean-distance heuristic can select $P$3 rather than $P$4. SpecVerify encodes the intended property as a C assertion equivalent to $P$5, and ESBMC produces a counterexample trace showing the implementation-level bug (Wang et al., 7 Jul 2025). The paper uses this to argue that assertion synthesis over concrete C semantics can expose falsifiable cases missed by higher-level reasoning with rational approximations.

A common misunderstanding would be to read “beyond LTL” as unbounded temporal completeness. The paper does not make that claim. The resulting properties are still checked over bounded executions, and many unresolved cases are attributed to depth bounds and state-space explosion in bounded model checking (Wang et al., 7 Jul 2025).

4. Benchmark, automation, and logical equivalence

The evaluation uses the Lockheed Martin CPS benchmark. The benchmark originally contains ten Simulink models, of which nine were usable because one autopilot task could not be converted to C due to missing Simulink libraries. The usable tasks span signal processing and monitoring, finite-state control, navigation and guidance, and system integration and safety (Wang et al., 7 Jul 2025). Simulink models were translated to C with MATLAB Embedded Coder, giving components of 41–251 lines of code.

The comparison baseline reuses the LMCPS results for CoCoSim and SLDV, where 12 requirements were provably satisfied, 17 were falsifiable by counterexample, and 29, or 39%, remained undetermined because of complexity or tool limits (Wang et al., 7 Jul 2025). Against this backdrop, SpecVerify is evaluated not only on raw verification outcomes but also on workflow automation and logical correspondence to the manually formalized properties.

On automation, the paper classifies CoCoSim and SLDV as semi-automated, requiring manual mapping and operating under fixed language constraints such as FRETish templates and LTL fragments. By contrast, SpecVerify with Claude 3.5 Sonnet and ESBMC is described as fully automated for the studied tasks, with no manual mapping required and flexible natural-language input rather than FRETish (Wang et al., 7 Jul 2025). ChatGPT is said to offer “full” automation in principle, but in practice its properties frequently contain syntactic or semantic errors.

The logical-equivalence study compares LLM-generated and CoCoSim-generated requirement interpretations as Hoare triples. Of 58 cases, 46 were judged logically equivalent, corresponding to 79.31%. Two cases, or 3.45%, reflected LLM misunderstanding; two more, 3.45%, reflected missing assumptions; four cases, or 6.90%, were benchmark cases skipped because CoCoSim could not express them; two cases, 3.45%, involved sequence reversal; and there was one case each of over-verification by the LLM and over-verification by CoCoSim (Wang et al., 7 Jul 2025).

Two examples illustrate the limits of the approach. In a triple-redundancy-voting requirement, the requirement text stated only that “Errors will appear as differences in the values of the input set signals. This difference is called a miscompare.” The LLM did not infer the full majority-voting semantics implemented by the system, because the requirement document itself was oversimplified. In a SWIM example, the CoCoSim team had manually introduced an unstated assumption that aircraft weight is strictly greater than zero; the LLM did not invent this assumption, so its property was logically weaker (Wang et al., 7 Jul 2025). The comparison therefore exposes both the strengths of direct code-grounded mapping and the dependence on requirements quality.

5. Verification performance and comparative findings

The main quantitative comparison covers CoCoSim, SLDV, SpecVerify with Claude and ESBMC, and ChatGPT with ESBMC (Wang et al., 7 Jul 2025).

Toolchain	Verified / Formed / Total	Key metrics
CoCoSim	27 / 54 / 58	46.5%; FP 2; FN 6
SLDV	21 / 54 / 58	36.2%; FP 0; FN 0
SpecVerify (Claude + ESBMC)	27 / 58 / 58	46.5%; FP 0; FN 2
ChatGPT + ESBMC	15 / 35 / 58	25.9%; FP 8; FN 2; assertion errors 23

These results show that SpecVerify matched CoCoSim’s 46.5% verification rate while forming properties for all 58 requirements and reducing both false positives and false negatives (Wang et al., 7 Jul 2025). SLDV reported no false positives or false negatives but left more requirements unresolved. ChatGPT with ESBMC was markedly less reliable, with only 35 usable properties out of 58 and 23 assertion errors.

The paper also reports two additional real errors found by Claude+ESBMC that CoCoSim and SLDV missed. The first class comes from floating-point semantics: CoCoSim used rational approximations for functions such as sin and cos, which can mask IEEE-754-specific behaviors. The second class comes from model connection issues in manually constructed verification models (Wang et al., 7 Jul 2025).

The REG-003 example is the most explicit model-connection case. In the CoCoSim study, the branch intended to compute counter + 1 was misconnected to constant 0: $P$6 As reported, the “increment by 1” branch was erroneously wired to zero, so the intended behavior was never exercised. Because SpecVerify reasons directly over the generated C implementation rather than a manually reconstructed model, it is not vulnerable to that specific modeling error (Wang et al., 7 Jul 2025).

The paper treats this as evidence that code-level verification can uncover implementation-level numeric bugs and eliminate some categories of human modeling mistakes. A plausible implication is that the framework’s main comparative advantage is less in absolute verification rate than in where semantic precision is located: directly in the generated C code, with IEEE-754 reasoning, rather than in a higher-level temporal or Lustre abstraction.

6. Limitations, human oversight, and broader context

The paper presents SpecVerify as fully automatable, but it does not treat human review as optional in practice. Its comparative model study found that Claude 3.5 Sonnet consistently produced syntactically correct C assertions and achieved 79.31% logical equivalence to manual properties, whereas ChatGPT 4o1 produced many uncompilable or semantically incorrect assertions and Llama 3.1-8B failed completely and was excluded from the final evaluation (Wang et al., 7 Jul 2025). The stated conclusion is that high-quality requirements documentation and human monitoring remain critical.

Several limitations are explicit. The evaluation uses only nine models, all from aerospace CPS, and all are generated from Simulink rather than hand-written legacy code. The C components are relatively small, at 41–251 lines of code. The comparison with Llama 3.1-8B is acknowledged as unfair relative to models above 100 billion parameters, and future work is said to require larger open models such as Llama 70B or DeepSeek-R1 for a fairer baseline (Wang et al., 7 Jul 2025). The paper also notes that its Hoare-triple comparison method can hide subtle quantificational or control-flow differences, and that many unknown verification outcomes are due to bounded model checking limits rather than specification-generation failures.

Within the wider literature on specification-centered verification, SpecVerify occupies a distinctive position. Other systems pursue related goals through different verification loci. SpecMon uses a formal symbolic protocol model and black-box runtime monitoring to check whether implementation traces are valid traces of a specification model (Morio et al., 2024). DeepAssert extracts module-level functional specifications from RTL hierarchy and signal propagation, then uses them to generate fine-grained deep assertions for hardware modules (Wang et al., 18 Sep 2025). VeriAct argues that verifier pass rates alone do not establish specification correctness or completeness, and introduces Spec-Harness to evaluate JML specifications through symbolic verification of correctness and completeness metrics (Misu et al., 31 Mar 2026). VerifyThisBench, in turn, frames end-to-end specification, implementation, and proof synthesis as a benchmark problem and reports that state-of-the-art models remain below 4% zero-shot pass rate on full verification tasks (Deng et al., 25 May 2025).

This broader landscape suggests that SpecVerify’s contribution is specifically the direct connection between natural-language requirements, concrete C variables, and SMT-based bounded model checking under implementation-level floating-point semantics. Its reported gains come with two corresponding constraints: verification remains bounded, and semantic quality remains coupled to the clarity and completeness of the requirement documents (Wang et al., 7 Jul 2025).