An Empirical Study of LLM-Generated Specifications for VeriFast

Abstract: Static verification tools can assure industrial scale software, but require significant human labor to write specifications. This is particularly true of static verifiers based on separation logic (SL verifiers), which excel at verifying heapmanipulating programs, but require many complex auxiliary specifications to reason about heap structure. Recent work applies LLMs to generate code, tests, and proofs, including specifications for verifiers, but mostly targeting non-SL verifiers. To address this gap, this paper thoroughly evaluates how well LLMs perform when prompted to generate specifications for verifying 303 C functions with the SL verifier VeriFast. We explored eight prompting approaches, ten LLMs, and three input types in two stages. Quantitative and qualitative analyses are used to assess the LLM-generated code and specifications for functional behavior, verifiability and errors. The results show that LLMs preserve functional behavior in source code and specifications (both over 91%), but achieve modest verification success (31.4%). Using Gemini 2.5 Pro and providing formal contracts lead to higher success rates in our setting. Moreover, most errors (94%) come from LLMs' mistakes in the domainspecific knowledge of SL verifiers such as VeriFast. These findings provide guidance for optimizing LLM-generated specifications for SL verifiers.

• The paper presents an empirical evaluation of LLM-generated specifications for VeriFast, showing high behavioral fidelity yet modest overall verification success.
• The study analyzes a dataset of 303 C functions using various prompt strategies and LLMs, identifying key error modes and the impact of input specification richness.
• The research underscores challenges in automating separation logic verification and recommends hybrid, repair-augmented approaches to address domain-specific gaps.

Empirical Evaluation of LLM-Based Specification Synthesis for VeriFast

Introduction and Motivation

Static verification tools for heap-manipulating programs, particularly those based on separation logic (SL) such as VeriFast, offer a scalable methodology for enforcing functional and safety properties on low-level code. However, these tools require intricate formal specifications—pre/postconditions, loop invariants, inductive lemmas, and explicit fold/unfold statements—to precisely capture heap manipulation and ownership, imposing a significant burden on verification engineers. Recent advances in LLMs have demonstrated promise in automatic code, test, and proof synthesis, but a gap remains in systematically evaluating LLM-based specification generation for SL verifiers, where specification complexity is markedly higher and domain knowledge is critical.

This study presents a comprehensive empirical evaluation of LLMs for the automatic generation of VeriFast specifications over a diverse corpus of C functions, analyzing both syntactic and semantic preservation, verifiability, and error modes. The work systematically investigates prompt engineering strategies, model selection, input formats, and pinpoints where and why LLM-generated specifications fail, thus providing actionable direction for future tool and model development.

Methodological Framework

The evaluation is grounded in a dataset of 303 C functions spanning diverse program structures (concurrency, loops, recursive predicates) and three partially specified input variants per function: (1) natural language (NL) behavioral summaries, (2) formal functional behavior (FB) contracts, and (3) extended functional behavior plus (FBP) specifications including auxiliary constructs required for successful verification. Each function also has a fully verified ground truth form, supporting fine-grained assessment.

Eight prompt engineering strategies (including RAG-sparse/dense and Chain-of-Thought) and ten LLMs were initially screened using two pilot studies. RAG-sparse with per-function splitting demonstrated the highest verification success and minimal behavioral deviation; LLM selection (Claude-3-7 Sonnet, Gemini 2.5 Pro, GPT-4o) was based on empirical performance across behavioral preservation and error rate. The study pipeline involves invoking each LLM+prompt+input combination per function, parsing/validating with VeriFast, and qualitative code review by multiple authors to annotate functional deviations and error categories.

Figure 1: The end-to-end evaluation workflow: prompt+input conditioning for each function, LLM generation, followed by automatic and manual validation.

Results: Fidelity and Verifiability

Functional Behavior Preservation

Across 2,727 evaluated function variants, LLMs preserved the input source code behavior in 91% of cases and functional contracts in 92%. Deviations were typically minor (e.g., redundant code, trivial refactoring, or misplaced specifications); large-scale semantic drift was rare, especially when formal contract input was given. NL-only inputs, by contrast, induced ambiguity, resulting in more frequent contract weakening or non-equivalent output.

For contracts, 87% were marked behaviorally equivalent and an additional 5% as strengthened relative to input. Notably, Gemini 2.5 Pro achieved the best robustness to ambiguity, consistently preserving correct semantics even with under-specified prompts.

Verification Success

Despite strong behavioral fidelity, the overall proportion of LLM-synthesized functions passing full VeriFast verification was 31.4%. Success rates increased with richer input specifications (39.9% for FBP vs. 24.6% for NL). Among LLMs, Gemini 2.5 Pro achieved the highest verification rates (39.7%) across all input variants, outperforming Claude-3-7 (29.8%) and GPT-4o (24.5%).

Stratified by program feature, verification succeeded for 51% of "normal" SL functions, but rates fell precipitously for recursive predicate use (31%), concurrency (12%), and loops (11%), indicating that auxiliary proof structure and deeper heap reasoning represent ongoing scalability barriers.

Error Analysis

Manual triage of 354 failing function variants revealed 1,652 errors, classified as follows:

• Practical errors (71%): Sufficient for automated reasoning but not logically unsound; overwhelmingly due to missing or misplaced SL-specific constructs (open/close/assert/leak). 52% stemmed from unsatisfied predicate-based heap chunk requirements.
• Compiler errors (23%): Syntactic or typing failures, often induced by inaccurate code completions or hallucinated constructs, aligned primarily with NL input under-specification.
• Theoretical errors (6%): Incorrect or insufficient specifications for the logic; rare, suggesting LLMs learn task-goal fidelity but struggle with specific SL instantiations.

Errors were disproportionately concentrated in NL (less guided) settings and mitigated for FBP and Gemini 2.5 Pro in particular. Most practical errors relate directly to SL and VeriFast-specific patterns not typically covered in pretraining corpora, including fold/unfold control, auxiliary lemma synthesis, and explicit heap state tracking. Enabling VeriFast's auto features (e.g., for open/close synthesis) could resolve nearly 40% of these cases, motivating hybrid approaches integrating symbolic execution and LLM repair.

Implications and Prospects

This study demonstrates that even state-of-the-art LLMs, with advanced prompt conditioning and RAG, achieve only modest verification success for SL-based code, with performance heavily gated by the completeness and formality of the input specification and exacerbated on tasks requiring intricate heap reasoning (e.g., concurrency and loops). The results clarify that most failures are not due to generic logical inference deficits, but rather to domain-specific gaps—syntactic misalignments, predicate protocol errors, and failure to robustly instantiate heap ownership invariants.

These findings indicate a need for (i) tailored pretraining or finetuning on SL-verifier-centric corpora, (ii) closer integration of verifier-generated debugging/heap-state information into the LLM generation/repair loop, and (iii) symbolic or search-based augmentation (e.g., auto or repair features in VeriFast) to close the gap for auxiliary proof artifacts. LLM-based specification generation for separation logic verifiers, while promising, will require innovations in model design, data, and human-in-the-loop verification interfaces to realize scalable, high-fidelity automation.

Conclusion

An exhaustive empirical assessment of LLMs for VeriFast specification synthesis demonstrates that while behavioral preservation is strong, verification remains bottlenecked by SL- and tool-specific gaps, not general logical reasoning. Verification success is highly sensitive to input specification strength and model selection (with Gemini 2.5 Pro notably superior), but overall rates remain moderate. Addressing practical and compilation-specific errors, especially via hybrid and repair-augmented strategies, represents a tractable path for progress in automated verification for heap-manipulating C code. This work establishes a detailed performance baseline, identifies actionable error categories, and delineates concrete directions for future research in LLM-powered formal methods.