C-ACSL Specification Pairs (CASP)

• C-ACSL Specification Pairs (CASP) is a dataset of standalone C functions paired with formal ACSL specifications, all verified via Frama-C.
• It is built from deduplicated Stack code samples and refined through automated filtering, LLM-assisted repair, and manual inspection.
• The dataset underpins research in neuro-symbolic specification synthesis and LLM-supported code generation by providing a binary verification benchmark.

Searching arXiv for CASP and closely related ACSL/C specification synthesis work. C-ACSL Specification Pairs (CASP) is a curated dataset of C functions paired with formal specifications written in ANSI/ISO C Specification Language (ACSL), designed to support benchmarking for formal verification of C code and for evaluation of large-language-model workflows that generate code from specifications, specifications from code, or repairs to non-verifying artifacts (Hertzberg et al., 26 Aug 2025). In CASP, the central object is a formally verified specification–implementation pair: a C function together with an ACSL contract such that the pair verifies with Frama-C. The dataset addresses the absence, at the time of publication, of a large public benchmark of verified C code paired with formal specifications, and it situates such pairs as a basis for rigorous, machine-checkable correctness evaluation rather than informal documentation (Hertzberg et al., 26 Aug 2025).

1. Definition and scope

CASP is short for C-ACSL Specification Pairs. The dataset consists of C code paired with formal specifications written in ANSI/ISO C Specification Language (ACSL), with each specification–implementation pair formally verified using Frama-C (Hertzberg et al., 26 Aug 2025). Its stated purpose is to provide a foundation for benchmarking and further research on integrating automated code generation with verified correctness (Hertzberg et al., 26 Aug 2025).

ACSL is presented as a contract-based specification language for C. In the CASP context, ACSL is used to state preconditions (requires), postconditions (ensures), memory-access constraints such as \valid, side effects via assigns, and invariants and other logical properties (Hertzberg et al., 26 Aug 2025). This makes the pairing of C with ACSL machine-checkable: correctness can be proved against the contract rather than inferred from comments or tests alone (Hertzberg et al., 26 Aug 2025).

A representative example in the source material is the swap contract:

/*@ requires \valid (x) && \valid (y) ; assigns *x, *y; ensures *x == \old (*y) && *y == \old (*x) ; */ void swap (int* x, int* y) { int temp = * x; 8 *x = * y ; *y = temp ; }

This example illustrates the basic CASP object: an implementation paired with a formal behavioral contract (Hertzberg et al., 26 Aug 2025).

The dataset is explicitly pair-oriented rather than file-oriented. This distinction matters because many pre-existing ACSL resources are organized as larger annotated files, whereas CASP extracts standalone function-level C–ACSL pairs suitable for direct use in benchmarking tasks such as spec → code, code → spec, and repair of non-verifying pairs (Hertzberg et al., 26 Aug 2025). A plausible implication is that CASP reduces preprocessing overhead for empirical work on neuro-symbolic or deductive-verification pipelines.

2. Motivation and research context

The motivation for CASP is framed by a three-part gap. First, LLMs can generate code, but the outputs typically lack rigorous correctness guarantees. Second, formal verification can provide those guarantees, but writing verified specifications is difficult, manual, and expertise-intensive. Third, there had been no large, public dataset of verified C code paired with formal ACSL specifications suitable for systematic benchmarking of such tasks (Hertzberg et al., 26 Aug 2025).

Within that landscape, CASP was introduced as a benchmark for formal verification of C code and, specifically, for evaluation settings in which success is determined by formal proof rather than surface similarity (Hertzberg et al., 26 Aug 2025). The source material emphasizes that this supports objective evaluation: a generated pair either verifies or it does not (Hertzberg et al., 26 Aug 2025).

This agenda aligns CASP with adjacent work on ACSL specification synthesis. “Specify What? Enhancing Neural Specification Synthesis by Symbolic Methods” studies the generation of ACSL annotations for C programs using GPT-4, with prompts augmented by PathCrawler and EVA outputs (Granberry et al., 2024). That work is not a CASP dataset paper, but it is directly relevant because it concerns creation of the specification side of C–ACSL pairs and reports that PathCrawler examples can yield more context-aware annotations while EVA reports produce annotations more attuned to runtime errors (Granberry et al., 2024). Likewise, “Seeking Specifications: The Case for Neuro-Symbolic Specification Synthesis” investigates ACSL generation from C using Deepseek-R1, focusing on the distinction between specifications of intent and specifications of implementation, and on prompt augmentation with Frama-C tools such as Pathcrawler and EVA (Granberry et al., 29 Apr 2025).

These neighboring studies clarify the role of CASP in the broader research ecosystem. CASP is not itself an overview method; it is a dataset of verified pairs. The synthesis papers, by contrast, investigate how such pairs might be produced or approximated automatically. This suggests that CASP can serve as an evaluation substrate for the kinds of neuro-symbolic workflows examined in those studies.

3. Construction pipeline

CASP is built in three main steps: extract ACSL-annotated C files from The Stack v1 and v2, verify and repair source files using Frama-C and LLMs, and extract standalone function-level C–ACSL pairs and verify them again (Hertzberg et al., 26 Aug 2025).

The construction begins from the deduplicated versions of The Stack v1 and The Stack v2. The reported totals are 8,625,559 raw code samples from Stack 1, 17,093,668 raw code samples from Stack 2, and a combined total of 25,719,227 (Hertzberg et al., 26 Aug 2025). Candidate files were identified using regular-expression filtering authored by a formal methods expert, targeting ACSL-related syntax including requires, ensures, assigns, invariant, axiomatic, lemma, predicate, logic, behavior, disjoint behaviors, complete behaviors, assumes, \old, \at, \valid, \valid_read, \result, quantifiers such as \forall, \exists, \sum, and loop clauses such as loop invariant, loop assigns, and loop variant (Hertzberg et al., 26 Aug 2025). The strict filter is explicitly given as:

/\*@. *? (predicate |requires | ensures) . *? \*/

This reduced the corpus to 14,525 files matching basic formal-pattern filters and then to 5,916 files with formal specifications after stricter filtering (Hertzberg et al., 26 Aug 2025).

A further restriction was imposed to retain only “minimally complete files”: self-contained files whose dependencies are limited to standard C libraries (Hertzberg et al., 26 Aug 2025). This step addresses a practical obstacle in verification corpora, namely dependence on missing headers, non-standard libraries, or other unavailable code. After this filtering stage, 1,180 minimally complete C files remained (Hertzberg et al., 26 Aug 2025).

The verified source-file stage was followed by function-level extraction. Only functions that do not depend on other functions in the file and are not main were kept (Hertzberg et al., 26 Aug 2025). Two parallel pipelines were then used: a regular-expression pipeline and an LLM-based pipeline using Gemini 2.0 Flash (Hertzberg et al., 26 Aug 2025). The union of both outputs was verified and cleaned, ultimately producing the final pair set (Hertzberg et al., 26 Aug 2025).

4. Verification, repair, and quality control

Verification in CASP used Frama-C 30.0 (Zinc) with the WP plugin and the RTE plugin (Hertzberg et al., 26 Aug 2025). The SMT solvers were Z3 4.8.12, Alt-Ergo 2.6.0, and CVC4 1.8, with a 500,000-step limit and a 60-second timeout per proof goal (Hertzberg et al., 26 Aug 2025). The target condition for each file was that all proof obligations generated by WP and WP+RTE should verify (Hertzberg et al., 26 Aug 2025).

If a file did not verify, the pipeline extracted the non-verifying Frama-C goals and sent the file to Gemini 2.0 Flash, instructing the model to repair the ACSL specifications or minimally adjust the code so that Frama-C would prove all goals (Hertzberg et al., 26 Aug 2025). This repair loop was iterated for up to seven iterations (Hertzberg et al., 26 Aug 2025). The prompt emphasized fixing invalid ACSL syntax, correcting type mismatches, strengthening loop invariants, fixing pre/postcondition failures, improving memory specifications such as \valid, \valid_read, and \separated, adding loop assigns, using complete behaviors and disjoint behaviors correctly, inserting strategic assertions, avoiding undefined variables, and not adding a main() function if not present (Hertzberg et al., 26 Aug 2025).

The source material reports that 177 files were corrected out of 888 that initially failed verification, corresponding to a 19.9% success rate among files requiring modification (Hertzberg et al., 26 Aug 2025). After verification and repair, the process produced 469 verified files in the final verified source-file set (Hertzberg et al., 26 Aug 2025).

At the pair-extraction stage, the LLM-based pipeline was prompted to extract independent functions, the ACSL specification preceding each function, and only the minimal dependencies needed for Frama-C verification, while excluding main and returning output in JSON (Hertzberg et al., 26 Aug 2025). The final union of regex-extracted and LLM-extracted outputs was again checked with Frama-C, and any remaining unverified pairs were manually post-fixed (Hertzberg et al., 26 Aug 2025).

An important final step was manual inspection. The source material states that a manual inspection was conducted to confirm the correctness of every pair, ensuring that the specification matched the implementation, that the pair formally verified, and that the extracted function was self-contained and suitable for benchmarking (Hertzberg et al., 26 Aug 2025). In this sense, manual review serves as a final sanity check after automated filtering, deductive verification, LLM-assisted repair, and extraction.

5. Dataset composition and empirical characteristics

The construction pipeline yields the following reported counts (Hertzberg et al., 26 Aug 2025):

Stage	Count
Total initial code samples from deduplicated Stack v1 and v2	25,719,227
Files matching basic formal-pattern filters	14,525
Files with formal specifications after stricter filtering	5,916
Minimally complete C files containing ACSL	1,180
Verified minimally complete files after Frama-C verification and LLM repair	469
Function/specification pairs produced from the two extraction pipelines	513
Final CASP pairs after exact deduplication and removing one-line implementations	506

The final dataset contains 506 verified C–ACSL specification pairs (Hertzberg et al., 26 Aug 2025). The source material also states that CASP is, at the time of publication, the largest openly released dataset containing ACSL specifications and C code, and that it has been publicly available since June 2025 (Hertzberg et al., 26 Aug 2025).

Beyond size, the paper reports several comparative observations. CASP source files are said to be broader in semantic space than prior datasets, with only about 35 CASP source files found to be very similar to some file in comparison datasets (Hertzberg et al., 26 Aug 2025). The comparison set includes Frama-C Problems, X509-parser, Verker, ACSL By Example, WP examples, ACSL proved, and VecoSet (Hertzberg et al., 26 Aug 2025). The analysis uses CodexEmbed 2B embeddings, t-SNE visualization, and nearest-neighbor distance analysis (Hertzberg et al., 26 Aug 2025). The length distributions indicate that most CASP pairs are short to medium length in both lines of C code and lines of ACSL (Hertzberg et al., 26 Aug 2025).

These characteristics matter because they distinguish CASP from narrowly scoped example suites. The reported limited overlap with previous ACSL datasets suggests that CASP is not merely a repackaging of well-known Frama-C teaching examples (Hertzberg et al., 26 Aug 2025). This suggests a broader empirical basis for benchmarking specification synthesis or verified code generation.

6. Relation to specification synthesis and verification research

CASP occupies a specific position among C/ACSL research artifacts. It is a dataset of verified pairs, whereas several related papers study how to create, translate, or reason about such pairs.

“Specify What? Enhancing Neural Specification Synthesis by Symbolic Methods” examines ACSL generation with GPT-4 (gpt-4-0125-preview) under three prompt variants: baseline_set, pathcrawler_set, and eva_set (Granberry et al., 2024). The study focuses mainly on function contracts—especially requires, ensures, and assigns—and reports that PathCrawler output tends to produce more context-aware annotations, while EVA reports bias synthesis toward runtime-error-aware preconditions (Granberry et al., 2024). It also reports that generated specifications for buggy programs often reflect the intended behavior more than the buggy implementation (Granberry et al., 2024). CASP is relevant here because it supplies verified code–spec pairs that could support systematic evaluation of such synthesis methods.

“Seeking Specifications: The Case for Neuro-Symbolic Specification Synthesis” extends this line by studying ACSL generation from C using Deepseek-R1, again with optional augmentation from Pathcrawler and EVA (Granberry et al., 29 Apr 2025). A central result is the distinction between an implementation specification, which mirrors the code as written, and an intent specification, which reflects the programmer’s intended behavior even when the code is buggy (Granberry et al., 29 Apr 2025). The study is largely qualitative and does not define CASP formally, but it directly engages the practical problem of how C–ACSL pairs should be interpreted when programs contain defects (Granberry et al., 29 Apr 2025). This is relevant to CASP because a verified pair in the dataset necessarily represents a realizable and internally consistent relation between code and contract, whereas synthesis systems may need to choose between describing implementation and describing intent.

A different but conceptually adjacent line appears in “The Right Kind of Non-Determinism: Using Concurrency to Verify C Programs with Underspecified Semantics” (Kamburjan et al., 2022). That paper translates C programs with underspecified semantics into an Active Object model and automatically translates ACSL into method contracts and object invariants, so that verification of the extracted model with Crowbar ensures the original C property for all possible evaluation orders (Kamburjan et al., 2022). Source requires/ensures clauses become target method contracts, and source strong global invariants become target object invariants plus auxiliary contracts on getters and setters (Kamburjan et al., 2022). Although this work does not define CASP, it demonstrates a systematic source/target pairing discipline for ACSL specifications. A plausible implication is that CASP-style pairs are not only useful for LLM benchmarking but also for studying semantics-preserving translation of verification artifacts across formalisms.

7. Significance, use cases, and limitations

The source material identifies three principal advantages of specification–implementation pairs over whole annotated files: better benchmarking for LLMs, logical consistency, and bidirectional evaluation (Hertzberg et al., 26 Aug 2025). Because each CASP entry is a standalone verified pair, it can be used to evaluate code generation from a specification, specification generation from code, and repair of non-verifying code or specifications under a common correctness criterion (Hertzberg et al., 26 Aug 2025).

The dataset is also intended to support research on integrating LLMs with formal methods (Hertzberg et al., 26 Aug 2025). In this setting, CASP functions as an evaluation benchmark rather than a modeling claim. Its correctness notion is operationalized through Frama-C verification, which provides a binary success condition absent from purely text-based code-generation benchmarks (Hertzberg et al., 26 Aug 2025). This is particularly relevant to research such as neuro-symbolic specification synthesis, where prompt design, symbolic context, and model behavior can substantially alter the generated ACSL (Granberry et al., 2024, Granberry et al., 29 Apr 2025).

Several limitations or cautions are explicit in the surrounding literature. In the CASP construction itself, the need for LLM repair of non-verifying files, combined with a 19.9% success rate among files requiring modification, indicates that automatically obtaining high-quality verifying ACSL is difficult even when starting from existing annotations (Hertzberg et al., 26 Aug 2025). In the synthesis literature, PathCrawler examples can improve abstraction but may also induce overgeneralization when the examples are sparse (Granberry et al., 2024), while EVA can yield stronger safety-oriented preconditions at the cost of focusing too narrowly on runtime-error prevention rather than full functional intent (Granberry et al., 2024, Granberry et al., 29 Apr 2025).

A further conceptual issue concerns the interpretation of a specification. The neuro-symbolic synthesis work shows that a model may generate specifications of implementation or of intent, depending on prompt wording and auxiliary symbolic evidence (Granberry et al., 29 Apr 2025). CASP’s verified-pair orientation avoids some of this ambiguity by requiring realizable, formally checked pairs, but it does not eliminate the broader research question of whether a desirable specification should describe the current code, the intended algorithm, or a safety envelope around the implementation. This suggests that CASP is best understood as infrastructure for studying verified C–ACSL pairing under a precise tool-based notion of correctness, while leaving open deeper questions about the semantics and provenance of specifications.