---
title: C-ACSL Specification Pairs (CASP)
url: https://www.emergentmind.com/topics/c-acsl-specification-pairs-casp
type: topic
---

# C-ACSL Specification Pairs (CASP)

Searching arXiv for CASP and closely related ACSL/C specification synthesis work.
C-ACSL Specification Pairs (CASP) is a curated dataset of C functions paired with formal specifications written in ANSI/ISO C Specification Language (ACSL), designed to support benchmarking for formal verification of C code and for evaluation of large-language-model workflows that generate code from specifications, specifications from code, or repairs to non-verifying artifacts [2508.18798]. In CASP, the central object is a formally verified specification–implementation pair: a C function together with an ACSL contract such that the pair verifies with Frama-C. The dataset addresses the absence, at the time of publication, of a large public benchmark of verified C code paired with formal specifications, and it situates such pairs as a basis for rigorous, machine-checkable correctness evaluation rather than informal documentation [2508.18798].

## 1. Definition and scope

CASP is short for **C-ACSL Specification Pairs**. The dataset consists of **C code paired with formal specifications written in ANSI/ISO C Specification Language (ACSL)**, with each specification–implementation pair formally verified using Frama-C [2508.18798]. Its stated purpose is to provide a foundation for benchmarking and further research on integrating automated code generation with verified correctness [2508.18798].

ACSL is presented as a **contract-based specification language** for C. In the CASP context, ACSL is used to state **preconditions** (`requires`), **postconditions** (`ensures`), **memory-access constraints** such as `\valid`, **side effects** via `assigns`, and **invariants** and other logical properties [2508.18798]. This makes the pairing of C with ACSL machine-checkable: correctness can be proved against the contract rather than inferred from comments or tests alone [2508.18798].

A representative example in the source material is the `swap` contract:

```c
/*@ requires \valid (x) && \valid (y) ;
    assigns *x, *y;
    ensures *x == \old (*y) && *y == \old (*x) ;
*/
void swap (int* x, int* y) {
  int temp = * x;
  8 *x = * y ;
  *y = temp ;
}
```

This example illustrates the basic CASP object: an implementation paired with a formal behavioral contract [2508.18798].

The dataset is explicitly pair-oriented rather than file-oriented. This distinction matters because many pre-existing ACSL resources are organized as larger annotated files, whereas CASP extracts **standalone function-level C–ACSL pairs** suitable for direct use in benchmarking tasks such as **spec → code**, **code → spec**, and repair of non-verifying pairs [2508.18798]. A plausible implication is that CASP reduces preprocessing overhead for empirical work on neuro-symbolic or deductive-verification pipelines.

## 2. Motivation and research context

The motivation for CASP is framed by a three-part gap. First, **LLMs can generate code**, but the outputs typically lack rigorous correctness guarantees. Second, **formal verification can provide those guarantees**, but writing verified specifications is difficult, manual, and expertise-intensive. Third, there had been **no large, public dataset of verified C code paired with formal ACSL specifications** suitable for systematic benchmarking of such tasks [2508.18798].

Within that landscape, CASP was introduced as a benchmark for formal verification of C code and, specifically, for evaluation settings in which success is determined by formal proof rather than surface similarity [2508.18798]. The source material emphasizes that this supports objective evaluation: a generated pair either verifies or it does not [2508.18798].

This agenda aligns CASP with adjacent work on ACSL specification synthesis. “Specify What? Enhancing Neural Specification Synthesis by Symbolic Methods” studies the generation of ACSL annotations for C programs using GPT-4, with prompts augmented by PathCrawler and EVA outputs [2406.15540]. That work is not a CASP dataset paper, but it is directly relevant because it concerns creation of the specification side of C–ACSL pairs and reports that **PathCrawler examples** can yield more context-aware annotations while **EVA reports** produce annotations more attuned to runtime errors [2406.15540]. Likewise, “Seeking Specifications: The Case for Neuro-Symbolic Specification Synthesis” investigates ACSL generation from C using Deepseek-R1, focusing on the distinction between specifications of **intent** and specifications of **implementation**, and on prompt augmentation with Frama-C tools such as Pathcrawler and EVA [2504.21061].

These neighboring studies clarify the role of CASP in the broader research ecosystem. CASP is not itself a synthesis method; it is a dataset of verified pairs. The synthesis papers, by contrast, investigate how such pairs might be produced or approximated automatically. This suggests that CASP can serve as an evaluation substrate for the kinds of neuro-symbolic workflows examined in those studies.

## 3. Construction pipeline

CASP is built in **three main steps**: **extract ACSL-annotated C files from The Stack v1 and v2**, **verify and repair source files using Frama-C and LLMs**, and **extract standalone function-level C–ACSL pairs and verify them again** [2508.18798].

The construction begins from the **deduplicated versions** of **The Stack v1** and **The Stack v2**. The reported totals are **8,625,559 raw code samples** from Stack 1, **17,093,668 raw code samples** from Stack 2, and a **combined total of 25,719,227** [2508.18798]. Candidate files were identified using **regular-expression filtering** authored by a **formal methods expert**, targeting ACSL-related syntax including `requires`, `ensures`, `assigns`, `invariant`, `axiomatic`, `lemma`, `predicate`, `logic`, `behavior`, `disjoint behaviors`, `complete behaviors`, `assumes`, `\old`, `\at`, `\valid`, `\valid_read`, `\result`, quantifiers such as `\forall`, `\exists`, `\sum`, and loop clauses such as `loop invariant`, `loop assigns`, and `loop variant` [2508.18798]. The strict filter is explicitly given as:

> `/\*@. *? (predicate |requires | ensures) . *? \*/`

This reduced the corpus to **14,525** files matching basic formal-pattern filters and then to **5,916** files with formal specifications after stricter filtering [2508.18798].

A further restriction was imposed to retain only **“minimally complete files”**: self-contained files whose dependencies are limited to **standard C libraries** [2508.18798]. This step addresses a practical obstacle in verification corpora, namely dependence on missing headers, non-standard libraries, or other unavailable code. After this filtering stage, **1,180 minimally complete C files** remained [2508.18798].

The verified source-file stage was followed by function-level extraction. Only functions that **do not depend on other functions in the file** and are **not `main`** were kept [2508.18798]. Two parallel pipelines were then used: a **regular-expression pipeline** and an **LLM-based pipeline** using **Gemini 2.0 Flash** [2508.18798]. The union of both outputs was verified and cleaned, ultimately producing the final pair set [2508.18798].

## 4. Verification, repair, and quality control

Verification in CASP used **Frama-C 30.0 (Zinc)** with the **WP** plugin and the **RTE** plugin [2508.18798]. The SMT solvers were **Z3 4.8.12**, **Alt-Ergo 2.6.0**, and **CVC4 1.8**, with a **500,000-step limit** and a **60-second timeout per proof goal** [2508.18798]. The target condition for each file was that all proof obligations generated by **WP** and **WP+RTE** should verify [2508.18798].

If a file did not verify, the pipeline extracted the non-verifying Frama-C goals and sent the file to **Gemini 2.0 Flash**, instructing the model to **repair the ACSL specifications or minimally adjust the code** so that Frama-C would prove all goals [2508.18798]. This repair loop was iterated for **up to seven iterations** [2508.18798]. The prompt emphasized fixing invalid ACSL syntax, correcting type mismatches, strengthening loop invariants, fixing pre/postcondition failures, improving memory specifications such as `\valid`, `\valid_read`, and `\separated`, adding loop `assigns`, using `complete behaviors` and `disjoint behaviors` correctly, inserting strategic assertions, avoiding undefined variables, and not adding a `main()` function if not present [2508.18798].

The source material reports that **177 files** were corrected out of **888** that initially failed verification, corresponding to a **19.9% success rate** among files requiring modification [2508.18798]. After verification and repair, the process produced **469 verified files** in the final verified source-file set [2508.18798].

At the pair-extraction stage, the LLM-based pipeline was prompted to extract independent functions, the ACSL specification preceding each function, and only the minimal dependencies needed for Frama-C verification, while excluding `main` and returning output in JSON [2508.18798]. The final union of regex-extracted and LLM-extracted outputs was again checked with Frama-C, and any remaining unverified pairs were **manually post-fixed** [2508.18798].

An important final step was **manual inspection**. The source material states that a manual inspection was conducted to confirm the correctness of **every pair**, ensuring that the specification matched the implementation, that the pair formally verified, and that the extracted function was self-contained and suitable for benchmarking [2508.18798]. In this sense, manual review serves as a final sanity check after automated filtering, deductive verification, LLM-assisted repair, and extraction.

## 5. Dataset composition and empirical characteristics

The construction pipeline yields the following reported counts [2508.18798]:

| Stage | Count |
|---|---:|
| Total initial code samples from deduplicated Stack v1 and v2 | 25,719,227 |
| Files matching basic formal-pattern filters | 14,525 |
| Files with formal specifications after stricter filtering | 5,916 |
| Minimally complete C files containing ACSL | 1,180 |
| Verified minimally complete files after Frama-C verification and LLM repair | 469 |
| Function/specification pairs produced from the two extraction pipelines | 513 |
| Final CASP pairs after exact deduplication and removing one-line implementations | 506 |

The final dataset contains **506 verified C–ACSL specification pairs** [2508.18798]. The source material also states that CASP is, at the time of publication, the **largest openly released dataset containing ACSL specifications and C code**, and that it has been publicly available since **June 2025** [2508.18798].

Beyond size, the paper reports several comparative observations. CASP source files are said to be **broader in semantic space** than prior datasets, with only about **35 CASP source files** found to be very similar to some file in comparison datasets [2508.18798]. The comparison set includes **Frama-C Problems**, **X509-parser**, **Verker**, **ACSL By Example**, **WP examples**, **ACSL proved**, and **VecoSet** [2508.18798]. The analysis uses **CodexEmbed 2B** embeddings, **t-SNE** visualization, and **nearest-neighbor distance** analysis [2508.18798]. The length distributions indicate that most CASP pairs are **short to medium length** in both lines of C code and lines of ACSL [2508.18798].

These characteristics matter because they distinguish CASP from narrowly scoped example suites. The reported limited overlap with previous ACSL datasets suggests that CASP is not merely a repackaging of well-known Frama-C teaching examples [2508.18798]. This suggests a broader empirical basis for benchmarking specification synthesis or verified code generation.

## 6. Relation to specification synthesis and verification research

CASP occupies a specific position among C/ACSL research artifacts. It is a dataset of verified pairs, whereas several related papers study how to create, translate, or reason about such pairs.

“Specify What? Enhancing Neural Specification Synthesis by Symbolic Methods” examines ACSL generation with **GPT-4 (gpt-4-0125-preview)** under three prompt variants: **baseline_set**, **pathcrawler_set**, and **eva_set** [2406.15540]. The study focuses mainly on function contracts—especially `requires`, `ensures`, and `assigns`—and reports that **PathCrawler** output tends to produce more context-aware annotations, while **EVA** reports bias synthesis toward runtime-error-aware preconditions [2406.15540]. It also reports that generated specifications for buggy programs often reflect the **intended behavior** more than the buggy implementation [2406.15540]. CASP is relevant here because it supplies verified code–spec pairs that could support systematic evaluation of such synthesis methods.

“Seeking Specifications: The Case for Neuro-Symbolic Specification Synthesis” extends this line by studying ACSL generation from C using **Deepseek-R1**, again with optional augmentation from **Pathcrawler** and **EVA** [2504.21061]. A central result is the distinction between an **implementation specification**, which mirrors the code as written, and an **intent specification**, which reflects the programmer’s intended behavior even when the code is buggy [2504.21061]. The study is largely qualitative and does not define CASP formally, but it directly engages the practical problem of how C–ACSL pairs should be interpreted when programs contain defects [2504.21061]. This is relevant to CASP because a verified pair in the dataset necessarily represents a realizable and internally consistent relation between code and contract, whereas synthesis systems may need to choose between describing implementation and describing intent.

A different but conceptually adjacent line appears in “The Right Kind of Non-Determinism: Using Concurrency to Verify C Programs with Underspecified Semantics” [2208.04630]. That paper translates C programs with underspecified semantics into an **Active Object** model and automatically translates ACSL into method contracts and object invariants, so that verification of the extracted model with **Crowbar** ensures the original C property for **all possible evaluation orders** [2208.04630]. Source **requires/ensures** clauses become target **method contracts**, and source **strong global invariants** become target **object invariants** plus auxiliary contracts on getters and setters [2208.04630]. Although this work does not define CASP, it demonstrates a systematic source/target pairing discipline for ACSL specifications. A plausible implication is that CASP-style pairs are not only useful for LLM benchmarking but also for studying semantics-preserving translation of verification artifacts across formalisms.

## 7. Significance, use cases, and limitations

The source material identifies three principal advantages of **specification–implementation pairs** over whole annotated files: **better benchmarking for LLMs**, **logical consistency**, and **bidirectional evaluation** [2508.18798]. Because each CASP entry is a standalone verified pair, it can be used to evaluate **code generation from a specification**, **specification generation from code**, and repair of non-verifying code or specifications under a common correctness criterion [2508.18798].

The dataset is also intended to support research on integrating **LLMs with formal methods** [2508.18798]. In this setting, CASP functions as an evaluation benchmark rather than a modeling claim. Its correctness notion is operationalized through Frama-C verification, which provides a binary success condition absent from purely text-based code-generation benchmarks [2508.18798]. This is particularly relevant to research such as neuro-symbolic specification synthesis, where prompt design, symbolic context, and model behavior can substantially alter the generated ACSL [2406.15540] [2504.21061].

Several limitations or cautions are explicit in the surrounding literature. In the CASP construction itself, the need for **LLM repair** of non-verifying files, combined with a **19.9% success rate** among files requiring modification, indicates that automatically obtaining high-quality verifying ACSL is difficult even when starting from existing annotations [2508.18798]. In the synthesis literature, **PathCrawler** examples can improve abstraction but may also induce overgeneralization when the examples are sparse [2406.15540], while **EVA** can yield stronger safety-oriented preconditions at the cost of focusing too narrowly on runtime-error prevention rather than full functional intent [2406.15540] [2504.21061].

A further conceptual issue concerns the interpretation of a specification. The neuro-symbolic synthesis work shows that a model may generate specifications of **implementation** or of **intent**, depending on prompt wording and auxiliary symbolic evidence [2504.21061]. CASP’s verified-pair orientation avoids some of this ambiguity by requiring realizable, formally checked pairs, but it does not eliminate the broader research question of whether a desirable specification should describe the current code, the intended algorithm, or a safety envelope around the implementation. This suggests that CASP is best understood as infrastructure for studying verified C–ACSL pairing under a precise tool-based notion of correctness, while leaving open deeper questions about the semantics and provenance of specifications.

Source: https://www.emergentmind.com/topics/c-acsl-specification-pairs-casp