VerifierBench: Multi-Domain Verification Benchmark

• VerifierBench is a generic benchmark label used to assess verification workflows and tools in diverse domains such as C programs, smart contracts, and answer validation.
• It covers various evaluation schemes including formal verification, synthetic ranking tasks, and formal specification-driven vericoding, each with tailored scoring metrics.
• The framework emphasizes reliability and robustness by using detailed scoring systems and test cases that capture soundness, completeness, and discriminative power.

Searching arXiv for papers using the term "VerifierBench" and closely related benchmark variants to ground the article in the current literature. VerifierBench is a recurrent benchmark name in verification research rather than a single universally fixed artifact. In the arXiv literature, the label has been used for a benchmark for formal verification of C programs, a framework for evaluating synthetic verifiers on code, an open benchmark for Solidity verification tools, the benchmark component of CompassVerifier for answer verification, and a large-scale benchmark for vericoding from formal specifications (Eekelen et al., 2019, Ficek et al., 19 Feb 2025, Bartoletti et al., 2024, Liu et al., 5 Aug 2025, Bursuc et al., 26 Sep 2025). Across these usages, the central object of evaluation is not merely a model output, but the reliability, expressiveness, or discriminative power of a verifier or verification workflow.

1. Terminology and scope

The name has been attached to distinct benchmark constructions in different subfields. This suggests that “VerifierBench” functions as a generic benchmark-design label for verifier evaluation rather than as a single canonical dataset spanning all domains (Eekelen et al., 2019, Ficek et al., 19 Feb 2025, Bartoletti et al., 2024, Liu et al., 5 Aug 2025, Bursuc et al., 26 Sep 2025).

Usage in the literature	Domain	Defining benchmark object
Formal verification of C programs	C verification	Twenty-five C programs in five thematic families, with a 100-point scoring formula
Synthetic verification for code	Code evaluation	HE-R, HE-R+, MBPP-R, and MBPP-R+ scoring/ranking datasets
Solidity verification tools	Smart contracts	323 verification tasks built from contract implementations and properties
CompassVerifier benchmark component	Answer verification	A 2,817-sample test set with Correct, Incorrect, and Invalid labels
Vericoding benchmark	Formally verified synthesis	12,504 formal specifications across Dafny, Verus/Rust, and Lean

The repeated reuse of the name reflects a common methodological concern: how to construct benchmarked tasks that expose verifier behavior under realistic ambiguity, incomplete expressiveness, weak supervision, or heterogeneous output formats. The differences lie in what is being verified—program behavior, candidate-solution quality, smart-contract properties, answer equivalence, or proof-carrying implementations—and in whether the benchmark emphasizes soundness, completeness, ranking fidelity, or reward quality.

2. VerifierBench as a benchmark for C program verification

In its earliest usage here, VerifierBench is a deliberately small but representative benchmark suite for formal verification of C programs. It contains twenty-five C programs organized into five thematic families: Factorial, Cat, Allocate memory, Quicksort, and Square root. The suite was created to obtain a clearer overview of frameworks for practical C program verification in the Sovereign project, and it was explicitly framed as useful for system demonstration, comparison of verification effort between systems, and friendly competition (Eekelen et al., 2019).

A defining feature of this benchmark is its intended spectrum of granularity. The programs range from very small fragments to one or more functions, to a single source file, to real-life multi-file code, labeled respectively as C, F, S, and R. Three programs are based on GNU implementations used in practice: coreutils/src/[cat](https://www.emergentmind.com/topics/context-as-a-tool-cat-paradigm).c, glibc/malloc/malloc.c, and glibc/soft-fp/sqrtsf2.c. The benchmark was designed to represent “real C programming” rather than obfuscated or contrived puzzles, and it accordingly includes issues such as input/output behavior, memory allocation and freeing, recursion, arrays, pointers, function pointers, structs, unions, typedefs, floating-point behavior and IEEE 754 properties, threads, synchronization, and real-life library code.

The scoring scheme is central to this VerifierBench formulation. The abstract gives a 100-point maximum, formalized as

$25 \times 4 = 100.$

Each program is worth 4 points if properly verified. Full credit requires both that the program does not exhibit undefined behavior and that its functional correctness is proved. The paper also defines point deductions for simplifications: no verification of defined behavior $(-1)$, no verification of functional correctness $(-1)$, verified modified code that is still valid C $(-1)$, and verified only a model of the code $(-2)$. The same scheme is also presented positively as four separate $+1$ contributions: verification of defined behavior, verification of functional correctness, verified valid C, and verified unmodified code.

The five program families are designed to expose distinct verification phenomena. Factorial ranges from simple loops to recursive and big-number versions, illustrating overflow, recursion depth, library specification, and arbitrary-precision arithmetic. Cat covers getchar/putchar, fread/fwrite, simplified GNU cat, and the full GNU implementation, emphasizing the authors’ view that I/O is “underrated” in C verification. The allocator family spans bump-pointer allocation, typed allocation, union-cell tracking, simplified GNU malloc/free, and the real GNU allocator, stressing alignment, linked structures, memory safety, and separation-logic-style reasoning. Quicksort includes recursive, goto-based, macro-heavy, generic qsort-style, and pthread-based parallel variants, thereby combining algorithmic correctness with recursion, macro reasoning, function pointers, mutexes, and stack-usage considerations. Square root moves from Newton iteration and the 0x5f3759df-style approximation to union-based and GNU soft-fp implementations, focusing on strict aliasing, denormals, infinities, NaNs, rounding modes, and implementation-defined assumptions such as __STDC_IEC_559__.

The benchmark intentionally leaves several evaluation details open-ended. Each verification system may choose its own specification style, though in practice it should include restrictions on the C implementation, preconditions on inputs, and postconditions describing behavior. This openness is part of the benchmark’s stated character as a “friendly challenge,” not a rigid competition with fully fixed semantics.

3. VerifierBench for synthetic verification of code and reasoning

A later use of the name denotes a benchmark family for evaluating synthetic verifiers in code generation. Here the problem is not proving a program correct directly, but determining whether generated tests or reward scores can correctly estimate solution correctness and rank candidate programs. The benchmark family consists of HE-R, HE-R+, MBPP-R, and MBPP-R+, transformed from HumanEval and MBPP into scoring and ranking datasets (Ficek et al., 19 Feb 2025).

The construction begins from benchmarks with ground-truth tests, specifically the EvalPlus extended versions of HumanEval and MBPP. Instead of retaining only pass/fail labels, the framework reconstructs evaluation so that each solution receives a score equal to the fraction of tests it passes. Candidate solutions are generated with GPT-4o, including fully correct and partially incorrect programs obtained via different prompts, sampling settings, and seeds. Solutions are then filtered and deduplicated; duplicates with the same score are removed, ties are broken by lower execution time, the ground-truth solution is kept if multiple programs pass all tests, and solutions that crash for non-assertion reasons are discarded. A subset of solutions with evenly spaced quality levels is then selected, creating a ranking task rather than a binary correctness task.

The formal setup sorts solutions by their fraction of tests passed. If

$S = \{ s_1, s_2, \dots, s_n \},$

then the solutions are ordered so that

$s_1 \ge s_2 \ge \cdots \ge s_n$

with $s_1 = M = 1.0$. The paper then defines a minimum nontrivial score $m$, target scores $(-1)$0, and a selection rule that chooses solutions closest to those targets, yielding a ranked set $(-1)$1. The point of this construction is to force a verifier to discriminate among multiple imperfect solutions, not merely to recognize one correct answer.

Evaluation uses Top-1 Accuracy, Bottom-1 Accuracy, Spearman’s $(-1)$2, Kendall’s $(-1)$3, and Mean Absolute Error. Top-1 Accuracy asks whether the verifier places the best solution first; Bottom-1 Accuracy asks whether it places the worst solution last. Spearman’s $(-1)$4 and Kendall’s $(-1)$5 quantify rank agreement, while MAE measures score-estimation error:

$(-1)$6

The paper explicitly compares standard LLMs, reward-based LLMs, and reasoning models, including Meta-Llama, Qwen, GPT-4o, AceCodeRM, Nemotron reward models, DeepSeek-R1, o1-mini, and o3-mini.

The reported findings emphasize that synthetic verification can meaningfully rank solutions and that reasoning models are particularly strong in test generation. DeepSeek-R1 reaches HE-R+ Top-1 of 83.8% and MBPP-R+ Top-1 of 77.5%, while o3-mini reaches HE-R+ Top-1 of 88.2% and MBPP-R+ Top-1 of 79.9%. The benchmark also studies scaling the number of generated test cases from 5 to 25 and reports that more test cases generally improve verification accuracy, especially for stronger reasoning models. The paper’s interpretation is that stronger reasoning improves test-case generation, and that test-budget scaling improves detection quality, but in a model-dependent manner.

4. VerifierBench for Solidity verification tools

In Solidity verification, VerifierBench is an open benchmark for tools that prove or refute user-specified correctness properties of smart contracts. Its motivation is the mismatch between general-purpose verification benchmarks and the semantics of Solidity, where properties of interest concern balances, withdrawals, liquidity, external calls, msg.sender, tx.origin, selfdestruct, and the distinction between contracts and externally-owned accounts. The benchmark therefore targets general verification tools for Solidity rather than bug-finding tools for fixed vulnerability classes (Bartoletti et al., 2024).

The benchmark is organized around three components: informal use-case specifications, Solidity implementations with tool-specific property encodings, and manually curated ground truth. A verification task is defined as a pair

$(-1)$7

The current benchmark contains 323 verification tasks. Multiple implementations may correspond to the same use case, with some satisfying the intended property and others violating it, sometimes subtly.

Two tool ecosystems structure the benchmark design. For SolCMC, properties are encoded in Solidity itself by adding wrapper functions with assert statements or by injecting ghost code fragments, with the constraint that observable behavior of the original contract should not be changed except to signal violation. SolCMC then transforms the instrumented contract into Constrained Horn Clauses. For Certora, properties are written in CVL, using invariants for state properties and rules for trace or sequence properties. The paper treats expressiveness as part of the benchmarked phenomenon: some properties can be stated in one framework but not faithfully in another.

The scoring system is asymmetric and security-oriented. Null-score outcomes are ND for non-expressible properties and UNK for timeout, memory exhaustion, or no output. Correct strong claims receive 2 points, correct weak claims 1 point, a weak false positive receives $(-1)$8, a strong false negative receives $(-1)$9, and a strong false positive receives $(-1)$0. This asymmetry encodes the benchmark’s view that an unsound claim that a violated property holds is especially severe in smart-contract verification.

The reported comparison shows that Certora is more expressive, particularly for properties over multiple method calls and richer trace structure, but that both Certora and SolCMC exhibit incompleteness and soundness issues. The benchmark reports the following aggregate scores: Certora 176, SolCMC with Z3 165, and SolCMC with Eldarica 90. The paper also stresses that some property classes remain beyond current expressiveness, including certain liveness, probabilistic, fairness, and game-theoretic properties, as well as precise reasoning about EOAs and exact ETH-transfer behavior under all semantic corner cases.

5. VerifierBench as the benchmark component of CompassVerifier

Within the CompassVerifier work, VerifierBench is a benchmark for answer verification built from real LLM responses paired with verified labels. The benchmark is designed to expose the limitations of brittle regex or string matching and of generic LLM judges when answers have diverse formats, ambiguous equivalences, or abnormal outputs. The construction starts from a very large response pool collected with OpenCompass: the main text states “over one million” responses, and the appendix gives more than 1,325,293 samples from 53 LLMs evaluated on 16 benchmark datasets (Liu et al., 5 Aug 2025).

The data are first described as triplets

$(-1)$1

where $(-1)$2 is the question, $(-1)$3 the reference answer, and $(-1)$4 the response from model $(-1)$5. VerifierBench augments these into quadruples

$(-1)$6

where

$(-1)$7

The ternary label space is a defining design choice. Invalid covers truncated outputs, excessive repetition, refusals to answer, garbled text, and cases where the question or reference answer is itself flawed or ambiguous.

The pipeline is multi-stage. First, multiple Qwen2.5-Instruct models and, for math, Math-Verify, remove trivial consensus cases. Disputed cases are then judged with multiple reasoning paths from DeepSeek-V3; selected datasets can receive additional domain-specific prompts, including specially crafted Chinese-language prompts for Chinese SimpleQA. Remaining disputed samples are manually annotated, and annotators must write detailed judgment rationales. The authors analyze these rationales into more than 30 “meta error patterns,” later used both diagnostically and as training signals for CompassVerifier. The benchmark excludes proof-based questions, open-ended problems, and numerical answers with ambiguous thresholds from the test set.

After strict filtering, around 5,000 samples were reviewed and the final high-quality VerifierBench test set contains 2,817 samples. The appendix states that 54,420 consistent samples from the same pipeline are used as the base train set. The exact test-set label distribution is A 38.76%, B 54.17%, and C 7.06%, corresponding to the three categories, and the domain distribution is General Reasoning 40.86%, Mathematical Reasoning 31.95%, Knowledge 13.74%, and Scientific Reasoning 13.45%. Answer types include Multiple Choice 31.63%, Sequence 16.61%, Numerical 15.41%, Short Text 12.57%, Formula 12.18%, Multi-subproblem 9.98%, and Boolean Answer 1.63%.

In evaluation, the benchmark reports Accuracy and F1 for main evaluation and Accuracy with macro-F1 for the ternary setting. Its role is not confined to benchmarking: the same data foundation is used to train CompassVerifier as an outcome reward model, with 54,420 base train samples, 24,294 adversarial samples, and 18,118 formula samples, totaling 96,832. VerifierBench therefore functions simultaneously as a testbed for answer-verification robustness and as supervised infrastructure for reward modeling.

6. VerifierBench in vericoding, and related benchmark lineages

Another usage of the name refers to a large-scale benchmark for vericoding, defined as LLM generation of formally verified code from formal specifications. This benchmark contains 12,504 formal specifications, with 3,029 in Dafny, 2,334 in Verus/Rust, and 7,141 in Lean; 6,174 tasks are new unseen problems. A task is solved only if the generated file preserves the original specification structure, the language verifier accepts the file, and the generated code and proof blocks pass additional validity checks designed to catch cheating or bypass patterns (Bursuc et al., 26 Sep 2025).

The benchmark formalizes success through a verification predicate over a specification, a file, and a target language $(-1)$8. Construction combines original formal-verification sources, vibe-coding benchmarks, and documentation sources, together with autoformalization and LLM-mediated translation across languages. The evaluation pipeline presents the context and specification, asks a model to fill code and proof blocks, applies custom checks against patterns such as assume(false), sorry, or trivialized specifications, reconstructs the file, and invokes the language verifier. The headline aggregate success rates are 82.2% in Dafny, 44.3% in Verus/Rust, and 26.8% in Lean. The same paper reports that adding natural-language descriptions does not provide statistically significant improvement on vericoding performance.

This usage of the name sits within a broader family of similarly named verification benchmarks. “VeriEquivBench” introduces a ground-truth-free equivalence score for formally verifiable code and contains 2,389 algorithmic problems centered on proving both $(-1)$9 and $(-1)$0 between code and specification (Zeng et al., 7 Oct 2025). “VerifyBench” is a cross-domain benchmark with 4,000 expert-level STEM questions for evaluating reasoning verifiers under four combinations of input type and output length (Li et al., 14 Jul 2025). “VerifyThisBench” evaluates end-to-end program verification from natural language across seven verification tools and includes 41 challenges and 154 tasks, with a relaxed variant called VerifyThisBenchXS containing 481 tasks (Deng et al., 25 May 2025). These neighboring names indicate an active benchmark lineage in which verifier evaluation is becoming more formalized, more domain-specific, and increasingly separated from simple final-answer checking.

Taken together, the various uses of “VerifierBench” show a convergence around benchmarked verification as an object of study in its own right. The specific targets differ—C semantics, synthetic test quality, smart-contract properties, answer equivalence, or formal-specification-driven synthesis—but each benchmark encodes a view of what makes verification reliable: soundness under realistic semantics, ranking fidelity against ground-truth execution, expressiveness for domain-specific properties, robustness to abnormal outputs, or resistance to trivial proof-by-bypass strategies.