SCI-VerifyBench: Scientific Verification Benchmark

• SCI-VerifyBench is a cross-disciplinary benchmark for verifying scientific answers by assessing semantic and algebraic equivalence across multiple domains.
• It spans mathematics, physics, chemistry, biology, and general science, employing both real model outputs and synthetic equivalence transformations to simulate realistic answer variations.
• The benchmark’s systematic annotation protocol and evaluation strategy address prior limitations by focusing on expert-defined equivalence rather than exact string matches.

SCI-VerifyBench is a cross-disciplinary benchmark for scientific answer verification introduced alongside "SCI-Verifier: Scientific Verifier with Thinking" (Zheng et al., 29 Sep 2025). It was designed to provide a systematic, rigorous, and cross-disciplinary scientific answer verification benchmark at a time when existing verification studies in scientific domains were described as limited by insufficient disciplinary coverage and inadequate treatment of discipline-specific answer equivalence. Its central objective is to simulate real-world scientific answer equivalence, where multiple surface forms may be fully correct, and thereby to evaluate whether a verifier can judge candidate responses against a reference answer under mathematically, physically, chemically, biologically, and semantically nontrivial equivalence conditions (Zheng et al., 29 Sep 2025).

1. Scope and problem setting

SCI-VerifyBench spans five major scientific domains: mathematics, physics, chemistry, biology, and general scientific QA (Zheng et al., 29 Sep 2025). The task is defined over tuples $(q_i, a_i, r_i, l_i)$, where $q_i$ is the question, $a_i$ is the reference answer, $r_i$ is the candidate response to be judged, and $l_i$ is the label indicating whether the response is equivalent or not. In operational terms, the benchmark asks a verifier, given $(q, a, r)$, to determine whether $r$ is semantically and/or algebraically equivalent to $a$ according to domain judgment criteria (Zheng et al., 29 Sep 2025).

The benchmark is explicitly motivated by the observation that scientific verification cannot be reduced to exact-match comparison. The paper identifies two major limitations in prior scientific verification work: the absence of systematic evaluation standards with insufficient disciplinary coverage, and heavy reliance on cumbersome rule design or prompt engineering. SCI-VerifyBench addresses these limits by combining broad domain coverage with explicit modeling of equivalence forms that are common in scientific reasoning but often invisible to string-level matching (Zheng et al., 29 Sep 2025).

A common misconception is that answer verification in scientific settings is principally a formatting problem. SCI-VerifyBench is structured around the opposite premise: correct scientific answers often differ in notation, units, symbolic form, or paraphrastic expression while remaining fully equivalent. This benchmark therefore treats equivalence recognition, rather than literal overlap, as the core verification problem.

2. Construction pipeline

SCI-VerifyBench was constructed from both real model outputs and synthetic equivalence-augmented data (Zheng et al., 29 Sep 2025). The data pipeline begins with a diverse, curated set of more than 15,000 question-answer pairs drawn from well-established datasets for each domain. Over 100,000 responses were then collected from eight LLMs of various sizes. This use of real LLM responses grounds the benchmark in actual model behavior rather than only in manually authored variants (Zheng et al., 29 Sep 2025).

A second component of the construction process is synthetic augmentation through domain-specific equivalence transformations. These transformations are applied to base question-answer pairs to generate challenging and realistic data, with the stated purpose of mimicking realistic variant answers found in expert settings. For each selected question, five equivalence transformations are generated (Zheng et al., 29 Sep 2025). The benchmark therefore combines naturally occurring model errors and reformulations with controlled equivalence cases designed to probe edge conditions.

Difficulty control is built into the curation procedure. The paper states that difficulty is increased by prioritizing samples with high model disagreement, and that challenges are balanced across all domains. This means the benchmark is not merely broad in coverage but intentionally populated with cases that LLMs find difficult to adjudicate. A plausible implication is that SCI-VerifyBench functions not only as a coverage benchmark but also as a stress test for verifier calibration under disagreement-rich conditions.

3. Domain-specific equivalence transformations

The distinctive methodological feature of SCI-VerifyBench is its systematic use of equivalence transformations specific to each scientific domain (Zheng et al., 29 Sep 2025). These transformations are intended to simulate realistic answer diversity rather than artificial perturbations.

In mathematics, the benchmark includes different algebraic manipulations such as factoring, expanding, and simplifying; interval notation versus inequalities; fraction versus decimal or partial fractions; trigonometric identities; and Taylor expansions and approximations. The paper gives examples such as $\frac{1}{x^2}$ being represented as $x^{-2}$, $q_i$0, $q_i$1, and $q_i$2, as well as transformations such as $q_i$3, $q_i$4, and $q_i$5 (Zheng et al., 29 Sep 2025).

In physics, equivalence includes unit conversions, scientific notation, and dimensional analysis. The benchmark examples include $q_i$6, $q_i$7 “reflux,” and $q_i$8 (Zheng et al., 29 Sep 2025). These cases directly test whether a verifier can normalize units and preserve physical meaning across representation changes.

In chemistry, the benchmark includes IUPAC names, common names, formulas, SMILES, synonyms, and equivalence in solvents, retrosynthesis, and property values under unit or scale conversions. The illustrative equivalence “Ethanol” $q_i$9 “EtOH” $a_i$0 “C2H5OH” $a_i$1 “alcohol” $a_i$2 “polar protic solvent” exemplifies the challenge of mapping across nomenclature systems and contextual descriptors (Zheng et al., 29 Sep 2025).

In biology, equivalence transformations cover protein and RNA notation, one-letter versus three-letter codes, dot-bracket versus CT formats, conservatively substituted sequences, and different structural or sequence serializations such as JSON, arrays, and structured text. In general scientific QA, the transformations include paraphrasing and rewrites as lists, tables, or logical expressions, with tolerance for semantically equivalent but differently phrased statements. The example “Water boils at 100°C at sea level” versus “At sea level, the boiling temperature of water is 100 degrees Celsius” illustrates this broader semantic regime (Zheng et al., 29 Sep 2025).

These transformations define the benchmark’s epistemic scope. SCI-VerifyBench is not restricted to symbolic equivalence in a narrow mathematical sense; it also covers synonymy, notation conversion, unit normalization, and paraphrastic semantic preservation across scientific subfields.

4. Annotation protocol and dataset composition

SCI-VerifyBench uses a two-tier annotation process combining model-based annotation with expert review (Zheng et al., 29 Sep 2025). Five LLMs first annotate whether candidate answers are equivalent to gold answers, using prompts designed to surface subtle or non-obvious equivalence. Agreement and disagreement among these LLMs are then used to flag items for further review.

For each domain, 500 items with the greatest LLM disagreement are human-annotated, yielding 2,500 expert-reviewed items across the five domains. Each such item is labeled by at least two domain experts, with adjudication by a third expert if disagreement arises. Samples for the test set are drawn where expert agreement is unanimous (Zheng et al., 29 Sep 2025). This design concentrates human effort on the highest-uncertainty cases while preserving a high-confidence evaluation set.

The benchmark statistics reported for SCI-VerifyBench are as follows:

Component	Quantity
Test set size	2,500 samples
Domains	500 samples per domain
Per-domain composition	350 real LLM responses + 150 equivalence-transformed synthetic responses
Training set	~14,000 samples

All test set items have full expert consensus on the label. The training set is described as mostly LLM-annotated, filtered for reliability, with some expert labels (Zheng et al., 29 Sep 2025).

The paper also reports average lengths of approximately 25 answer tokens and approximately 3,000 response tokens. This asymmetry is important: the reference answer is often compact, while the candidate response may be lengthy and contain extraneous reasoning, formatting variation, or partial derivations. A plausible implication is that SCI-VerifyBench implicitly evaluates not only equivalence recognition but also the ability to isolate the answer-bearing content of long responses.

5. Evaluation protocol, labels, and metrics

The benchmark uses a protocol-structured verification prompt. The verifier is instructed to consider algebraic and numeric equivalence for mathematics and physics, accept unit conversions, synonyms, and valid paraphrases as equivalent, and reject incomplete, repetitive, or refusal responses (Zheng et al., 29 Sep 2025). The judgment categories are explicitly defined as:

• A – CORRECT: Fully equivalent, including exact matches and valid alternative forms.
• B – INCORRECT: Any deviation, including partial answers for multipart questions.
• C – INCOMPLETE/REPETITIVE/REFUSAL: Responses that do not provide a valid equivalent answer.

Although the benchmark uses these three judgment categories in the protocol, the dataset label $a_i$3 is presented as a boolean equivalence label in the tuple definition. This suggests that the operational evaluation ultimately reduces responses to an equivalent-versus-not-equivalent decision, while preserving finer annotation guidance during inference and adjudication.

For SCI-VerifyBench itself, the reported metric is accuracy, justified by the statement that positive and negative samples are balanced by construction (Zheng et al., 29 Sep 2025). The paper notes that F1 score is used on other benchmarks where class imbalance may occur. This metric choice is consistent with the benchmark’s intended use as a controlled verifier-evaluation testbed rather than a natural-distribution deployment corpus.

The benchmark is employed to evaluate the verification ability of LLMs and specialized verifiers under cross-domain, equivalence-aware conditions, to assess domain generalization, to support training of strong verifiers, and to monitor progress through head-to-head benchmarking (Zheng et al., 29 Sep 2025). In that sense, SCI-VerifyBench sits at the intersection of evaluator benchmarking and training-data infrastructure.

6. Relation to adjacent verifier benchmarks and subsequent use

SCI-VerifyBench belongs to a broader wave of benchmarks that evaluate verifier accuracy rather than only end-task generation quality. In "VerifyBench: Benchmarking Reference-based Reward Systems for LLMs" (Yan et al., 21 May 2025), the emphasis is on reference-based reward systems for reasoning tasks, with benchmarks constructed from 41 open datasets and organized around four answer types: numeric values, algebraic expressions, multiple-choice selections, and free-form strings. SCI-VerifyBench differs in focusing specifically on scientific verification with explicit domain-specific equivalence transformations across mathematics, physics, chemistry, biology, and general scientific QA (Zheng et al., 29 Sep 2025, Yan et al., 21 May 2025).

A different strand of verifier evaluation appears in "Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoning" (Ficek et al., 19 Feb 2025), which transforms coding benchmarks into scoring and ranking datasets such as HE-R, HE-R+, MBPP-R, and MBPP-R+ to evaluate synthetic verifiers through metrics such as Top-1 Accuracy, Bottom-1 Accuracy, Spearman’s $a_i$4, Kendall’s $a_i$5, and MAE. Relative to that coding-oriented ranking framework, SCI-VerifyBench defines verification as equivalence judgment against a reference answer in scientific domains rather than scoring and ranking multiple code solutions (Ficek et al., 19 Feb 2025, Zheng et al., 29 Sep 2025).

SCI-VerifyBench was introduced together with SCI-Verifier, a unified reasoning-augmented verifier for scientific domains. The paper states that SCI-Verifier, through post-training, demonstrates strong logical reasoning and equivalence judgment capabilities while maintaining concise and stable outputs (Zheng et al., 29 Sep 2025). This pairing is consequential: the benchmark is not only an evaluation artifact but also part of a proposed framework for improving scientific verification systems.

The benchmark’s broader significance lies in its treatment of equivalence as a first-class scientific verification problem. Rather than assuming that correctness is adequately captured by string identity, SCI-VerifyBench formalizes scientific answer verification as a domain-sensitive inference problem over notation, units, symbolic form, and semantic paraphrase. This suggests a research direction in which verifier quality is measured by its ability to preserve scientific meaning across heterogeneous representations, not merely by its agreement with canonical answer strings.