SCI-Verifier: Scientific Answer Verification

• SCI-Verifier is a unified verification system that employs concise chain-of-thought reasoning to determine equivalence in free-form scientific answers.
• It leverages domain-specific transformations, including algebraic rearrangements and unit conversions, coupled with expert-annotated benchmarks to enhance accuracy.
• Empirical evaluations demonstrate state-of-the-art performance across disciplines such as mathematics, physics, chemistry, biology, and general scientific QA.

SCI-Verifier is a unified, reasoning-augmented verifier for scientific domains introduced together with SCI-VerifyBench, a cross-disciplinary benchmark for scientific answer verification. It is designed to determine whether an LLM’s free-form response is mathematically or semantically equivalent to a reference answer in settings where algebraic rearrangements, unit conversions, domain-specific synonyms, and proof-style, multi-step reasoning make direct matching unreliable. The system couples benchmark design with post-training, aiming to produce concise chain-of-thought analysis and robust equivalence judgment across mathematics, physics, chemistry, biology, and general scientific question answering (Zheng et al., 29 Sep 2025).

1. Verification task and problem setting

Scientific answer verification concerns deciding whether a generated response is correct relative to a reference when the two may differ substantially in surface form. SCI-Verifier is explicitly motivated by two difficulties: answers often admit many equivalent expressions, and proof-style, multi-step reasoning is intertwined with the final result. The associated work argues that existing verification studies in scientific domains are limited by insufficient disciplinary coverage and by heavy reliance on brittle rule design or prompt engineering, which weakens performance in complex reasoning scenarios and across disciplines (Zheng et al., 29 Sep 2025).

This problem setting is adjacent to, but distinct from, several earlier scientific verification formulations. In the biomedical domain, VERT5ERINI frames claim verification as a three-stage pipeline of abstract retrieval, sentence selection, and label prediction over scientific abstracts, with outputs in {Supports, NoInfo, Refutes} rather than free-form answer equivalence (Pradeep et al., 2020). In multimodal scientific literature, SciVer evaluates whether a claim is entailed or refuted given text paragraphs, tables, and figures from a single paper, and therefore targets multimodal evidence grounding rather than reference-answer equivalence (Wang et al., 18 Jun 2025).

2. SCI-VerifyBench: benchmark design and data construction

SCI-VerifyBench is the benchmark paired with SCI-Verifier. It spans five domains—mathematics, physics, chemistry, biology, and general scientific QA—and is built from real LLM outputs together with domain-specific equivalence transformations. The benchmark is described as cross-disciplinary and high-difficulty, with 2,000+ questions and 100 K+ LLM responses collected. Its purpose is to expose verifiers to realistic answer variability rather than only canonical answer forms (Zheng et al., 29 Sep 2025).

The construction process combines real QA and synthetic equivalence generation. The real-QA stage begins from 15 K+ seed questions drawn from open datasets such as OMINI-Math, PHYSICS, ChemBench, Bioinformatics, and SimpleQA, followed by generation of 100 K+ responses from 8 different LLMs under length control. The synthetic stage selects 500 representative questions per domain and transforms each answer into five challenging, domain-specific equivalent forms. The examples given include algebraic identities, Taylor expansions, unit conversions, SMILES $\leftrightarrow$ IUPAC $\leftrightarrow$ trivial names, and dot-bracket $\leftrightarrow$ CT-format for RNA. Multiple LLMs propose and filter these equivalences to simulate real-world confusable cases (Zheng et al., 29 Sep 2025).

Annotation proceeds in two tiers. First, five LLMs vote on correctness, and samples with unanimous votes are filtered out to focus on ambiguous cases. Second, 2,500 high-disagreement samples—500 per domain—are annotated by at least two domain experts, with a third expert resolving conflicts. The benchmark’s label format is $D=\{(q,a_{ref},r_{resp},l)\}$ with $l\in\{\text{true},\text{false}\}$. The test split contains 2,500 samples in total, composed of 350 real-QA and 150 synthetic-equivalence instances per domain. The training split contains 14 K remaining LLM-annotated samples, balanced and filtered for label reliability. Average response length is approximately 3 K tokens, whereas the average reference answer is approximately 25 tokens (Zheng et al., 29 Sep 2025).

A central design claim is that SCI-VerifyBench differs from VerifyBench, VerifyBench-Hard, and VerifierBench by spanning five disciplines, including explicit equivalence transformations, and applying difficulty control. This benchmark role is integral: the model and dataset are presented as a single framework for systematic evaluation and practical verifier development (Zheng et al., 29 Sep 2025).

3. Model architecture and post-training pipeline

SCI-Verifier is built from Qwen3-4B and Qwen3-8B base models and trained through a two-stage post-training pipeline consisting of supervised fine-tuning followed by reinforcement learning. The target behavior is short, structured chain-of-thought plus a final judgment. The work emphasizes brevity and stability rather than unrestricted long-form reasoning (Zheng et al., 29 Sep 2025).

In the supervised fine-tuning stage, high-quality chain-of-thought traces are collected from larger models by rejection sampling and then filtered for brevity and structure. The stated objective is

$$\mathcal{L}_{\mathrm{SFT}}(\theta)=-\mathbb{E}_{(x,y)\sim D_{\mathrm{SFT}}}\bigl[\log \pi_\theta(y\mid x)\bigr].$$

This stage is intended to transfer essential verification knowledge and chain-of-thought patterns into the smaller verifier model (Zheng et al., 29 Sep 2025).

After SFT warm-up, SCI-Verifier applies a refined GRPO method called DAPO. Training dynamically filters out too trivial or too hard samples and rebalances positive and negative examples to avoid bias. For each sampled reasoning trace $o_i$, the method defines $$r_{i,t}(\theta)=\pi_\theta(o_{i,\le t})/\pi_{\theta_{\mathrm{old}}}(o_{i,\le t})$$ and $\bar{A}_{i,t}=(R_i-\mu_R)/\sigma_R$, where $R_i=R_{\mathrm{align},i}+P_{\mathrm{overlong},i}$. The overlength penalty is piecewise:

$\leftrightarrow$0

The clipped PPO-style objective is

$\leftrightarrow$1

At inference time, SCI-Verifier uses a fixed chain-of-thought prompt to produce a brief reasoning chain followed by a categorical judgment: A for correct, B for incorrect, and C for incomplete or refusal (Zheng et al., 29 Sep 2025).

4. Equivalence judgment as learned reasoning

SCI-Verifier’s central technical claim is that verification quality depends on explicit reasoning about equivalence rather than on rule templates alone. The system is described as learning built-in patterns during SFT for algebraic identities, unit changes, chemical synonyms, and RNA structure formats. Representative examples include $\leftrightarrow$2, $\leftrightarrow$3, EtOH $\leftrightarrow$4 $\leftrightarrow$5, and dot-bracket $\leftrightarrow$6 BPseq. During inference, its chain-of-thought may test numeric substitutions at multiple points, apply factorization or expansion, or translate representations before concluding equivalence (Zheng et al., 29 Sep 2025).

The work argues that reasoning is critical to scientific verification. Across models, enabling chain-of-thought yields large gains, especially on equivalence-augmented cases, and SCI-Verifier is reported to be far less sensitive to prompt rewording than general LLMs because it is post-trained on fixed prompt formats. It also argues that model scale is only a moderate factor; domain-specific training and reasoning augmentation matter more. A related misconception addressed by the ablations is that longer reasoning traces are necessarily better: distilling full, long chain-of-thought hurts efficiency without improving accuracy, whereas short, targeted chain-of-thought is sufficient (Zheng et al., 29 Sep 2025).

A neighboring design appears in CoSineVerifier, which targets computation-oriented scientific questions through external executors such as a Python interpreter and a unit-conversion tool. That system reports state-of-the-art performance on VerifyBench-Hard and SCI-Bench and emphasizes algebraic equivalence checking and physical constant substitution through tool use rather than only internal reasoning (Feng et al., 1 Dec 2025). This suggests that SCI-Verifier and tool-augmented verifiers occupy closely related but not identical parts of the scientific-verification design space.

5. Empirical performance and ablation findings

SCI-Verifier is evaluated on SCI-VerifyBench, VerifierBench, and VerifyBench-Hard against closed-source models, open-source instruct and reasoning models, and specialized verifiers. On SCI-VerifyBench, GPT-5 reports an average accuracy of 84.92, specialized verifiers are reported at approximately 75–78 average, and the best open-source instruct or reasoning baselines at approximately 81–82 average. SCI-Verifier-4B reaches 85.40, while SCI-Verifier-8B reaches 86.28 and is described as overall state-of-the-art among open-source models (Zheng et al., 29 Sep 2025).

System	SCI-VerifyBench Accuracy	Selected comparative figures
GPT-5	84.92	Math 82.6, Physics 74.6, Chemistry 88.0, Biology 89.0, QA 90.4
SCI-Verifier-4B	85.40	Surpasses GPT-5 in several domains
SCI-Verifier-8B	86.28	Math 87.6, Physics 79.6, Chemistry 80.6, Biology 94.4, QA 89.2

On VerifierBench, SCI-Verifier-8B achieves 93.01 Accuracy and 93.06 F1, compared with GPT-5 at 91.80 and 90.48. On VerifyBench-Hard, SCI-Verifier-8B achieves 90.30 Accuracy and 87.45 F1, compared with GPT-5 at 90.40 and 85.34. The pattern is therefore not uniform across all metrics: GPT-5 is slightly higher on VerifyBench-Hard Accuracy, whereas SCI-Verifier-8B is higher on F1 (Zheng et al., 29 Sep 2025).

The domain-level analysis identifies mathematics and physics as the hardest domains because of subtle algebraic transformations, while chemistry, biology, and general QA are easier once domain facts are mastered. On transformed answers, performance can drop below 50% for GPT-5 but remains above 70–80% for SCI-Verifier. This is presented as evidence that explicit reasoning and benchmark-specific training are especially important on equivalence-augmented cases (Zheng et al., 29 Sep 2025).

Ablation studies reinforce the two-stage post-training design. Removing either SFT or RL degrades performance, and training on SCI-Verifier’s curated data outperforms using a generic large reward-model dataset, especially under RL. Inference without chain-of-thought is faster but produces a substantial accuracy drop of approximately 5–8%. These findings support the paper’s claim that concise reasoning is not merely optional prompting but part of the verifier’s learned competence (Zheng et al., 29 Sep 2025).

6. Relation to adjacent verification paradigms and open problems

SCI-Verifier belongs to a broader scientific-verification landscape that includes several task formulations. SciVer evaluates multimodal scientific claim verification using 3,000 expert-annotated examples over 1,113 papers, with text, tables, and figures, and reports a substantial gap between foundation models and human experts, whose average test accuracy is 93.8. Its error analysis identifies failure to retrieve relevant information, visual element misinterpretation, and failure in multi-step reasoning as major bottlenecks (Wang et al., 18 Jun 2025). VERT5ERINI, by contrast, is a corpus-grounded biomedical verifier built on T5-3B that performs abstract retrieval, sentence selection, and three-way label prediction on SciFact, achieving 64.9 label-only F1 and 62.7 label-plus-rationale F1 on the SciFact test set (Pradeep et al., 2020).

Other verification infrastructures target different units of analysis. sciwrite-lint is a local manuscript-verification pipeline that checks whether references exist, inspects retraction status, compares metadata against canonical records, downloads and parses cited papers, verifies whether they support cited claims, and assigns a per-reference reliability score. Its integrity component is evaluated on 30 unseen papers from arXiv and bioRxiv with error injection, and its final score can incorporate an experimental contribution component based on five frameworks from philosophy of science (Samsonau, 9 Apr 2026). AutoVerifier operates at the level of technical-intelligence analysis, decomposing assertions into $\leftrightarrow$7 claim triples and propagating them through six layers that include corpus ingestion, intra-document verification, cross-source verification, external signal corroboration, and hypothesis-matrix generation (Du et al., 3 Apr 2026).

Within that landscape, SCI-Verifier is specifically an answer verifier: it judges equivalence between a candidate response and a reference answer rather than grounding claims against an external corpus or auditing a complete manuscript. This suggests that its main contribution is to formalize scientific answer verification as a cross-disciplinary, equivalence-centric problem with standardized evaluation and a reasoning-oriented model.

The limitations stated for SCI-Verifier remain consequential. Inference still requires chain-of-thought, which is costly for ultra-low-latency use, and some highly exotic equivalence forms, such as advanced asymptotics, may remain challenging. The proposed future directions are to distill explicit reasoning into implicit verification capabilities so that chain-of-thought may be omitted at inference, extend the framework to multimodal scientific verification such as proofs involving diagrams or chemical reaction schemes, and incorporate formal methods such as symbolic solvers within the reasoning loop (Zheng et al., 29 Sep 2025).