SciFact: Benchmark for Scientific Claim Verification

• SciFact is an expert-annotated benchmark for scientific claim verification, defining claims as verifiable scientific assertions extracted from research literature.
• It evaluates systems on abstract retrieval, label prediction, and evidence rationale selection with metrics like F1 scores and Cohen’s kappa.
• Extensions of SciFact include open-domain retrieval over 500K abstracts and adaptations for diagnostic studies, claim correction, and evidence-aware reranking.

SciFact is an expert-annotated benchmark and task for scientific claim verification in which a system is asked to retrieve research abstracts, determine whether each abstract Supports, Refutes, or provides NoInfo for a claim, and identify the sentence-level rationales that justify that decision (Wadden et al., 2020). The original dataset contains 1,409 scientific claims and a corpus of 5,183 abstracts, and later work has extended its use from closed-domain verification to open-domain retrieval over 500K abstracts, retrieval benchmarking in BEIR, scientific RAG verification, and diagnostic studies of evidence quality and dataset artifacts (Wadden et al., 2020, Wadden et al., 2022, Dey et al., 23 Jun 2026, Kumar, 11 Jun 2026).

1. Origins and dataset construction

SciFact was introduced as a benchmark for what its authors called scientific claim verification, with the explicit goal of selecting abstracts from the research literature that contain evidence supporting or refuting a claim and identifying the rationale sentences that justify the label (Wadden et al., 2020). A SciFact claim is defined as an atomic verifiable statement expressing a finding about one aspect of a scientific entity or process, verifiable from a single source, and the dataset deliberately avoids treating the task as assigning a single global truth value to a claim across the whole literature (Wadden et al., 2020).

The dataset was built from citation sentences. Annotators were shown a citance in context and asked to write up to three claims based on it, without seeing the cited abstract during claim writing, so that the resulting claims would be less trivially tied to eventual evidence wording (Wadden et al., 2020). The source corpus came from S2ORC, filtered to a curated set of respected journals across basic science and clinical medicine, and later expanded with distractor abstracts so that abstract retrieval would be nontrivial (Wadden et al., 2020).

The released SciFact dataset contains 1,409 claims with label counts 556 Supports, 516 NoInfo, and 337 Refutes (Wadden et al., 2020). Its train/dev/test splits are 809 / 300 / 300 claims, with the test set balanced at 100 Supports / 100 NoInfo / 100 Refutes (Wadden et al., 2020). Evidence structure is deliberately sparse but nontrivial: 1,278 claims have 1 cited abstract, 830 claims have 1 evidence abstract, and most rationales are single-sentence, with 1,542 one-sentence rationales, 92 two-sentence rationales, and 11 three-sentence rationales (Wadden et al., 2020).

Annotation quality was measured on 232 claim–abstract pairs, yielding Cohen’s $\kappa = 0.75$ for labels and Cohen’s $\kappa = 0.71$ for rationale agreement (Wadden et al., 2020). The original paper presented these values as strong agreement given the scientific domain, and they remain one reason SciFact became a widely reused evidence-grounded benchmark.

2. Task formulation and evaluation protocol

In SciFact, the basic decision unit is the claim–abstract pair. For a claim $c$ and abstract $a$, the benchmark assigns a label

$$y(c,a)\in\{\text{Supports},\text{Refutes},\text{NoInfo}\}$$

and defines the evidence-bearing abstracts $E(c)$ together with rationale sets $R(c,a)$ for each evidentiary abstract (Wadden et al., 2020). Rationales are defined as a minimal collection of sentences which, taken together in the context of the abstract, allow a domain expert to infer the claim (Wadden et al., 2020).

The benchmark supports both an Open setting, in which the system must retrieve evidence abstracts from the corpus, and an Oracle abstract setting, in which the gold evidence abstracts are given (Wadden et al., 2020). Evaluation is strict at both the abstract and sentence levels. In the abstract-level formulation reproduced by later SciFact systems, an abstract is correctly labelled only if it is a true evidence abstract and the predicted label matches the gold label; it is correctly rationalized only if the predicted rationale sentences contain a gold rationale $R\subseteq \hat S(q,a)$ (Pradeep et al., 2020). At the sentence level, a predicted sentence counts only if it belongs to a gold rationale and the full gold rationale is covered by the prediction (Wadden et al., 2020, Pradeep et al., 2020).

This evaluation design is one of SciFact’s defining features. It makes the benchmark more than a three-way NLI dataset: retrieval, evidence sufficiency, and label prediction are all part of the task. Later work repeatedly returned to this point, either by preserving the evidence-centered structure or by diagnosing what is lost when SciFact is simplified to claim–evidence classification only (Ljajić et al., 2024, Dey et al., 23 Jun 2026).

3. Core benchmark systems and architectural evolution

The original baseline, VeriSci, used a three-stage pipeline with TF-IDF abstract retrieval, sentence-level rationale selection, and label prediction from selected rationales (Wadden et al., 2020). In the open setting on the SciFact test set, VeriSci reached 46.5 abstract-level Label+Rationale F1 and 39.5 sentence-level Selection+Label F1 (Wadden et al., 2020). The same paper also showed that simple domain adaptation materially improved performance relative to models trained only on FEVER or UKP Snopes, particularly for label prediction (Wadden et al., 2020).

A first major line of follow-up work replaced sentence-by-sentence encoding with paragraph- or abstract-level encoders. The paragraph-level multi-task model of Li et al. encoded the claim and full abstract as one BERT sequence, jointly optimized rationale selection and stance prediction, and reported open-setting scores of 59.9 abstract Label+Rationale F1 and 55.2 sentence Selection+Label F1 for Paragraph-Joint (Li et al., 2020). The paper’s central claim was that contextualized sentence representations and joint learning reduced error propagation relative to pipeline systems (Li et al., 2020).

A second line of work pursued end-to-end joint learning across more of the SciFact pipeline. ARSJoint jointly modeled abstract retrieval, rationale selection, and stance prediction, and added a rationale regularization term linking retrieval attention to rationale predictions (Zhang et al., 2021). On the SciFact development set, ARSJoint (BioBERT) reported 62.4 abstract Label+Rationale F1 and 57.8 sentence Selection+Label F1, improving over earlier partially joint models (Zhang et al., 2021).

A third line adopted pretrained text-to-text models. VerT5erini used T5 for abstract retrieval, sentence selection, and label prediction, with BM25 candidate generation followed by T5 reranking (Pradeep et al., 2020). On the hidden SciFact test set, VerT5erini (T5) achieved 64.9 abstract Label-Only F1, 62.7 abstract Label+Rationale F1, 64.8 sentence Selection-Only F1, and 58.8 sentence Selection+Label F1, substantially improving on VeriSci (Pradeep et al., 2020). The same paper also showed that T5 reranking raised abstract retrieval from 79.90 / 84.69 $R@3 / R@5$ with BM25 to 86.12 / 89.95 with a T5 reranker initialized from MS MARCO (Pradeep et al., 2020).

Subsequent systems explored alternative decompositions. RerrFact used reduced abstract representations $\{\text{title}, s_1, s_{n/2}, s_n\}$, trained rationale selection on retrieved abstracts rather than oracle abstracts, and decomposed stance into NoInfo vs enough-information followed by Supports vs Refutes (Rana et al., 2022). On the SciFact test set, it reported 66.31 abstract Label+Rationale F1 and 62.09 sentence Selection+Label F1, ranking 4th on the SciFact leaderboard at the time of publication (Rana et al., 2022). BEVERS, by contrast, transferred a FEVER-style pipeline to SciFact and reported the best Abstract LO score in its comparison table, 73.2 on the blind test set, while remaining weaker on the full evidence-sensitive SS + L metric at 58.1 (DeHaven et al., 2023).

Taken together, these systems established several recurring design themes: claim-aware encoding, evidence-conditioned stance prediction, stronger retrieval, and tighter coupling between evidence identification and label prediction (Li et al., 2020, Zhang et al., 2021, Pradeep et al., 2020).

4. Open-domain extensions and retrieval-centric reformulations

SciFact-Open extended the benchmark from the original $\kappa = 0.71$0K-abstract setting to 500K research abstracts, using 279 claims derived from the original SciFact test set and pooling annotations from four strong systems (Wadden et al., 2022). It retained the original 209 SciFact evidence claim–abstract pairs and added 251 new pooled evidentiary pairs, for 460 total ECAPs (Wadden et al., 2022). The central empirical finding was that systems developed on the small original corpus suffered performance drops of at least 15 F1 in the open-domain setting, with examples including 72.5 $\kappa = 0.71$1 52.4 for MultiVerS and 68.2 $\kappa = 0.71$2 36.4 for VerT5erini (Wadden et al., 2022).

SciFact-Open also exposed phenomena that the closed benchmark underemphasized. In a subset of 206 claim–abstract pairs with extra analysis, 91 showed some specificity mismatch between claim and evidence, and among the 81 claims with at least two evidentiary abstracts, 16 had conflicting evidence (Wadden et al., 2022). This strengthened the view that open-domain scientific verification is as much an evidence-retrieval and evidence-aggregation problem as a stance-classification problem.

A parallel line of work used SciFact in the BEIR retrieval formulation, where the task is to rank documents for a query and score with metrics such as NDCG@10. In vstash, SciFact appears as one of the five BEIR datasets used to evaluate hybrid dense–sparse retrieval, adaptive reciprocal-rank fusion, and self-supervised embedding refinement (Steffens, 16 Apr 2026). For SciFact specifically, the paper reports 5,183 biomedical documents, 300 human-annotated queries, and an adaptive RRF score of 0.7263 NDCG@10 with BGE-small; a tuned hybrid pipeline reaches 0.6945, slightly above the paper’s cited 0.693 ColBERTv2 value under different preprocessing (Steffens, 16 Apr 2026). The same study found a 63.4% top-10 disagreement rate on SciFact queries between vector-heavy and FTS-heavy retrieval, using that disagreement as self-supervised training signal (Steffens, 16 Apr 2026).

Other work made the retrieval objective more explicitly evidence-aware. +VeriRel argued that document retrieval for scientific fact checking should optimize verification usefulness, not only semantic relevance, and used verifier feedback to rerank documents (Deng et al., 14 Aug 2025). On SciFact, +VeriRel $\kappa = 0.71$3 improved document evidence recall over monoT5-3B from 55.02 to 62.20 at $\kappa = 0.71$4, from 78.47 to 82.30 at $\kappa = 0.71$5, and from 85.65 to 87.08 at $\kappa = 0.71$6; when fed into MultiVerS, it raised SciFact(offline) top-3 verification F1 from 65.57 to 70.00 (Deng et al., 14 Aug 2025).

SciFact has also become a target for query-expansion research. A 2026 study on automated in-domain exemplar construction and multi-LLM query expansion reports that on SciFact, Two-LLM QE (Refine) reaches 72.07 NDCG@10, compared with 67.89 for BM25 and 69.69 for the paper’s single-LLM in-domain cluster baseline (Li et al., 9 Feb 2026). This reframing treats SciFact as a compact biomedical retrieval benchmark rather than only a claim-verification dataset.

5. SciFact as supervision for verification, RAG, and correction

SciFact is now widely used as a supervised resource inside larger systems rather than only as a standalone benchmark. In Verif.ai, SciFact is the direct training and evaluation dataset for a post-generation scientific verification engine: the system retrieves PubMed abstracts, generates referenced answers with a fine-tuned Mistral 7B model, then verifies each generated statement against its cited abstract using SciFact-style three-way document inference (Ljajić et al., 2024). The deployed verifier is a SciFact-fine-tuned DeBERTa-v3-large model; on SciFact label prediction it achieved weighted-average precision 0.88, recall 0.88, and F1 0.88, compared with precision 0.81, recall 0.80, and F1 0.79 for GPT-4 in the paper’s zero-shot baseline (Ljajić et al., 2024).

SciFact also supports data-augmentation studies. A 2025 paper on LLM-generated synthetic data simplified SciFact to binary support-versus-contradict verification, generated synthetic claim–text pairs from the training abstracts, and fine-tuned a SciBERT classifier (Zhang et al., 28 Aug 2025). In the printed results table, the SciFact baseline is precision 0.716, recall 0.769, F1 0.741, while the best augmented setting at 100% sentence selection reaches precision 0.666, recall 0.979, and F1 0.792 (Zhang et al., 28 Aug 2025).

Few-shot work has used SciFact to study low-resource scientific verification. MAPLE evaluated both SciFact_oracle and SciFact_retrieved, the latter using BM25 top-3 abstracts as evidence (Zeng et al., 2024). On SciFact_retrieved, MAPLE reached 0.4846 macro-F1 at 5-shot, compared with 0.3726 for SEED and 0.3467 for the paper’s main PET baseline (Zeng et al., 2024). The paper’s argument was that unlabeled in-domain claim–evidence pairs can be exploited through pairwise language-evolution features without depending on large NLI-tuned models (Zeng et al., 2024).

Several systems have repurposed SciFact for claim correction rather than only verification. SciFix turns supported scientific claims into synthetic incorrect–correct pairs using GPT-3.5-generated corruptions, explanations, and augmented correct claims, then trains a domain-adapted T5-base correction model with claim-aware decoding (Ashok et al., 2023). On SciFact with gold evidence, the paper reports human-evaluated correction accuracy of 84.73 for SciFix All and 87.5 for SciFix Bio, compared with 58.74 for few-shot GPT-3.5 and single-digit scores for earlier verifier-guided correction baselines (Ashok et al., 2023). In a complementary direction, Zero-shot Faithful Factual Error Correction repurposed SciFact into an evidence-grounded correction benchmark and reported that its biomedical-adapted variant ZEROFEC-DA achieved human-evaluated faithfulness 0.628 on SciFact, slightly above 0.610 for the paper’s supervised T5-FULL baseline (Huang et al., 2023).

6. Limitations, artifacts, and changing evaluation norms

Several later papers have argued that high SciFact scores can be misleading if evaluation focuses too narrowly on closed-domain label prediction. The most direct critique comes from a 2026 study of small LLMs for biomedical claim verification, which identifies a previously unreported structural artifact in the common SciFact claim–evidence formulation: in the authors’ SciFact split, all 243 NEI training examples have empty evidence fields, while every supports and refutes example contains evidence (Kumar, 11 Jun 2026). In-domain fine-tuned models therefore achieve 100.0 NEI F1 on SciFact, but collapse on HealthVer when the NEI class contains non-empty but inconclusive evidence; for example, BioLinkBERT trained on SciFact drops to 0.3 NEI F1 on HealthVer, and Mistral-7B trained on SciFact reaches only 20.1 NEI F1 there (Kumar, 11 Jun 2026). The paper’s conclusion is that dataset structure can matter more than data quantity for cross-domain generalization (Kumar, 11 Jun 2026).

A second critique concerns the difference between verdict correctness and warrant admissibility. The 2026 paper on the warrant gap argues that systems can predict SUPPORTS correctly while citing evidence that does not actually license the full claim (Dey et al., 23 Jun 2026). Its metric, Warranted Supports Proportion (WSP),

$\kappa = 0.71$7

measures the fraction of predicted supports whose cited warrant entails the claim (Dey et al., 23 Jun 2026). Using SciFact’s human-curated evidence, the paper reports that DeBERTa-WSP on SciFact improves from AUC 0.775 and precision 0.972 at the direct stage to AUC 0.896 and precision 0.988 after SIFT claim-conditioned re-scoring (Dey et al., 23 Jun 2026). This reframes SciFact as a test not only of classification, but of whether cited scientific evidence truly justifies the verdict.

Open-domain evaluation introduces further caveats. SciFact-Open adopts IR-style pooling because exhaustive annotation over 500K abstracts is infeasible, and the authors show that evaluation stability depends on both pool depth and the number of contributing systems (Wadden et al., 2022). They also report an inclusion bias effect: when a pooled system’s own predictions are excluded from the collection, relative F1 changes by about 15% for several models (Wadden et al., 2022). This does not invalidate the benchmark, but it means that new systems are best compared against the excluded performance of pooled baselines rather than their included scores (Wadden et al., 2022).

These critiques have shifted how SciFact is interpreted. The benchmark remains central for evidence-grounded scientific verification, but later work increasingly treats it as one component in a broader evaluation regime involving open-domain retrieval, cross-domain transfer, evidence-aware reranking, hallucination detection, and warrant-sensitive metrics (Wadden et al., 2022, Deng et al., 14 Aug 2025, Dey et al., 23 Jun 2026, Kumar, 11 Jun 2026). This suggests that SciFact’s long-term value lies not only in its original scores, but also in the way it has exposed the interaction between retrieval, evidence sufficiency, scientific reasoning, and benchmark design.