---
title: 'SciFact: Benchmark for Scientific Claim Verification'
url: https://www.emergentmind.com/topics/scifact
type: topic
---

# SciFact: Benchmark for Scientific Claim Verification

SciFact is an expert-annotated benchmark and task for **scientific claim verification** in which a system is asked to retrieve research abstracts, determine whether each abstract **Supports**, **Refutes**, or provides **NoInfo** for a claim, and identify the sentence-level rationales that justify that decision [2004.14974]. The original dataset contains **1,409 scientific claims** and a corpus of **5,183 abstracts**, and later work has extended its use from closed-domain verification to open-domain retrieval over **500K abstracts**, retrieval benchmarking in BEIR, scientific RAG verification, and diagnostic studies of evidence quality and dataset artifacts [2004.14974] [2210.13777] [2606.24627] [2606.12854].

## 1. Origins and dataset construction

SciFact was introduced as a benchmark for what its authors called **scientific claim verification**, with the explicit goal of selecting abstracts from the research literature that contain evidence supporting or refuting a claim and identifying the rationale sentences that justify the label [2004.14974]. A SciFact claim is defined as an **atomic verifiable statement expressing a finding about one aspect of a scientific entity or process, verifiable from a single source**, and the dataset deliberately avoids treating the task as assigning a single global truth value to a claim across the whole literature [2004.14974].

The dataset was built from **citation sentences**. Annotators were shown a citance in context and asked to write up to **three claims** based on it, without seeing the cited abstract during claim writing, so that the resulting claims would be less trivially tied to eventual evidence wording [2004.14974]. The source corpus came from **S2ORC**, filtered to a curated set of respected journals across basic science and clinical medicine, and later expanded with distractor abstracts so that abstract retrieval would be nontrivial [2004.14974].

The released SciFact dataset contains **1,409 claims** with label counts **556 Supports**, **516 NoInfo**, and **337 Refutes** [2004.14974]. Its train/dev/test splits are **809 / 300 / 300** claims, with the test set balanced at **100 Supports / 100 NoInfo / 100 Refutes** [2004.14974]. Evidence structure is deliberately sparse but nontrivial: **1,278 claims** have **1 cited abstract**, **830 claims** have **1 evidence abstract**, and most rationales are single-sentence, with **1,542** one-sentence rationales, **92** two-sentence rationales, and **11** three-sentence rationales [2004.14974].

Annotation quality was measured on **232 claim–abstract pairs**, yielding **Cohen’s \(\kappa = 0.75\)** for labels and **Cohen’s \(\kappa = 0.71\)** for rationale agreement [2004.14974]. The original paper presented these values as strong agreement given the scientific domain, and they remain one reason SciFact became a widely reused evidence-grounded benchmark.

## 2. Task formulation and evaluation protocol

In SciFact, the basic decision unit is the **claim–abstract pair**. For a claim \(c\) and abstract \(a\), the benchmark assigns a label
\[
y(c,a)\in\{\text{Supports},\text{Refutes},\text{NoInfo}\}
\]
and defines the evidence-bearing abstracts \(E(c)\) together with rationale sets \(R(c,a)\) for each evidentiary abstract [2004.14974]. Rationales are defined as a **minimal collection of sentences** which, taken together in the context of the abstract, allow a domain expert to infer the claim [2004.14974].

The benchmark supports both an **Open** setting, in which the system must retrieve evidence abstracts from the corpus, and an **Oracle abstract** setting, in which the gold evidence abstracts are given [2004.14974]. Evaluation is strict at both the abstract and sentence levels. In the abstract-level formulation reproduced by later SciFact systems, an abstract is correctly labelled only if it is a true evidence abstract and the predicted label matches the gold label; it is correctly rationalized only if the predicted rationale sentences contain a gold rationale \(R\subseteq \hat S(q,a)\) [2010.11930]. At the sentence level, a predicted sentence counts only if it belongs to a gold rationale and the full gold rationale is covered by the prediction [2004.14974] [2010.11930].

This evaluation design is one of SciFact’s defining features. It makes the benchmark more than a three-way NLI dataset: retrieval, evidence sufficiency, and label prediction are all part of the task. Later work repeatedly returned to this point, either by preserving the evidence-centered structure or by diagnosing what is lost when SciFact is simplified to claim–evidence classification only [2407.11485] [2606.24627].

## 3. Core benchmark systems and architectural evolution

The original baseline, **VeriSci**, used a three-stage pipeline with **TF-IDF** abstract retrieval, sentence-level rationale selection, and label prediction from selected rationales [2004.14974]. In the open setting on the SciFact test set, VeriSci reached **46.5** abstract-level **Label+Rationale** F1 and **39.5** sentence-level **Selection+Label** F1 [2004.14974]. The same paper also showed that simple domain adaptation materially improved performance relative to models trained only on FEVER or UKP Snopes, particularly for label prediction [2004.14974].

A first major line of follow-up work replaced sentence-by-sentence encoding with paragraph- or abstract-level encoders. The paragraph-level multi-task model of Li et al. encoded the claim and full abstract as one BERT sequence, jointly optimized rationale selection and stance prediction, and reported open-setting scores of **59.9** abstract **Label+Rationale** F1 and **55.2** sentence **Selection+Label** F1 for **Paragraph-Joint** [2012.14500]. The paper’s central claim was that contextualized sentence representations and joint learning reduced error propagation relative to pipeline systems [2012.14500].

A second line of work pursued end-to-end joint learning across more of the SciFact pipeline. **ARSJoint** jointly modeled abstract retrieval, rationale selection, and stance prediction, and added a **rationale regularization** term linking retrieval attention to rationale predictions [2110.15116]. On the SciFact development set, **ARSJoint (BioBERT)** reported **62.4** abstract **Label+Rationale** F1 and **57.8** sentence **Selection+Label** F1, improving over earlier partially joint models [2110.15116].

A third line adopted pretrained text-to-text models. **VerT5erini** used **T5** for abstract retrieval, sentence selection, and label prediction, with BM25 candidate generation followed by T5 reranking [2010.11930]. On the hidden SciFact test set, **VerT5erini (T5)** achieved **64.9** abstract **Label-Only** F1, **62.7** abstract **Label+Rationale** F1, **64.8** sentence **Selection-Only** F1, and **58.8** sentence **Selection+Label** F1, substantially improving on VeriSci [2010.11930]. The same paper also showed that T5 reranking raised abstract retrieval from **79.90 / 84.69** \(R@3 / R@5\) with BM25 to **86.12 / 89.95** with a T5 reranker initialized from MS MARCO [2010.11930].

Subsequent systems explored alternative decompositions. **RerrFact** used **reduced abstract representations** \(\{\text{title}, s_1, s_{n/2}, s_n\}\), trained rationale selection on retrieved abstracts rather than oracle abstracts, and decomposed stance into **NoInfo vs enough-information** followed by **Supports vs Refutes** [2202.02646]. On the SciFact test set, it reported **66.31** abstract **Label+Rationale** F1 and **62.09** sentence **Selection+Label** F1, ranking **4th on the SciFact leaderboard** at the time of publication [2202.02646]. **BEVERS**, by contrast, transferred a FEVER-style pipeline to SciFact and reported the best **Abstract LO** score in its comparison table, **73.2** on the blind test set, while remaining weaker on the full evidence-sensitive **SS + L** metric at **58.1** [2303.16974].

Taken together, these systems established several recurring design themes: claim-aware encoding, evidence-conditioned stance prediction, stronger retrieval, and tighter coupling between evidence identification and label prediction [2012.14500] [2110.15116] [2010.11930].

## 4. Open-domain extensions and retrieval-centric reformulations

**SciFact-Open** extended the benchmark from the original \(\sim 5\)K-abstract setting to **500K research abstracts**, using **279** claims derived from the original SciFact test set and pooling annotations from four strong systems [2210.13777]. It retained the original **209** SciFact evidence claim–abstract pairs and added **251** new pooled evidentiary pairs, for **460** total ECAPs [2210.13777]. The central empirical finding was that systems developed on the small original corpus suffered performance drops of **at least 15 F1** in the open-domain setting, with examples including **72.5 \(\rightarrow\) 52.4** for MultiVerS and **68.2 \(\rightarrow\) 36.4** for VerT5erini [2210.13777].

SciFact-Open also exposed phenomena that the closed benchmark underemphasized. In a subset of **206** claim–abstract pairs with extra analysis, **91** showed some specificity mismatch between claim and evidence, and among the **81 claims** with at least two evidentiary abstracts, **16** had conflicting evidence [2210.13777]. This strengthened the view that open-domain scientific verification is as much an evidence-retrieval and evidence-aggregation problem as a stance-classification problem.

A parallel line of work used SciFact in the **BEIR retrieval formulation**, where the task is to rank documents for a query and score with metrics such as **NDCG@10**. In **vstash**, SciFact appears as one of the five BEIR datasets used to evaluate hybrid dense–sparse retrieval, adaptive reciprocal-rank fusion, and self-supervised embedding refinement [2604.15484]. For SciFact specifically, the paper reports **5,183 biomedical documents**, **300 human-annotated queries**, and an **adaptive RRF** score of **0.7263 NDCG@10** with BGE-small; a tuned hybrid pipeline reaches **0.6945**, slightly above the paper’s cited **0.693** ColBERTv2 value under different preprocessing [2604.15484]. The same study found a **63.4%** top-10 disagreement rate on SciFact queries between vector-heavy and FTS-heavy retrieval, using that disagreement as self-supervised training signal [2604.15484].

Other work made the retrieval objective more explicitly evidence-aware. **+VeriRel** argued that document retrieval for scientific fact checking should optimize **verification usefulness**, not only semantic relevance, and used verifier feedback to rerank documents [2508.11122]. On SciFact, **+VeriRel \(N=5\)** improved document evidence recall over monoT5-3B from **55.02** to **62.20** at \(R@1\), from **78.47** to **82.30** at \(R@5\), and from **85.65** to **87.08** at \(R@10\); when fed into MultiVerS, it raised SciFact(offline) top-3 verification F1 from **65.57** to **70.00** [2508.11122].

SciFact has also become a target for query-expansion research. A 2026 study on automated in-domain exemplar construction and multi-LLM query expansion reports that on SciFact, **Two-LLM QE (Refine)** reaches **72.07 NDCG@10**, compared with **67.89** for BM25 and **69.69** for the paper’s single-LLM in-domain cluster baseline [2602.08917]. This reframing treats SciFact as a compact biomedical retrieval benchmark rather than only a claim-verification dataset.

## 5. SciFact as supervision for verification, RAG, and correction

SciFact is now widely used as a **supervised resource** inside larger systems rather than only as a standalone benchmark. In **Verif.ai**, SciFact is the direct training and evaluation dataset for a post-generation scientific verification engine: the system retrieves PubMed abstracts, generates referenced answers with a fine-tuned **Mistral 7B** model, then verifies each generated statement against its cited abstract using SciFact-style three-way document inference [2407.11485]. The deployed verifier is a SciFact-fine-tuned **DeBERTa-v3-large** model; on SciFact label prediction it achieved weighted-average **precision 0.88**, **recall 0.88**, and **F1 0.88**, compared with **precision 0.81**, **recall 0.80**, and **F1 0.79** for GPT-4 in the paper’s zero-shot baseline [2407.11485].

SciFact also supports data-augmentation studies. A 2025 paper on **LLM-generated synthetic data** simplified SciFact to binary support-versus-contradict verification, generated synthetic claim–text pairs from the training abstracts, and fine-tuned a **SciBERT** classifier [2508.20525]. In the printed results table, the SciFact baseline is **precision 0.716**, **recall 0.769**, **F1 0.741**, while the best augmented setting at **100%** sentence selection reaches **precision 0.666**, **recall 0.979**, and **F1 0.792** [2508.20525].

Few-shot work has used SciFact to study low-resource scientific verification. **MAPLE** evaluated both **SciFact_oracle** and **SciFact_retrieved**, the latter using **BM25** top-3 abstracts as evidence [2401.16282]. On **SciFact_retrieved**, MAPLE reached **0.4846** macro-F1 at 5-shot, compared with **0.3726** for SEED and **0.3467** for the paper’s main PET baseline [2401.16282]. The paper’s argument was that unlabeled in-domain claim–evidence pairs can be exploited through pairwise language-evolution features without depending on large NLI-tuned models [2401.16282].

Several systems have repurposed SciFact for **claim correction** rather than only verification. **SciFix** turns supported scientific claims into synthetic incorrect–correct pairs using GPT-3.5-generated corruptions, explanations, and augmented correct claims, then trains a domain-adapted **T5-base** correction model with claim-aware decoding [2305.14707]. On SciFact with gold evidence, the paper reports human-evaluated correction accuracy of **84.73** for **SciFix All** and **87.5** for **SciFix Bio**, compared with **58.74** for few-shot GPT-3.5 and single-digit scores for earlier verifier-guided correction baselines [2305.14707]. In a complementary direction, **Zero-shot Faithful Factual Error Correction** repurposed SciFact into an evidence-grounded correction benchmark and reported that its biomedical-adapted variant **ZEROFEC-DA** achieved human-evaluated **faithfulness 0.628** on SciFact, slightly above **0.610** for the paper’s supervised T5-FULL baseline [2305.07982].

## 6. Limitations, artifacts, and changing evaluation norms

Several later papers have argued that high SciFact scores can be misleading if evaluation focuses too narrowly on closed-domain label prediction. The most direct critique comes from a 2026 study of small LLMs for biomedical claim verification, which identifies a **previously unreported structural artifact** in the common SciFact claim–evidence formulation: in the authors’ SciFact split, **all 243 NEI training examples have empty evidence fields**, while every supports and refutes example contains evidence [2606.12854]. In-domain fine-tuned models therefore achieve **100.0** NEI F1 on SciFact, but collapse on HealthVer when the NEI class contains non-empty but inconclusive evidence; for example, **BioLinkBERT** trained on SciFact drops to **0.3** NEI F1 on HealthVer, and **Mistral-7B** trained on SciFact reaches only **20.1** NEI F1 there [2606.12854]. The paper’s conclusion is that dataset structure can matter more than data quantity for cross-domain generalization [2606.12854].

A second critique concerns the difference between **verdict correctness** and **warrant admissibility**. The 2026 paper on **the warrant gap** argues that systems can predict **SUPPORTS** correctly while citing evidence that does not actually license the full claim [2606.24627]. Its metric, **Warranted Supports Proportion (WSP)**,
\[
\mathrm{WSP}=\frac{|\{i\in S : \mathrm{NLI}(w_i,c_i)=\mathrm{ENTAIL}\}|}{|S|}
\]
measures the fraction of predicted supports whose cited warrant entails the claim [2606.24627]. Using SciFact’s human-curated evidence, the paper reports that **DeBERTa-WSP** on SciFact improves from **AUC 0.775** and **precision 0.972** at the direct stage to **AUC 0.896** and **precision 0.988** after **SIFT** claim-conditioned re-scoring [2606.24627]. This reframes SciFact as a test not only of classification, but of whether cited scientific evidence truly justifies the verdict.

Open-domain evaluation introduces further caveats. SciFact-Open adopts IR-style pooling because exhaustive annotation over **500K** abstracts is infeasible, and the authors show that evaluation stability depends on both pool depth and the number of contributing systems [2210.13777]. They also report an **inclusion bias** effect: when a pooled system’s own predictions are excluded from the collection, relative F1 changes by about **15%** for several models [2210.13777]. This does not invalidate the benchmark, but it means that new systems are best compared against the **excluded** performance of pooled baselines rather than their included scores [2210.13777].

These critiques have shifted how SciFact is interpreted. The benchmark remains central for evidence-grounded scientific verification, but later work increasingly treats it as one component in a broader evaluation regime involving open-domain retrieval, cross-domain transfer, evidence-aware reranking, hallucination detection, and warrant-sensitive metrics [2210.13777] [2508.11122] [2606.24627] [2606.12854]. This suggests that SciFact’s long-term value lies not only in its original scores, but also in the way it has exposed the interaction between retrieval, evidence sufficiency, scientific reasoning, and benchmark design.

Source: https://www.emergentmind.com/topics/scifact