--- title: 'FEVER Dataset: Fact Verification Benchmark' url: https://www.emergentmind.com/topics/fever-dataset type: topic --- # FEVER Dataset: Fact Verification Benchmark The FEVER (Fact Extraction and VERification) dataset is a large-scale, publicly available benchmark designed to advance the development and evaluation of automated systems for the verification of factual claims against textual evidence drawn from Wikipedia. FEVER establishes a precise task formulation centered on verifying whether a given natural-language claim can be supported, refuted, or remains unanswerable based on explicit evidence. By providing a rigorously annotated corpus of 185,445 human-generated claims, extensive evidence annotations, and standardized evaluation metrics, FEVER serves as a foundational resource for research in natural language inference, fact verification, interpretable entailment, and multi-hop retrieval. ## 1. Dataset Construction and Annotation Protocol ### Claim Generation Claims in FEVER are derived through a multi-stage, human-centered protocol. Annotators begin with facts extracted from the introductory sections of sampled Wikipedia pages (∼50,000 from a June 2017 dump, processed via CoreNLP) [1803.05355]. Each source sentence is either paraphrased or mutated according to defined operators: negation, entity swapping with semantically similar or dissimilar mentions, and adjustment of specificity (either entailment-preserving or reversing). These controlled mutations yield diverse claims whose veracity is ambiguous a priori and may require complex reasoning to adjudicate. ### Claim Categorization and Evidence Annotation Each claim is independently labeled as one of **Supported**, **Refuted**, or **NotEnoughInfo (NEI)**, based on whether the claim can be substantiated, contradicted, or remains unresolved given the totality of Wikipedia evidence [1811.10971]. Annotators, blinded to the original source sentence, identify one or more *minimal* sets of sentences in Wikipedia such that their union is sufficient (and no proper subset is sufficient) to justify their judgment. These minimal evidence sets may draw from multiple Wikipedia pages, and annotators are instructed to provide alternative evidence groupings if available. ### Annotation Guidelines and Quality Control Annotation guidelines stipulate that only those sentences directly necessary to establish the label be included, and that each evidence set be both minimal and complete. Inter-annotator agreement is measured via Fleiss’ κ, achieving 0.6841 on a 5-way sample (4% of data), indicating substantial agreement [1803.05355]. Super-annotator validation yields a retrieval precision of 95.42% and recall of 72.36% compared to exhaustive expert annotation, with 91.2% correct label+evidence combinations in manual evaluation. Early trade-offs favoring annotation throughput were later mitigated by supplementary annotation rounds to augment the gold standard for challenging claims. ## 2. Dataset Composition and Splitting Strategy FEVER comprises 185,445 claims, rigorously stratified into training, development, and test sets. The splits are constructed to ensure **disjointness with respect to source Wikipedia pages**, preventing potential overlap in background knowledge [1811.10971]. | Split | Supported | Refuted | NotEnoughInfo | Total | |---------------|-----------|---------|---------------|-----------| | Training | 80,035 | 29,775 | 35,639 | 145,449 | | Development | 6,666 | 6,666 | 6,666 | 19,998 | | Test | 6,666 | 6,666 | 6,666 | 19,998 | - **Training set** is naturally imbalanced (∼47% Supported, 18% Refuted, 21% NEI). - **Dev/test sets** are fully balanced (each class exactly one-third). The average claim consists of 9.4 tokens (comparable to other entailment corpora). In terms of evidence annotation, 16.82% of claims require the *union of more than one sentence*, with an average of 1.17 sentences per claim (weighted by the single- and multi-sentence evidence proportions). Evidence from multiple Wikipedia pages is required for 12.15% of claims [1803.05355]. ## 3. Evidence Retrieval, Reasoning, and Annotation Challenges Evidence selection for each claim is performed at annotation time, with annotators permitted to search or browse Wikipedia and select sentences from any requisite page. The requirement that evidence sets be minimal and complete introduces significant complexity—multi-hop reasoning (across discontiguous sentences and pages) and entity disambiguation abound. Overall, 31.75% of claims (within Supported/Refuted) require more than one sentence for complete evidential justification, and ∼17% of all claims necessitate true multi-hop aggregation [1803.05355]. The annotation pipeline was designed to favor throughput, sacrificing some recall of all possible valid evidence sets, with later augmentation steps to increase gold standard coverage for evaluation. Key annotation challenges include: - Multi-hop inference over non-contiguous or cross-page facts - Semantic ambiguity and pronoun/ellipsis resolution - Incompleteness of evidence enumeration due to the open-domain and evolving nature of Wikipedia ## 4. Evaluation Metrics and Scoring Formulations FEVER introduces a rigorous set of evaluation metrics capturing both prediction correctness and evidential completeness: ### Label Accuracy \[ \text{Acc}_\text{label} = \frac{|\{\,i : \hat{l}_i = l_i\,\}|}{N} \] Fraction of claims with correctly predicted labels. ### Evidence-based Precision, Recall, and F₁ Union over all predicted and gold sentence-level evidence annotations is compared: - Precision: \( P_e = \frac{|{\cup}\hat{E} \cap {\cup}E^*|}{|{\cup}\hat{E}|} \) - Recall: \( R_e = \frac{|{\cup}\hat{E} \cap {\cup}E^*|}{|{\cup}E^*|} \) - F₁: \( F_1 = \frac{2 P_e R_e}{P_e + R_e} \) ### FEVER Score \[ \text{FEVER}_\text{Score} = \frac{1}{N}\sum_{i=1}^N s_i \] where \( s_i = 1 \) iff \( [\hat{l}_i = l_i] \) and \( [\exists E^*_i \subseteq \hat{E}_i] \). This metric strictly requires both the correct label and at least one *complete* minimal gold evidence set; NEI predictions require only correct labeling. The FEVER score penalizes systems that fail to justify Supported/Refuted labels with sufficient, minimal evidence, enforcing interpretable, explainable predictions [1811.10971]. ## 5. Baseline Systems and Shared Task Outcomes A pipeline architecture is specified as the canonical baseline: (1) document retrieval (DrQA’s TF–IDF, cosine similarity), (2) sentence selection (ranking sentences by TF–IDF similarity), and (3) recognizing textual entailment (RTE) using a single-layer MLP or Decomposable Attention (Parikh et al.). NEI labels are generated by pairing claims with random or nearest sentences [1803.05355]. Upper-bound oracle experiments show strong performance ceilings: 70.20% accuracy with perfect RTE and gold document retrieval (k=5). However, the best joint accuracy with correct evidence is 31.87%, while label-only accuracy reaches 50.91%. The inaugural FEVER Shared Task [1811.10971] attracted 23 competitive entries. Nineteen surpassed the published baseline; the best-performing system achieved a FEVER score of 64.21%. This outcome demonstrates substantial annotation and modeling bottlenecks, chiefly in evidence retrieval and multi-hop reasoning. Pipeline ablation studies indicate that omitting sentence selection reduces performance by approximately 10 absolute points. ## 6. Extensions, Applications, and Influence FEVER is the progenitor of an entire paradigm for fact verification and explainable entailment in open-domain settings. Its core design—intertwining minimal supporting/refuting evidence with claim-level labels—forms the archetype for numerous follow-on datasets in domain-specialized (e.g., CLIMATE-FEVER [2012.00614], COVID-Fact [2106.03794]), multilingual (e.g., CFEVER [2402.13025], Poly-FEVER [2503.16541], XFEVER [2310.16278]), and real-world claim contexts. All preserve or adapt the dual requirement for prediction and explicit citation of evidence. Practical applications include: - Open-domain claim verification and misinformation detection - Multi-hop reading comprehension and retrieval model benchmarking - Interpretable NLI/RTE system development - Exploration of neural theorem-proving and natural logic Notable limitations stem from Wikipedia’s domain coverage (restricted primarily to intros), incomplete evidence annotation, and the artificiality of some mutated claims. Nevertheless, FEVER remains a critical benchmark for developing robust, citation-aware automated fact-verification systems [1803.05355][1811.10971]. ## 7. Legacy and Comparative Context The FEVER dataset’s methodology—claim mutation, minimal evidence annotation, and joint correctness metrics—has become canonical in the field. Subsequent datasets extending this paradigm routinely emulate the FEVER protocol, varying only domain (scientific, climate, COVID-19), language (Chinese, multilingual), or annotation scalings. Comparative works highlight the challenges in transferring FEVER-honed models to real-world or specialized domains, revealing significant drops in accuracy and evidential sufficiency (e.g., CLIMATE-FEVER reports zero-shot FEVER model accuracy at 38.78%, F₁=32.85%, far below in-domain FEVER results) [2012.00614]. The tri-label schema (Supported, Refuted, NotEnoughInfo), stringent evidence annotation, and disclosure of all annotation, code, and interface tools underpin its widespread adoption as a fact verification standard for NLI, fact-checking, and explainable AI research [1803.05355][1811.10971]. Source: https://www.emergentmind.com/topics/fever-dataset