Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (2103.08541v1)

Abstract: Typical fact verification models use retrieved written evidence to verify claims. Evidence sources, however, often change over time as more information is gathered and revised. In order to adapt, models must be sensitive to subtle differences in supporting evidence. We present VitaminC, a benchmark infused with challenging cases that require fact verification models to discern and adjust to slight factual changes. We collect over 100,000 Wikipedia revisions that modify an underlying fact, and leverage these revisions, together with additional synthetically constructed ones, to create a total of over 400,000 claim-evidence pairs. Unlike previous resources, the examples in VitaminC are contrastive, i.e., they contain evidence pairs that are nearly identical in language and content, with the exception that one supports a given claim while the other does not. We show that training using this design increases robustness -- improving accuracy by 10% on adversarial fact verification and 6% on adversarial natural language inference (NLI). Moreover, the structure of VitaminC leads us to define additional tasks for fact-checking resources: tagging relevant words in the evidence for verifying the claim, identifying factual revisions, and providing automatic edits via factually consistent text generation.

Robust Fact Verification with Contrastive Evidence

The paper "Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence" addresses the challenge of fact verification in evolving textual landscapes. The authors introduce VitaminC, a benchmark designed to enhance the robustness of fact verification models in adapting to subtle factual changes. This work is predicated on the dynamic nature of sources like Wikipedia, which undergo frequent revisions that can impact the status of existing facts.

VitaminC Benchmark

VitaminC is a large-scale dataset comprising over 400,000 claim-evidence pairs, derived from more than 100,000 real Wikipedia revisions. The dataset captures the nuances of factual revisions by presenting contrastive evidence—evidence pairs differing minimally in language but supporting or refuting a given claim. This structure is pivotal as it encourages models to rely on evidence-sensitive inference rather than memorizing static knowledge.

Performance Improvements

Training models with VitaminC leads to notable performance improvements. Experiments reported in the paper show a 10% increase in accuracy on adversarial fact verification and a 6% improvement in adversarial natural language inference. The results are significant as they underscore the importance of contrastive examples in fortifying models against adversarial attacks.

Novel Tasks and Challenges

VitaminC's unique contrastive design enables exploration of several new tasks in the fact verification domain:

1. Factual Revision Flagging: Classification tasks distinguishing factual from non-factual changes in Wikipedia text.
2. Word-level Rationales: Fine-grained explanations highlighting relevant words in evidence that influence the verification of claims.
3. Factually Consistent Generation: Automation of text updates to align documents with current facts.

These tasks reflect critical nuances in maintaining factual consistency in dynamically updated repositories like Wikipedia.

Implications and Future Work

The research has broader implications for AI applications reliant on mutable knowledge bases. By facilitating models that dynamically adjust their factual veracity judgments, VitaminC pushes the envelope on context-sensitive AI systems capable of integrating new information fluidly. Future developments could explore further applications in domains requiring real-time fact-checking, continuous learning from textual updates, and enhanced interpretability of AI decisions.

Moreover, the dataset's release on a public platform encourages the adoption and enhancement of fact-checking models across diverse settings—spurring collaborative progress in AI research focused on factual accuracy and adaptability.