MultiVerS: Improving scientific claim verification with weak supervision and full-document context (2112.01640v2)

Abstract: The scientific claim verification task requires an NLP system to label scientific documents which Support or Refute an input claim, and to select evidentiary sentences (or rationales) justifying each predicted label. In this work, we present MultiVerS, which predicts a fact-checking label and identifies rationales in a multitask fashion based on a shared encoding of the claim and full document context. This approach accomplishes two key modeling goals. First, it ensures that all relevant contextual information is incorporated into each labeling decision. Second, it enables the model to learn from instances annotated with a document-level fact-checking label, but lacking sentence-level rationales. This allows MultiVerS to perform weakly-supervised domain adaptation by training on scientific documents labeled using high-precision heuristics. Our approach outperforms two competitive baselines on three scientific claim verification datasets, with particularly strong performance in zero / few-shot domain adaptation experiments. Our code and data are available at https://github.com/dwadden/multivers.

MultiVerS: Enhancing Scientific Claim Verification Through Weak Supervision and Full-Document Context

The paper "MultiVerS: Improving scientific claim verification with weak supervision and full-document context" presents a novel approach aimed at refining the task of scientific claim verification within NLP. This approach, termed MultiVerS, focuses on labeling scientific documents as supporting or refuting specific claims, while simultaneously selecting evidentiary sentences critical for justifying each decision made by the system. The paper emphasizes two primary advancements: utilizing a multitask learning approach that integrates full-document context and employing weak supervision for domain adaptation in scientific claim verification tasks.

Key Concepts and Methodology

Scientific claim verification, a task increasingly vital in the era of rampant misinformation, typically involves assessing the relationship between an input claim and scientific literature. The traditional "extract-then-label" approach often results in reliance on evidence sentences that lack sufficient contextual information for accurate label prediction. The paper counters this by introducing MultiVerS, which generates a shared encoding of the claim coupled with the entire document context using Longformer. This model facilitates the consistent prediction of abstract-level labels and sentence-level rationales, ensuring decisions integrate all relevant context. Moreover, MultiVerS extends its capabilities via weak supervision, utilizing scientific documents labeled through high-precision heuristics to improve performance in scenarios where sentence-level rationale annotations are scarce.

Comparative Performance Analysis

MultiVerS demonstrates its superiority over existing methods in a variety of experiments. Evaluating it against the benchmarks Vert5Erini and ParagraphJoint, MultiVerS consistently outperforms these models across three scientific claim verification datasets: SciFact, HealthVer, and COVIDFact. Notably, MultiVerS achieves relative improvement in zero-shot, few-shot, and fully-supervised setting experiments with average F1 score gains of 26%, 14%, and 11% respectively. These enhancements affirm MultiVerS's ability to effectively utilize the full-document context and its robustness in domain adaptation, particularly with limited labeled data.

Implications, Challenges, and Future Prospects

The implications of this research are substantial, particularly regarding the development of systems that can efficiently tackle misinformation within scientific discourse. By integrating weakly-supervised domain adaptation mechanisms, MultiVerS provides a flexible framework applicable across various specialized scientific domains with minimal labeled data requirements. Moreover, the multitask learning strategy positions this model well for future explorations into complex document verification tasks, potentially expanding beyond abstracts to entire research papers.

However, challenges remain, notably in optimizing negative sampling and addressing the complexity of sentences lacking standalone rationale capabilities. These challenges signal opportunities for further innovation in this field.

In conclusion, by advancing the methodologies for scientific claim verification, MultiVerS stands as a promising contributor to ensuring the accuracy of information within scientific writing. Its design and empirical evidence strongly support its applicability in addressing the rising tide of scientific misinformation, setting a solid foundation for future explorations and improvements in automated fact-checking systems.