Combining Fact Extraction and Verification with Neural Semantic Matching Networks (1811.07039v1)

Abstract: The increasing concern with misinformation has stimulated research efforts on automatic fact checking. The recently-released FEVER dataset introduced a benchmark fact-verification task in which a system is asked to verify a claim using evidential sentences from Wikipedia documents. In this paper, we present a connected system consisting of three homogeneous neural semantic matching models that conduct document retrieval, sentence selection, and claim verification jointly for fact extraction and verification. For evidence retrieval (document retrieval and sentence selection), unlike traditional vector space IR models in which queries and sources are matched in some pre-designed term vector space, we develop neural models to perform deep semantic matching from raw textual input, assuming no intermediate term representation and no access to structured external knowledge bases. We also show that Pageview frequency can also help improve the performance of evidence retrieval results, that later can be matched by using our neural semantic matching network. For claim verification, unlike previous approaches that simply feed upstream retrieved evidence and the claim to a natural language inference (NLI) model, we further enhance the NLI model by providing it with internal semantic relatedness scores (hence integrating it with the evidence retrieval modules) and ontological WordNet features. Experiments on the FEVER dataset indicate that (1) our neural semantic matching method outperforms popular TF-IDF and encoder models, by significant margins on all evidence retrieval metrics, (2) the additional relatedness score and WordNet features improve the NLI model via better semantic awareness, and (3) by formalizing all three subtasks as a similar semantic matching problem and improving on all three stages, the complete model is able to achieve the state-of-the-art results on the FEVER test set.

Semantic Matching for Fact Verification: An Evaluation of Combining Extraction and Verification Techniques

The paper, "Combining Fact Extraction and Verification with Neural Semantic Matching Networks," addresses the critical issue of misinformation by developing a system for fact verification using neural models trained on the Fact Extraction and VERification (FEVER) dataset. This research explores an innovative architecture involving neural semantic matching for various stages of evidence retrieval and claim verification, an area of growing interest given the proliferation of unverified information online.

The authors propose a coherent framework employing three homogeneous neural semantic matching models to tackle document retrieval, sentence selection, and claim verification. This unified neural approach leverages the raw textual input and eschews traditional intermediate term representations, which is a departure from classical vector space IR models. In the context of information retrieval, this technique allows the system to conduct deep semantic matching and generate representations optimized for fact verification. This approach shows empirical improvements, significantly outperforming baseline TF-IDF and encoder models on the FEVER dataset, evident by superior performance metrics across all stages of evidence retrieval.

The proposed method introduces a novel combination of semantic relatedness scores integrated into the NLI model and utilizes external knowledge from WordNet to boost the semantic understanding capability of the system. These contributions culminate in a method that achieves a state-of-the-art performance on the FEVER test set, doubling the baseline results. The paper highlights the methodological advancements in formulating the entire task as a semantic matching problem, which provides an avenue for refining the future design of fact-checking systems.

The implications of this work are significant for both practical applications and theoretical developments. Practically, the creation of a system that effectively softens the boundaries between evidence retrieval and verification can lead to more comprehensive solutions in automated fact-checking applications across various domains, from journalism to academia, where misinformation frequently arises. Theoretically, the paper's approach encourages further exploration into using semantic matching to address similar evidence retrieval challenges in different settings, broadening the scope of natural language processing and information retrieval research.

Speculating on future developments, one can anticipate enhanced systems that incorporate more diversified external knowledge bases and extend semantic matching networks to other tasks. Given the promising results achieved by combining relatedness scores and ontological features, future work could delve into richer linguistic resources, potentially leading to even more robust fact verification models capable of processing complex natural language inference tasks with higher accuracy. Additionally, advancements in neural network architectures and fine-tuning techniques specific to fact-checking tasks could further enhance the capabilities of such systems.

In summary, the paper presents a systematic approach leveraging neural semantic matching for solving the critical task of fact extraction and verification. The strong numerical results underscore the methodological rigor and the potential of neural techniques to advance the field of AI-driven fact verification.