Fact-Checking Complex Claims with Program-Guided Reasoning (2305.12744v1)

Abstract: Fact-checking real-world claims often requires collecting multiple pieces of evidence and applying complex multi-step reasoning. In this paper, we present Program-Guided Fact-Checking (ProgramFC), a novel fact-checking model that decomposes complex claims into simpler sub-tasks that can be solved using a shared library of specialized functions. We first leverage the in-context learning ability of LLMs to generate reasoning programs to guide the verification process. Afterward, we execute the program by delegating each sub-task to the corresponding sub-task handler. This process makes our model both explanatory and data-efficient, providing clear explanations of its reasoning process and requiring minimal training data. We evaluate ProgramFC on two challenging fact-checking datasets and show that it outperforms seven fact-checking baselines across different settings of evidence availability, with explicit output programs that benefit human debugging. Our codes and data are publicly available at https://github.com/mbzuai-nlp/ProgramFC.

Overview of Program-Guided Fact-Checking with ProgramFC

The research paper "Fact-Checking Complex Claims with Program-Guided Reasoning" introduces ProgramFC, a novel framework designed to enhance the process of fact-checking complex claims through program-guided reasoning. ProgramFC addresses the challenge of verifying real-world claims that require the integration of multiple pieces of evidence and complex multi-step reasoning.

Core Contributions

ProgramFC innovatively decomposes complex claims into simpler sub-tasks that are executed via a shared library of specialized functions. This decomposition is guided by reasoning programs, generated using the in-context learning capabilities of LLMs, specifically Codex. Such programs provide detailed, step-by-step guidance for the verification process, making the model both explanatory and data-efficient.

Experimental Validation

The framework has been evaluated on two datasets designed for fact-checking complex claims: HOVER and FEVEROUS. ProgramFC demonstrated substantial performance improvements over seven existing fact-checking baselines. Particularly, it showed superior effectiveness as the complexity (or the reasoning depth) of the claims increased. In scenarios where reasoning depth was at its highest, ProgramFC provided significant gains, outperforming the best baseline by over 14% on four-hop claims.

Mechanisms and Details

1. Program Generation: At the heart of ProgramFC is the generation of reasoning programs, which are stepwise, executable plans derived from an input claim. This is achieved through the few-shot capabilities of Codex with custom-designed prompts.
2. Program Execution: Once the reasoning program is generated, each sub-task is executed by specialized modules. These include question-answering, claim verification, and logical reasoning components, which together ensure a comprehensive approach to verification.
3. Iterative and Multi-Step Retrieval: The model leverages an iterative retrieval process that surpasses traditional one-step methods, increasing recall by over 37% in certain cases, particularly for more complex claims.

Implications and Future Directions

The implications of ProgramFC are pronounced in its combination of symbolic reasoning and the data-driven adaptability of neural networks, providing a framework that potentially bridges the gap between explainability and performance.

Practically, ProgramFC could offer substantial utility in real-world applications where the transparency of decision processes is critical—such as journalism, regulatory compliance, and fact-checking organizations. Theoretically, the paper sets the stage for future exploration into more complex, implicit reasoning tasks and further integration with advanced retrieval mechanisms.

In conclusion, while the ProgramFC framework offers a promising direction for the fact-checking community, there remain opportunities for enhancing its capabilities in handling more implicit logical inferences and improving computational efficiency. Future work could expand on integrating commonsense and world knowledge into the reasoning programs and further explore efficient fine-tuning of LLMs to enhance their application in fact-checking and beyond.