Agentic Fact-Checking Architecture

• The topic Agentic Fact-Checking System Architecture is a computational framework that orchestrates autonomous modules for claim verification using retrieval, ranking, and NLI processes.
• It employs a modular pipeline that combines BM25-based document retrieval, sentence ranking with positional and semantic scoring, and robust evidence classification.
• The design emphasizes transparency, scalability, interactive human feedback, and adaptability to evolving misinformation challenges.

Agentic Fact-Checking System Architecture refers to computational frameworks where autonomous or semi-autonomous agents orchestrate information retrieval, evidence evaluation, reasoning, and verdict explanation to assess the veracity of claims. Recent work conceptualizes "agentic" systems as those that decompose, coordinate, and dynamically adapt fact-checking workflows across modular components, supporting scalable, transparent, and often interactive operations in complex, real-world misinformation settings (Miranda et al., 2019).

1. Modular System Design and Pipeline Structure

Agentic fact-checking architectures are organized into pipelines comprised of modular, sequential components that reflect the multi-step workflow of human fact-checkers. A canonical design includes:

1. Document Retrieval: An index-backed module retrieves a large set of candidate documents relevant to the claim. For instance, a BM25-based inverted index is used over news articles, leveraging lemmas, words, and named entities as index features. Average retrievals number ~10,000 documents with median latency ~50 ms.
2. Sentence Ranking: Extracted sentences from those documents are scored for relevance via a two-stage process:
   • Positional Feature Matching: Calculates $S_1(s_i, c) = \sum_{j=1}^N \exp( - d_{i,j} )$, where $d_{i,j}$ measures ordered feature distances, favoring sentences where claim features co-occur and co-locate.
   • Embedding Similarity: Computes cosine similarity between claim and sentence embeddings (TF-IDF weighted averages over One Billion Word Benchmark-trained vectors), averaging this with $S_1$ to determine final relevance.
   • A strict threshold is applied to retain only highly relevant sentences (e.g., final set $\approx$25).
3. Evidence Classification (Natural Language Inference, NLI): A state-of-the-art NLI model, e.g., from Hexa-F, labels each evidence sentence as supporting, refuting, or other (neutral/related). The model aggregates per-sentence decisions to produce an overall claim verdict.

This pipeline is visualized as:

$$\text{Claim} \rightarrow [\text{Retrieval}] \rightarrow [\text{Ranking}] \rightarrow [\text{NLI}] \rightarrow \text{Verdict}$$

The clear demarcation and orchestration of modules allow for targeted optimization, maintenance, and future augmentation (Miranda et al., 2019).

2. Evidence Processing and Scoring Methodologies

The sentence ranking relies on a mathematically precise feature mapping:

• Feature Matching Score:

$$S_1(s_i,c) = \sum_{j=1}^{N} \exp \left( - ( \operatorname{pos}(\phi(s_i)_j) - \operatorname{pos}(\phi(s_i)_{j-1}) ) \right )$$

Where $\phi(s_i)_j$ and $\phi(c)_j$ are the ordered features in the sentence and claim, respectively. Exponential decay penalizes out-of-order or overly dispersed matches.

• Semantic Similarity Score: Sentence and claim embeddings ($e_s$, $e_c$) are calculated as TF-IDF-weighted averages over word vectors, followed by

$S_2 = \cos(e_s, e_c)$

• Score Aggregation and Filtering: The average score $(S_1+S_2)/2$ is thresholded (cutoff empirically set at 0.6) to filter relevant evidence; typically, $\sim$76 candidates after initial ranking, reduced to about 25 high-quality sentences.
• NLI-based Evidence Classification: The system's NLI classifier processes these sentences, labeling them as supporting, refuting, or other with a runtime of ~738 ms per claim. The model also aggregates individual evidence labels for an overall verdict.

Empirical studies revealed that the per-evidence relevance (rated by professional journalists) was 59%, with NLI evidence label correctness at 58%. Precision for support/refute labeling improved when filtered on journalist-relevant evidence (e.g., support precision rose to 67%), but overall global claim classification accuracy remained lower at 42% (Miranda et al., 2019).

3. User Interaction, Transparency, and Feedback Integration

Agentic systems emphasize explainability and real-time feedback through specialized user interfaces:

• Claim Input: Users (primarily journalists) submit claims via a distinct interface element.
• Evidence Display Panel: Evidence is arranged in three columns—support, refute, and related/other—with the top five sentences in each category prominently shown.
• Verdict Visualization: The system’s verdict (support, refute, other) is rendered clearly below the evidence.
• Transparency Features: Each evidence snippet includes bolded named entities, provenance (document snippet), and extractions to provide context.
• Interactive Feedback: Journalists supply feedback via buttons assessing:
  • NLI label correctness (“correct label?”)
  • Evidence relevance (“relevant?”)
  • Verdict appropriateness (global claim label)
  • This feedback is collected at both the per-evidence and final verdict levels.

Qualitative feedback from journalists indicates that such transparency aids trust and usability; requests for temporally-aware reasoning and evidence evolution visualization suggest key future enhancements (Miranda et al., 2019).

4. Evaluation in Journalistic Workflows

The platform was empirically validated with 11 BBC journalists using 67 claims, leading to these key observations:

• Relevance and Correctness: 59% of retrieved evidence deemed relevant; support and refute columns achieved respective relevance of 71% and 69%.
• Support/Refute Precision: Overall support evidence precision: 48% (full set), 67% (filtered by journalist relevance); refute precision: 27% (full), improved post filtering.
• Global Verdict: Only 42% of overall system predictions matched ground truth.
• Workflow Insights: Feedback revealed that while the architecture was helpful, enhancements in evidence opposition detection and temporal awareness (date-handling, present/past tense, evolution tracking) were necessary for real-world deployment.

These findings reveal the system’s strengths in modularity and explainability but signal a gap in achieving highly reliable end-to-end automated verification in journalistic environments (Miranda et al., 2019).

5. Architectural Features for Agentic Adaptation

The platform’s design demonstrates several characteristics critical for agentic fact-checking systems:

• Modularity and Extensibility: Separate modules for retrieval, ranking, and NLI classification promote targeted upgrades, domain adaptation, and ensembling with newer models or retrievers.
• Interactive Human-in-the-Loop: Feedback integration enables semi-automated operation, allowing for retraining, expert correction, and iterative system improvement.
• Evidence-Based Transparency: Rigorous evidence presentation and labeling provide the necessary foundation for explainable AI—a core requirement for both regulatory contexts and user trust.
• Scalability: BM25 and embedding-based retrieval, coupled with efficient ranking heuristics and batched NLI processing, enable adaptation to large and frequently updated news corpora.
• Platform Differentiators: Compared to fully end-to-end or hallucination-prone “prompt-only” LLM architectures, the explicit evidence aggregation, threshold filtering, and NLI-in-the-loop approach prevents spurious predictions and surfaces the basis for each decision transparently.

The architecture reveals a promising foundation for autonomous, explainable, and scalable fact-checking agents that could operate across distinct journalistic and information ecosystems (Miranda et al., 2019).

6. Open Problems and Future Directions

Open research questions and enhancement directions articulated by users include:

• Temporal Reasoning: Improved management of tense, date, and time-sensitive information in evidence selection and inference.
• Evidence Evolution: Visualization and tracking of claim–evidence relationships as new information becomes available or as stories progress.
• Better Opposition Retrieval: Enhanced methods to source and prioritize evidence that actively refutes, not merely relates to, the claim.
• Continuous Learning: Mechanisms for learning from user feedback loops, integrating new sources, and adapting to shifts in linguistic or factual patterns in the target domain.

A plausible implication is that future systems will require more advanced temporal NLP, cross-document reasoning, and web-scale, continually refreshed retrieval mechanisms.

7. Summary Table of Core Workflow Components

Workflow Stage	Methodology	Output
Document Retrieval	BM25 inverted index, entity & word matching	~10K candidate docs
Sentence Ranking	Exponential positional score, cosine TF-IDF similarity	~25 top sentences
Evidence Classification NLI	Hexa-F NLI, labels: support/refute/other, aggregation	Claim verdict + per-evidence
User Interface/Feedback	3-column display, interactive evidence feedback	Verdict, transparency

• "Automated Fact Checking in the News Room" (Miranda et al., 2019)