Active Fact-Checking

• Active Fact-Checking is a computational and sociotechnical process that proactively verifies factual claims using NLP, IR, and expert engagement.
• It integrates multi-stage pipelines—from claim ingestion and keyword extraction to evidence retrieval and expert opinion filtering—to accelerate validation.
• Evaluations show that systems like aedFaCT improve verification efficiency and evidence quality while maintaining essential human oversight.

Active Fact-Checking refers to a set of computational, organizational, and interactive processes designed to facilitate the rapid, proactive verification of factual claims in real-world media, research, and conversation streams. Unlike passive or post-hoc approaches—which often rely on later, manual verification—active fact-checking embeds detection, evidence gathering, adjudication, and user engagement into live workflows. It leverages NLP, information retrieval (IR), machine learning, and sociotechnical infrastructures to semi-automate and accelerate the cycle of claim investigation, supporting both professional fact-checkers and end-users in making informed judgments about information veracity.

1. Architectural Foundations and User Workflows

Active fact-checking platforms are structured as multi-stage pipelines integrating claim ingestion, evidence retrieval, expert opinion mining, verification, and decision support. A canonical workflow, as instantiated by "aedFaCT: Scientific Fact-Checking Made Easier via Semi-Automatic Discovery of Relevant Expert Opinions" (Altuncu et al., 2023), proceeds as follows:

• Claim Input: Users invoke the system in situ—e.g., through a browser extension—while viewing an article. The platform extracts textual content and enters the keyword extraction phase.
• Automatic Keyword Extraction (AKE): Algorithms such as SIFRank+ process the article, identifying top noun phrases based on smoothed inverse frequency (SIF) embedding scores and domain thesaurus/Wikipedia checks.
• Evidence Discovery: Selected keywords are joined in a Boolean AND query, dispatched in parallel to curated engines—mainstream news, scientific news, and schema-filtered credible sources—returning prioritized results.
• Expert Opinion Filtering: spaCy-based NER and reported-speech heuristics scan fetched paragraphs, retaining only those with person entities, academic organizations, and direct quotations.
• Scientific Literature Retrieval: The same query is issued to bibliometric databases like Scopus, presenting peer-reviewed abstracts and ranking co-authors by both output frequency and profile completeness.
• Integrated Decision Pane: End-users view collated expert quotes, publication evidence, and direct researcher contacts in a single interface, facilitating cross-validation and independent verdict formation without context switching.

2. Key Algorithmic Components and Retrieval Strategies

Active fact-checking systems commonly employ hybrid retrieval and filtering approaches:

• Keyword Scoring: Candidate phrases are assigned scores by $$SIFRank^+(p) = \sum_{w \in p} \alpha \cdot embedding(w) \cdot IDF(w)$$, followed by domain-specific post-filtering.
• Boolean Query Construction: Keywords $\{k_1, \ldots, k_n\}$ are joined as $$Q = "k_1" \ \text{AND}\ "k_2" \ \text{AND} \ldots \ "k_n"$$ for retrieval APIs.
• Site-Type Prioritization: When merging evidence, results are sorted by source credibility (e.g., mainstream news > scientific news > other credible outlets).
• Quoted-Expert Mining: Candidate news paragraphs are filtered by $$\exists \mathrm{PERSON} \land \exists \mathrm{ORG\_token} \land \exists \mathrm{quote\_marks}$$.
• Researcher Profiling and Ranking: Co-authors are scored by $$score(r) = \lambda \cdot \#(\text{papers by } r) + (1-\lambda)\cdot \text{profile\_completeness}(r)$$ with $\lambda$ tuned for publication emphasis.

These steps are explicitly designed to parallel—and semi-automate—standard investigative workflows in journalistic and scientific settings.

3. Data Sources, Filtering Heuristics, and System Integration

Active fact-checking tools integrate heterogeneous, high-credibility information streams:

• News corpora: Carefully curated sets of outlets, validated by schema.org NewsArticle filters and domain-quality indices (Iffy Index, Media Bias/Fact Check ratings).
• Scientific databases: APIs such as Scopus (via Pybliometrics) underpin access to peer-reviewed research, with author metadata, affiliations, and contact details screened.
• Heuristic exclusion: Site lists are continuously refined to drop low-quality or misinformation-prone domains, maintaining both evidence relevance and trustworthiness.
• User interface controls: Stakeholders retain the ability to select/deselect keywords, view source metadata, and manually investigate evidence chains—preserving a human-in-the-loop paradigm over fully-automated adjudication.

4. Evaluation Methodologies and Performance Findings

Preliminary evaluation of active fact-checking systems such as aedFaCT (Altuncu et al., 2023) focuses on methodology fidelity, comparative judgment accuracy, and user experience:

• Participants: Independent testers (PhD-level) assess platforms across curated article sets, simulating both lay and expert user scenarios.
• Metrics: “Needs Met” scores (five-point scale per Google Search Quality Guidelines) and inter-rater agreement (Fleiss’ Kappa) quantify both absolute and comparative performance.
• Findings:
  • Manual fact-checking: Avg. rating 4.35/5.0.
  • aedFaCT platform: Avg. rating 4.57/5.0.
  • Agreement: Fleiss’ Kappa 0.533 (moderate).
  • All testers reported aedFaCT as perceptually faster, with no decrease in evidence quality.

Qualitative feedback reinforces the centrality of unified, multi-source panes and on-demand expert access for rapid, high-quality claim triage.

5. Impact on Fact-Checking Practices and Human Alignment

Active fact-checking demonstrably accelerates the classical workflow loop (“investigate → gather evidence → decide”) by compressing multi-tabbed, search-intensive routines into a limited clickstream. This acceleration is achieved without sacrificing rigor or veracity and instead enforces structural alignment between semi-automated processes and domain expert judgment. Critically, the design principles underpinning current systems ensure:

• User agency at every selection, filtering, and final adjudication step.
• Evidence traceability, with direct links to diversified sources and expert commentary.
• Support for conventional investigation patterns, echoing editorial extraction, cross-source validation, and peer literature reviews.

Moreover, modular architectures facilitate future scaling—expansion of Boolean logic, claim detection, contradiction resolution, and institutional data crawling—without requiring wholesale system redesign.

6. Limitations and Future Directions

Current systems face constraints stemming from external API rates, Boolean query expressivity, and evaluation sample size. Key limitations include:

• Limited query logic: Only AND joins, no OR for synonyms or NOT for irrelevancy—reducing recall under polysemous or ambiguous claim scenarios.
• Retrieval bottlenecks: Dependence on SIFRank+ for AKE throttles throughput; lightweight alternatives may optimize response times.
• Evaluation breadth: Initial user studies are limited (n=3), focusing on homogeneous backgrounds; extensive field testing across professional and lay populations is required.
• Scalability constraints: API call quotas (Google Custom Search: $\sim$10k/day, Scopus: 5–20k/week) currently cap deployment at institutional scale.
• Planned improvements: Integration of institutional/governmental crawling, tight claim detection, enriched query logic, and contradiction detection among expert opinions.

7. Comparative Systems and Paradigms

While aedFaCT manifests expert-centric, browser-integrated fact-checking, other active paradigms—such as live audio stream verification (LiveFC (V et al., 2024)), chat-based interventions, and editor-integrated fact-checks—address different modalities and interaction styles. Across these systems, common design characteristics prevail:

• Emphasis on multi-modal, real-time detection (text, audio, citation networks).
• Modular pipelines for evidence ingestion, triage, and verification.
• Hybrid automation that preserves human validation and decision support.

Active fact-checking thus emerges as a cross-domain, multi-infrastructure approach to scalable, high-fidelity claim adjudication in contemporary information ecosystems.