Co-FactChecker: Collaborative Fact Verification

• Co-FactChecker is a collaborative fact-checking framework that integrates AI-driven evidence retrieval with human, community, and expert interventions to verify public claims.
• It employs modular pipelines that decompose the process into claim ingestion, evidence retrieval, verification, explanation extraction, and feedback-driven trace editing.
• Evaluations show improved explainability and faster human correction, while addressing challenges like bias, retrieval limitations, and scalability in diverse settings.

Searching arXiv for the cited Co-FactChecker and closely related fact-checking systems to ground the article with current paper metadata and IDs. Co-FactChecker denotes a class of collaborative fact-checking systems that combine automated claim verification with human, community, or expert intervention. In the literature, the label is used both for system blueprints that integrate crowd moderation, professional review, and AI support, and for a framework for human-AI collaborative claim verification using large reasoning models (LRMs), where a model’s “thinking trace” is treated as a shared scratchpad that experts can edit directly. Related work under the same or adjacent design vocabulary emphasizes explainable verdicts, retrieval of semantically similar prior fact-checks, modular end-to-end pipelines, and standardized evaluation of both claims and fact-checkers (Sahnan et al., 15 Apr 2026, Augenstein et al., 26 May 2025, Zhang et al., 2021, Wang et al., 2024).

1. Conceptual scope and formal definitions

In the community-moderation formulation, Co-FactChecker is a hybrid fact-checking system that unites community-driven moderation with professional expertise and AI support. Its core objects are defined explicitly: misinformation is “any public claim $c$ whose content deviates from the best available evidence or established facts”; a verification signal is a structured unit of evidence attached to $c$; a community note $n$ is a user-generated commentary citing one or more verification signals; a Crowd Consensus Score $S(n)$ measures how “helpful” the crowd judges $n$ to be; a Trust Score $t_u$ is a dynamic reliability rating for user $u$; and a Diversity Weight $d_u(n)$ represents how ideologically, culturally, or linguistically distinct $u$ is from the existing set of raters of $n$. The operational score is

$c$0

and a note is published if $c$1 (Augenstein et al., 26 May 2025).

In the 2026 LRM-based formulation, Co-FactChecker separates claim verification into a retriever $c$2, a verifier $c$3, and an editor $c$4. Given a claim $c$5, retrieved evidence $c$6, and label set $c$7, the verifier produces a candidate solution

$c$8

where $c$9 is the veracity label, $n$0 is a short explanation, and $n$1 is a thinking trace. Expert feedback $n$2 is mapped into an edited trace through a trace-editing operator

$n$3

after which the verifier resumes generation from the edited prefix to produce an improved solution $n$4 (Sahnan et al., 15 Apr 2026).

Taken together, these definitions indicate that Co-FactChecker is not a single invariant architecture. This suggests that the term functions as a collaborative design pattern centered on explicit evidence, inspectable intermediate representations, and feedback-weighted revision.

2. Pipeline architectures and modular decomposition

Several Co-FactChecker instantiations are organized as modular pipelines. FaxPlainAC is a three-stage pipeline: evidence retrieval, interpretable fact-verification and explanation extraction via a two-phase “predict-and-explain-then-re-predict” model (ExPred), and a human-in-the-loop feedback interface whose annotations drive life-long updates of the underlying model. The concrete implementation queries Google’s Programmable Search Engine, fetches the top-$n$5 Wikipedia pages through the MediaWiki REST API, prunes them to the first 30 sentences, and passes each $n$6claim, document$n$7 pair to a BERT-based encoder. Retrieval, verification, explanation, and feedback communicate via well-specified JSONs, which makes the system explicitly interchangeable at the module boundary (Zhang et al., 2021).

A FAKTA-derived Co-FactChecker uses five major sequential components plus a collaborative layer: claim ingestion and preprocessing; document retrieval and re-ranking; stance detection and rationale extraction; linguistic analysis; and aggregation with verdict classification. Its retrieval stage issues POS-filtered and named-entity-based queries to Wikipedia, high-reliability media, mixed-reliability media, and low-reliability media; its collaborative layer exposes retrieval scores, stances, evidences, and linguistic charts in a shared UI where registered users can up-vote or down-vote documents and sentences, suggest alternative evidence, or flag misclassifications (Nadeem et al., 2019).

A real-time newsroom-oriented design begins with stream ingestion, sentence-level claim detection, semantic retrieval of prior fact-checks, a shared claims database, online clustering, and a collaborative UI. Claims are embedded with the Google Universal Sentence Encoder Large, stored with embeddings and cluster IDs in a central Redis/Elasticsearch store, and surfaced to fact-checkers in real time. Completed fact-checks are written back to the corpus, re-indexed, and propagated into updated cluster assignments, with work-in-progress locks preventing duplication (Adler et al., 2019).

OpenFactCheck generalizes the modular view further by decomposing fact-checking infrastructure into three cooperating modules: CUSTCHECKER for user-configurable fact-checking pipelines, LLMEVAL for standardized factuality evaluation of LLMs, and CHECKEREVAL for benchmarking automatic checkers against human gold labels. The backend exposes a toolkit API plus a Streamlit web UI, and YAML configuration selects the claim processor, retriever, and verifier (Wang et al., 2024).

Across these systems, the stable architectural motif is decomposition into ingest, retrieve, verify, explain, evaluate, and revise. The significance of that motif is practical rather than merely stylistic: it permits organizations to substitute retrieval engines, evidence sources, verifiers, or review procedures without rewriting the entire pipeline.

3. Human participation: annotation, moderation, and trace-editing

Human participation enters Co-FactChecker through at least three distinct mechanisms: lightweight correction interfaces, newsroom collaboration, and structured intervention in model reasoning.

FaxPlainAC embodies the first mechanism. Its Flask front-end presents the top 3 documents for each claim together with the model’s SUPPORT/REFUTE label, extracted evidence spans, and an “Agree?” column. If a user selects “No,” the interface asks whether the document is misleading, irrelevant, or requires token-level highlighting of true support or refutation. Feedback is stored in a lightweight SQLite database as triplets $n$8claim, doc_id, annotation$n$9, and future training rounds treat these human labels as additional gold examples. The architecture is explicitly designed for life-long learning, with an augmented loss

$S(n)$0

and a recommended rehearsal strategy in which each update samples a mini-batch half from the original FEVER data and half from $S(n)$1 to prevent catastrophic forgetting (Zhang et al., 2021).

In newsroom studies, Procter et al. describe five core stages of fact-checking work: surfacing candidate claims, triage or check-worthiness decision, evidence gathering, review and quality assurance, and writing up and publication. Triage uses a “spread, severity, amplification” calculus, and review depth varies with complexity. Full Fact is reported as requiring a minimum of three pairs of eyes, sometimes escalating to eight to ten for complex checks, while mainstream newsroom workflows emphasize authoritative sourcing and editorial review. The same study identifies a computational requirement for a configurable triage dashboard using

$S(n)$2

where $S(n)$3 is normalized spread, $S(n)$4 is normalized harm, and $S(n)$5 is normalized amplification risk (Procter et al., 2023).

The community-moderation variant formalizes participation through eligibility, trust updating, and escalation. Users pass onboarding test questions and anti-bot checks; each user’s trust score is updated after every project by

$S(n)$6

Notes that remain in the “borderline” zone $S(n)$7 are forwarded to a professional fact-checker queue, and experts can overrule or endorse the note, generating a final certification (Augenstein et al., 26 May 2025).

The 2026 Co-FactChecker introduces a stricter expert-model interaction paradigm. Rather than giving feedback in ordinary multi-turn dialogue, experts inspect a single evolving trace, validate or reject individual reasoning steps, and issue natural-language corrections that are translated into removals, modifications, or appended guidance. The framework argues that trace-editing has an information-capacity advantage over dialogue because, in dialogue, expert intent $S(n)$8 must pass through a bounded internal state $S(n)$9, yielding $n$0, whereas in trace-editing the edited trace space can carry at least $n$1 bits of correction signal. The paper’s theorem statements formalize this as an information-capacity advantage and an optimization advantage under the stated capacity assumptions (Sahnan et al., 15 Apr 2026).

A recurrent controversy concerns whether participation by non-experts can substitute for professional verification. The literature does not support that conclusion. Community efforts are described as valuable for scale and breadth, but not as a replacement for “the indispensable role of professional fact-checkers” (Augenstein et al., 26 May 2025).

4. Retrieval, evidence modeling, and verification algorithms

The verification core of Co-FactChecker systems spans interpretable classifiers, stance models, semantic retrievers, and domain-specific scientific evidence pipelines.

ExPred, the model shipped with FaxPlainAC, is a multi-task BERT design. From the pooled representation $n$2, a binary fact-verification head computes

$n$3

predicting SUPPORT if $n$4 and REFUTE otherwise. In parallel, a token-level decoder computes evidence probabilities

$n$5

and marks tokens whose score exceeds a threshold $n$6 as rationale. The model is trained jointly on FEVER with total loss

$n$7

followed by a second phase in which a fresh binary classifier is retrained on a masked training set that preserves only predicted evidence tokens and replaces all others with “.”. This second phase is intended to improve robustness to spurious cues (Zhang et al., 2021).

The real-time claim-detection system represents each sentence $n$8 with the Google Universal Sentence Encoder Large, producing $n$9, then applies logistic regression

$t_u$0

to determine whether $t_u$1 is a checkable claim. Retrieved claims and existing fact-checks are compared by cosine similarity, and the corpus is indexed with FAISS for sub-second or millisecond-scale lookup (Adler et al., 2019).

CrowdChecked addresses a narrower but operationally important task: detecting whether a social-media claim has already been fact-checked by a professional source. It mines tweet-article pairs from replies or quote-tweets containing Snopes URLs, labels them by distant supervision using Jaccard-based or SBERT-based heuristics, and trains a Siamese bi-encoder based on BERT-base as in Sentence-BERT. Its modified Multiple Negative Ranking loss uses refurbished labels $t_u$2 and weight $t_u$3, with self-adaptive label updates

$t_u$4

after epoch $t_u$5. At inference time, precomputed article embeddings support top-$t_u$6 retrieval, optional LambdaMART re-ranking, and thresholded decisions about whether a claim is previously fact-checked (Hardalov et al., 2022).

FAKTA-derived systems emphasize heterogeneous evidence and stance. Queries are issued against multiple corpora and ranked initially by BM25; a re-ranking function then refines candidates. Related documents advance through a two-level hierarchy that classifies $t_u$7 and then $t_u$8, while sentence-level rationales and lexicon-based cues for subjectivity, sentiment, and bias support aggregation into final verdict probabilities $t_u$9 and $u$0 (Nadeem et al., 2019).

Check-COVID adapts the pipeline to scientific evidence. Claims from mainstream English-language news are paired with sentence-level rationales from CORD-19 abstracts, retrieved by a Vespa search engine using BM25. The rationale selector is a RoBERTa-Large binary classifier over $u$1, and the label predictor is a RoBERTa-Large 3-way classifier over concatenated rationale sentences. The benchmark distinguishes extracted claims from composed claims, a distinction that is consequential because the former are longer and more difficult for retrieval even when downstream label prediction remains comparatively strong once rationales are found (Wang et al., 2023).

5. Evaluation regimes and reported results

Evaluation in Co-FactChecker research is distributed across claim detection, evidence retrieval, veracity classification, explanation quality, collaboration quality, latency, and robustness.

For explainable fact verification, FaxPlainAC’s core ExPred model achieves 79.2% accuracy and 76.5 F1 on the FEVER test set. Early in-lab user trials with $u$2 reported that users could correct spurious evidence 4.3× more quickly than re-labeling from scratch, and that 68% of system errors were repaired in fewer than 2 clicks. Because the system is a demonstration of tooling rather than a new fact-verification model, these numbers are used to illustrate the interaction between explainability and lightweight human correction rather than to define a new benchmark (Zhang et al., 2021).

For real-time journalistic support, held-out news evaluation of sentence-level claim detection reports precision $u$3, recall $u$4, and $u$5. The same pipeline reports average end-to-end latency per article of approximately 150 ms for detection, approximately 20 ms/query for embedding lookup, FAISS search, and thresholding, an overall claim-to-factchecks pipeline of approximately 170 ms, and incremental clustering of approximately 300 ms per claim. The collaborative interface is therefore designed for near-interactive use in live news settings (Adler et al., 2019).

For detection of previously fact-checked claims, CrowdChecked reports on the CLEF’21 CheckThat’21 test set that BM25 achieves $u$6, $u$7, and $u$8; SBERT trained only on CheckThat’21 achieves $u$9, $d_u(n)$0, and $d_u(n)$1; the best prior state of the art reaches $d_u(n)$2, $d_u(n)$3, and $d_u(n)$4; and the Co-FactChecker pipeline using a TF·IDF+SBERT ensemble with re-ranking reaches $d_u(n)$5, $d_u(n)$6, and $d_u(n)$7, an absolute improvement of $d_u(n)$8 MRR, $d_u(n)$9 $u$0, and $u$1 $u$2 (Hardalov et al., 2022).

For scientific fact-checking, Check-COVID reports that, on test end-to-end evaluation ignoring NEI, oracle abstracts yield composed $u$3 and extracted $u$4, whereas Vespa retrieval yields composed $u$5 and extracted $u$6. GPT-3.5 few-shot results on the same test setting reach composed $u$7 and extracted $u$8. The benchmark also reports that numeric and temporal reasoning can reduce $u$9 from approximately 63% to approximately 30% on relevant development subsets (Wang et al., 2023).

OpenFactCheck extends evaluation beyond a single checker. It reports FactQA totals of 6,480 questions across 482 domains and FactBench totals of 4,835 claims with T/F/Unknown labels. Reported examples include Snowball $n$0 scores of 14.5% for LLaMA-2 7B, 19.5% for LLaMA-2 13B, and 34.5% for GPT-4; SelfAware precision of approximately 70%, recall of approximately 30%, and $n$1 of approximately 42%; FreshQA accuracy of 28–40%; and automatic-checker results such as $n$2 and $n$3 for FacTool+GPT-3.5+Serper on Factcheck-Bench (Wang et al., 2024).

The 2026 Co-FactChecker evaluates human-AI collaboration directly. On ExClaim and AmbiguousSnopes, the paper reports gains of +3 F1 points and +6 / +3 EntailmentScore over the best autonomous baseline on each dataset. In a 200-claim ExClaim subsample scored by Prometheus-2, explanation correctness is 4.68 for Co-FactChecker versus 4.56 for multi-turn dialogue, trace correctness is 4.12 versus 3.42, and trace comprehensibility is 3.52 versus 2.48. Human evaluation with 2 professional fact-checkers and 3 NLP researchers over 14 real fact-checks reports win-rates of 43% / 14% / 43% for reasoning and verdict quality, 64% / 36% / 0% for trace usefulness, 53% / 18% / 29% for instruction following, and 64% / 36% / 0% for effort, with overall preference 64% for Co-FactChecker and a practical usefulness rating of mean 3.3/5 (Sahnan et al., 15 Apr 2026).

6. Limitations, misconceptions, and open directions

A persistent misconception is that fact-checking can be reduced either to autonomous model inference or to crowd consensus alone. The literature instead identifies distinct failure modes for both views. In the LRM setting, fully automatic systems are described as lacking the grounding, deep contextual understanding, and ethical judgment of professional fact-checkers, and as prone to black-box decisions, overgeneralization from limited evidence, and unreliable incorporation of expert feedback in multi-turn dialogue (Sahnan et al., 15 Apr 2026). In the community-moderation setting, crowd processes are described as potentially scalable and transparent, yet vulnerable to brigading, selection bias, echo chambers, and delays introduced by cross-ideological consensus, with community efforts not replacing professional fact-checkers (Augenstein et al., 26 May 2025).

Interview-based studies add organizational and epistemic constraints. Fact-checkers report bottlenecks in sifting documentary sources, difficulties in finding and vetting credible experts, the absence of metadata for identifying original sources on social media, and reluctance to trust black-box tools. One interview summary explicitly states that “Explainability for fact-checkers would tend to be extracts, quotes and snippets… you don’t want to reduce it down to a number.” The same work highlights amplification risk: publication itself can send readers back toward the misinformation being checked (Procter et al., 2023).

Technical limitations remain prominent in retrieval-heavy settings. Check-COVID identifies retrieval over a rapidly evolving scientific corpus as a chief bottleneck, especially for complex extracted claims, and notes error propagation from rationale selection to label prediction, limited domain transfer from SciFact to news claims, and severe degradation under numeric and temporal reasoning demands. Its recommended responses include hybrid BM25 plus dense retrieval, domain-adaptive fine-tuning, joint rationale-label modeling, external medical ontologies such as UMLS and MeSH, richer evidence aggregation, explicit uncertainty handling for NOTENOUGHINFO, and broader claim diversification beyond Western English-language news (Wang et al., 2023).

The 2026 Co-FactChecker identifies additional limitations specific to trace-editing: incorrect interpretation of feedback can derail reasoning irrecoverably; the verifier often fails to recognize when evidence is insufficient; and fundamental reasoning mistakes, such as conflating reporting with direct quotes, remain common. Future directions include feedback-driven retrieval, stronger reasoning-model pre-training or fine-tuning on fact-checking practices, and application to other expert domains such as legal analysis and scientific peer review (Sahnan et al., 15 Apr 2026).

More broadly, the open research agenda around Co-FactChecker includes democratic truth versus objective accuracy, bias and diversity quantification, adversarial resilience to coordinated voting, psychological welfare and labor ethics for volunteer annotators, cross-platform interoperability of note formats and APIs, and AI-agent augmentation through LLM-based “Supernotes” or note-suggestion systems. This suggests that Co-FactChecker is best understood not as a settled product category, but as an evolving research program on how explicit evidence, modular automation, and accountable human oversight can be combined in claim verification (Augenstein et al., 26 May 2025).