QuanTemp++: Benchmark for Numerical Fact-Checking

• QuanTemp++ is a benchmark for numerical fact-checking that evaluates decomposition-aware retrieval and evidence verification to address temporal leakage.
• It incorporates 15.5k natural numerical claims and an extensive evidence corpus of 165.7k records labeled as True, False, or Conflicting.
• FCDecomp, a key innovation, decomposes claims into targeted sub-queries simulating fact-checker workflows to boost evidence retrieval and downstream verification.

QuanTemp++ is a benchmark for open-domain numerical fact-checking built around natural, real-world numerical claims, an open-domain corpus, and claim-specific relevant evidence collected through a decomposition process intended to approximately emulate the search workflow of a human fact-checker. It extends QuanTemp by retaining its focus on naturally occurring numerical claims while addressing a major realism problem—temporal leakage—and by explicitly treating retrieval quality, especially decomposition-aware retrieval, as foundational to downstream veracity prediction. In the benchmark, claims are verified against evidence retrieved from the open web under a three-way label space: True, False, and Conflicting (Venktesh et al., 24 Oct 2025).

1. Position within numerical fact-checking research

QuanTemp++ was introduced against the background of automatic fact-checking benchmarks that either are dominated by synthetic Wikipedia-style claims, do not focus on numerical claims, leak gold fact-check articles into the evidence pool, or include evidence published after the claim, making retrieval unrealistically easy. The benchmark is designed to evaluate systems that must retrieve and verify evidence from the open web without access to future evidence and without seeing the original professional fact-check article as evidence (Venktesh et al., 24 Oct 2025).

The immediate predecessor, QuanTemp, established the broader problem setting: real-world, open-domain fact-checking of numerical claims involving temporal, statistical, interval, and comparative phenomena, using claims sourced from fact-checking organizations and evidence retrieved from the web (V et al., 2024). QuanTemp++ preserves the original claim set, labels, and split structure, but reconstructs the evidence side more conservatively and more explicitly around fact-checker-style information seeking (Venktesh et al., 24 Oct 2025).

The benchmark’s central premise is that numerical claims are unusually difficult because they often require explicit numbers, implicit comparisons, temporal constraints, aggregation or threshold reasoning, and multi-aspect evidence gathering. A numerical claim may mention a date, a place, and a numerical or legal threshold, all of which must align for correct verification. This motivates the benchmark’s emphasis on decomposition-aware retrieval rather than direct matching of the raw claim to the web (Venktesh et al., 24 Oct 2025).

2. Corpus, labels, and construction protocol

QuanTemp++ reuses claims from QuanTemp, which originally harvested naturally occurring numerical claims from fact-checking websites. The benchmark contains 9,935 training claims, 3,084 validation claims, and 2,495 test claims, for a total of 15,514 claims. The veracity labels are T = True, F = False, and C = Conflicting (Venktesh et al., 24 Oct 2025).

The evidence side is newly constructed. After decomposition, filtered sub-queries are issued to public web search engines. The construction pipeline applies four safeguards and curation stages: Gold evidence leakage prevention by filtering evidence from 150+ fact-checking domains; temporal leakage prevention at collection time using a search-engine before: filter so that returned results are published before the claim date; deduplication; and diversity filtering with MMR to retain diverse evidence among retrieved items. Relevant retained snippets become the benchmark’s qrels, while noisier retrieved snippets are preserved as distractors (Venktesh et al., 24 Oct 2025).

QuanTemp++ includes an open-domain evidence corpus of 165.7k evidence records. The corpus contains both claim-linked relevant evidence snippets and distractors gathered during web search, and retrieval is evaluated over this full corpus, not over a pre-pruned gold evidence set (Venktesh et al., 24 Oct 2025).

The benchmark explicitly addresses temporal leakage in two places. First, corpus creation restricts search results to pages published before the claim date. Second, during open-domain retrieval evaluation, the retrieved top-$k$ documents are again filtered to retain only those published before the claim publication date, because evidence collected for other claims may still be newer and semantically related. The authors explicitly note that this second form of leakage can still occur in shared corpora (Venktesh et al., 24 Oct 2025).

Component	Value
Training claims	9,935
Validation claims	3,084
Test claims	2,495
Total claims	15,514
Evidence corpus	165.7k evidence records
Labels	True / False / Conflicting

The paper does not report exact class counts for QuanTemp++, but the majority baseline strongly suggests that the dataset is False-heavy: the naive classifier obtains 57.03 Accuracy with F1-F = 72.64 and F1-T = 0.00, F1-C = 0.00 (Venktesh et al., 24 Oct 2025).

3. FCDecomp and decomposition-aware evidence gathering

The benchmark’s distinctive methodological contribution is FCDecomp, a claim decomposition method used during data construction to emulate a human fact-checker’s search process. Instead of generating search queries only from the claim text, FCDecomp uses the claim $C$, the professional fact-checker’s justification document $J$, and an LLM prompt with a manually written example to generate short, diverse search queries that target both explicit and implicit aspects of the claim. This decomposition is used for data construction and is not assumed to be available at test time (Venktesh et al., 24 Oct 2025).

The benchmark defines query filtering with MMR relative to both the claim and the justification:

$$s_{q_i} = \alpha \cdot \text{MMR}(q_{i}, C,\lambda) + (1-\alpha) \cdot \text{MMR}(q_{i}, J,\lambda) ; \quad Q = \{ q_i \mid s_{q_i} > \gamma \text{ for } i \in \{1, k\} \}$$

Here $C$ is the claim, $J$ is the justification document, $q_i$ is a generated query, $\lambda$ is the MMR diversification factor, $\alpha$ balances claim-versus-justification relevance, and $\gamma$ is the selection threshold. FCDecomp outputs a variable number of queries, with an average of 6.76 queries per claim (Venktesh et al., 24 Oct 2025).

A representative example in the paper uses the claim: “Prime Minister Narendra Modi breached the election protocol by addressing a rally in Howrah on April 6.” FCDecomp produces sub-queries including Modi rally in Howrah 2021, Prime Minister Narendra Modi rally on April 6, election in Howrah 2021, locations of Howrah voting, and silence period affects campaigning and media coverage of elections. The intended behavior is therefore not mere paraphrase, but decomposition into event, date, quantitative or legal condition, and contextual rule retrieval needs (Venktesh et al., 24 Oct 2025).

Because FCDecomp depends on justification documents unavailable at deployment time, the paper distills its outputs into a smaller claim-only generator, QGen. A smaller LM $C$0, exemplified with FLAN-T5-LARGE (780M), is trained on training claims and LLM-generated query lists using next-token prediction:

$C$1

In this formulation, $C$2 is the claim, $C$3 is the target query list, and $C$4 is the $C$5-th token of the target query sequence (Venktesh et al., 24 Oct 2025).

4. Supported tasks and evaluation protocol

QuanTemp++ supports two linked tasks: open-domain evidence retrieval and evidence-based claim verification. In retrieval, given a claim, a system must retrieve relevant evidence from the 165.7k-record corpus, using either the raw claim or decomposition-generated sub-queries. Retrieval is evaluated with NDCG@10, Recall@10, Recall@100, and MRR. In verification, given a claim and retrieved evidence, the system predicts a three-way veracity label and is evaluated with Accuracy, Per-class F1 (F1-T, F1-F, F1-C), Weighted-F1 (W-F1), and Macro-F1 (M-F1) (Venktesh et al., 24 Oct 2025).

The experimental pipeline compares several query planning modes: Claim-Only, FCDecomp, ClaimDecomp, PgmFC, and QGen. For retrieval, the authors compare BM25, ANCE, TAS-B, and Contriever, ultimately choosing Contriever after validation comparisons. Per-query evidence is aggregated with rank fusion; CombMAX-Norm is reported as the best fusion method, while CombSUM and CombMAX are also better than naive merging (Venktesh et al., 24 Oct 2025).

For verification, the main NLI model is roberta-large-mnli fine-tuned for veracity prediction. The paper also evaluates gpt-3.5-turbo as an LLM verifier. Retrieved evidence depth is varied over $C$6, and $C$7 gives the best validation NLI performance. Reported NLI training settings are Adam, learning rate $C$8, and weight decay $C$9 as written in the paper; the LLM verification temperature is 0.1 (Venktesh et al., 24 Oct 2025).

Before full open-domain retrieval is evaluated, the paper first tests whether the collected evidence is intrinsically useful by pairing each claim directly with its linked evidence. This isolates evidence quality from retrieval quality (Venktesh et al., 24 Oct 2025).

5. Empirical results and the retrieval–verification gap

The evidence-quality experiment shows that the collected QuanTemp++ evidence is substantially better than claim-only prediction and approaches the “gold justification” upper bound. In particular, roberta-mnli-qtemp++ reaches M-F1 59.11, compared with M-F1 62.85 for roberta-mnli-gold, while roberta-mnli-claim-no-ev obtains M-F1 50.67. This indicates that the benchmark’s evidence collection procedure yields evidence that is close to human-authored justification evidence in downstream utility (Venktesh et al., 24 Oct 2025).

Open-domain retrieval results show a clear advantage for FCDecomp. Claim-Only achieves NDCG@10 = 0.33, Recall@10 = 0.32, Recall@100 = 0.51, and MRR = 0.54, whereas FCDecomp reaches NDCG@10 = 0.57, Recall@10 = 0.54, Recall@100 = 0.82, and MRR = 0.68. The paper treats FCDecomp as an upper bound because it uses justification documents unavailable at deployment time. Temporal filtering lowers some scores slightly but preserves realism (Venktesh et al., 24 Oct 2025).

Downstream verification exhibits a smaller but still important effect. With retrieved evidence, Claim-Only reaches Accuracy 66.53, W-F1 64.03, and M-F1 55.28. FCDecomp reaches Accuracy 66.45, W-F1 66.8, and M-F1 58.92, and QGen reaches Accuracy 66.25, W-F1 65.41, and M-F1 57.28. The improvements in Macro-F1 for FCDecomp and QGen over Claim-Only are reported as statistically significant at the 0.05 level (Venktesh et al., 24 Oct 2025).

A particularly important result concerns the Conflicting class. Claim-Only yields F1-C = 31.02, whereas FCDecomp yields F1-C = 50.25. QGen and PgmFC also improve this class, reaching F1-C = 36.95 and F1-C = 37.15 respectively. The paper attributes this to the heterogeneous evidence needs of conflicting claims, whose parts may evaluate differently (Venktesh et al., 24 Oct 2025).

The paper’s most diagnostic conclusion is the retrieval–verifier gap. FCDecomp greatly improves retrieval quality—e.g., Recall@10: 0.54 vs 0.32 and NDCG@10: 0.57 vs 0.33—but the downstream verification gain is much smaller, e.g. M-F1: 58.92 vs 55.28. The authors explain this gap by the continued weakness of current NLI and LLM verifiers on numerical contextualization, ranges and thresholds, comparison reasoning, temporal relations, and multi-evidence integration (Venktesh et al., 24 Oct 2025).

6. Annotation quality, limitations, and subsequent significance

QuanTemp++ is built with weak supervision at scale, followed by targeted qualitative assessment. For qualitative analysis, the authors sample 50 representative claims and have 3 researchers annotate generated queries and resulting evidence snippets for Completeness (1–5), Redundancy (1–5, lower better), and Relevance. Inter-annotator agreement by Fleiss’ $J$0 is 0.61 / 0.66 for completeness, 0.40 / 0.45 for redundancy, and 0.58 / 0.67 for relevance, for queries and evidence snippets respectively. Average ratings are 90% query relevance precision, 4.49 query completeness, 2.4 query redundancy, 3.94 evidence completeness, and 2.9 evidence redundancy (Venktesh et al., 24 Oct 2025).

Several limitations are explicit. FCDecomp itself is not deployable at test time, because it uses professional fact-check justifications. The practical approximation, QGen, still trails the FCDecomp upper bound. Verification models remain weak at numerical reasoning, so retrieval gains are bottlenecked downstream. Dataset construction is weakly supervised with only partial manual quality assessment, and exact class distribution and some temporal-span details are not fully reported (Venktesh et al., 24 Oct 2025).

Within the longer trajectory of QuanTemp-style research, QuanTemp++ sharpens the predecessor’s earlier finding that numerical fact-checking is a retrieval-heavy, temporally grounded, composition-sensitive reasoning problem rather than a conventional NLI task. QuanTemp had already shown strong gains from decomposition-aware retrieval and reported a best Macro-F1 of 58.32 on its unified evidence setting, with comparison and Conflicting cases remaining especially difficult (V et al., 2024). QuanTemp++ adds a stricter evidence regime and identifies temporal leakage prevention as a first-class evaluation requirement (Venktesh et al., 24 Oct 2025).

Subsequent work on QuanTemp-style pipelines is broadly consistent with this diagnosis. DS@GT at CheckThat! 2025 found that a longer context window did not improve validation macro-F1 and that right-to-left tokenization degraded performance on numerical veracity prediction, while weaker retrieval notably reduced Conflicting F1. This suggests that evidence quality remains the dominant bottleneck even when longer-context encoders or number-aware tokenization strategies are introduced (Heil et al., 8 Jul 2025).

The benchmark’s intended uses therefore extend beyond end-to-end fact verification. The paper explicitly identifies it as useful for studying decomposition or query planning, temporal retrieval, evidence fusion, numerical reasoning in NLI, and retrieval-aware verifier training, and it suggests future work on iterative decomposition and retrieval, using FCDecomp queries as supervision, and optimizing decomposition for downstream utility rather than only for retrieval relevance (Venktesh et al., 24 Oct 2025).