---
title: 'QuanTemp++: Benchmark for Numerical Fact-Checking'
url: https://www.emergentmind.com/topics/quantemp
type: topic
---

# QuanTemp++: Benchmark for Numerical Fact-Checking

QuanTemp++ is a benchmark for open-domain numerical fact-checking built around natural, real-world numerical claims, an open-domain corpus, and claim-specific relevant evidence collected through a decomposition process intended to approximately emulate the search workflow of a human fact-checker. It extends QuanTemp by retaining its focus on naturally occurring numerical claims while addressing a major realism problem—temporal leakage—and by explicitly treating retrieval quality, especially decomposition-aware retrieval, as foundational to downstream veracity prediction. In the benchmark, claims are verified against evidence retrieved from the open web under a three-way label space: **True**, **False**, and **Conflicting** [2510.22055].

## 1. Position within numerical fact-checking research

QuanTemp++ was introduced against the background of automatic fact-checking benchmarks that either are dominated by synthetic Wikipedia-style claims, do not focus on numerical claims, leak gold fact-check articles into the evidence pool, or include evidence published after the claim, making retrieval unrealistically easy. The benchmark is designed to evaluate systems that must retrieve and verify evidence from the open web **without access to future evidence** and **without seeing the original professional fact-check article as evidence** [2510.22055].

The immediate predecessor, QuanTemp, established the broader problem setting: real-world, open-domain fact-checking of numerical claims involving temporal, statistical, interval, and comparative phenomena, using claims sourced from fact-checking organizations and evidence retrieved from the web [2403.17169]. QuanTemp++ preserves the original claim set, labels, and split structure, but reconstructs the evidence side more conservatively and more explicitly around fact-checker-style information seeking [2510.22055].

The benchmark’s central premise is that numerical claims are unusually difficult because they often require explicit numbers, implicit comparisons, temporal constraints, aggregation or threshold reasoning, and multi-aspect evidence gathering. A numerical claim may mention a date, a place, and a numerical or legal threshold, all of which must align for correct verification. This motivates the benchmark’s emphasis on decomposition-aware retrieval rather than direct matching of the raw claim to the web [2510.22055].

## 2. Corpus, labels, and construction protocol

QuanTemp++ reuses claims from QuanTemp, which originally harvested naturally occurring numerical claims from fact-checking websites. The benchmark contains **9,935 training claims**, **3,084 validation claims**, and **2,495 test claims**, for a total of **15,514 claims**. The veracity labels are **T = True**, **F = False**, and **C = Conflicting** [2510.22055].

The evidence side is newly constructed. After decomposition, filtered sub-queries are issued to public web search engines. The construction pipeline applies four safeguards and curation stages: **Gold evidence leakage prevention** by filtering evidence from **150+ fact-checking domains**; **temporal leakage prevention at collection time** using a search-engine **before:** filter so that returned results are published before the claim date; **deduplication**; and **diversity filtering** with MMR to retain diverse evidence among retrieved items. Relevant retained snippets become the benchmark’s **qrels**, while noisier retrieved snippets are preserved as distractors [2510.22055].

QuanTemp++ includes an open-domain evidence corpus of **165.7k evidence records**. The corpus contains both claim-linked relevant evidence snippets and distractors gathered during web search, and retrieval is evaluated over this **full corpus**, not over a pre-pruned gold evidence set [2510.22055].

The benchmark explicitly addresses temporal leakage in two places. First, corpus creation restricts search results to pages published before the claim date. Second, during open-domain retrieval evaluation, the retrieved top-\(k\) documents are again filtered to retain only those published before the claim publication date, because evidence collected for other claims may still be newer and semantically related. The authors explicitly note that this second form of leakage can still occur in shared corpora [2510.22055].

| Component | Value |
|---|---:|
| Training claims | 9,935 |
| Validation claims | 3,084 |
| Test claims | 2,495 |
| Total claims | 15,514 |
| Evidence corpus | 165.7k evidence records |
| Labels | True / False / Conflicting |

The paper does not report exact class counts for QuanTemp++, but the majority baseline strongly suggests that the dataset is **False-heavy**: the naive classifier obtains **57.03 Accuracy** with **F1-F = 72.64** and **F1-T = 0.00, F1-C = 0.00** [2510.22055].

## 3. FCDecomp and decomposition-aware evidence gathering

The benchmark’s distinctive methodological contribution is **FCDecomp**, a claim decomposition method used during data construction to emulate a human fact-checker’s search process. Instead of generating search queries only from the claim text, FCDecomp uses the claim \(C\), the professional fact-checker’s justification document \(J\), and an LLM prompt with a manually written example to generate short, diverse search queries that target both explicit and implicit aspects of the claim. This decomposition is used for data construction and is not assumed to be available at test time [2510.22055].

The benchmark defines query filtering with MMR relative to both the claim and the justification:

\[
s_{q_i} = \alpha \cdot \text{MMR}(q_{i}, C,\lambda) + (1-\alpha) \cdot \text{MMR}(q_{i}, J,\lambda) ; \quad Q = \{ q_i \mid s_{q_i} > \gamma \text{ for } i \in \{1, k\} \}
\]

Here \(C\) is the claim, \(J\) is the justification document, \(q_i\) is a generated query, \(\lambda\) is the MMR diversification factor, \(\alpha\) balances claim-versus-justification relevance, and \(\gamma\) is the selection threshold. FCDecomp outputs a variable number of queries, with an **average of 6.76 queries per claim** [2510.22055].

A representative example in the paper uses the claim: *“Prime Minister Narendra Modi breached the election protocol by addressing a rally in Howrah on April 6.”* FCDecomp produces sub-queries including *Modi rally in Howrah 2021*, *Prime Minister Narendra Modi rally on April 6*, *election in Howrah 2021*, *locations of Howrah voting*, and *silence period affects campaigning and media coverage of elections*. The intended behavior is therefore not mere paraphrase, but decomposition into event, date, quantitative or legal condition, and contextual rule retrieval needs [2510.22055].

Because FCDecomp depends on justification documents unavailable at deployment time, the paper distills its outputs into a smaller claim-only generator, **QGen**. A smaller LM \(f_\theta\), exemplified with **FLAN-T5-LARGE** (780M), is trained on training claims and LLM-generated query lists using next-token prediction:

\[
\mathcal{L}(\theta) = - \sum_{(c, Q) \in \mathcal{D}_{\text{train}} \sum_{t=1}^{|Q|} \log P_\theta \big(y_t \mid y_{<t}, c \big)
\]

In this formulation, \(c\) is the claim, \(Q\) is the target query list, and \(y_t\) is the \(t\)-th token of the target query sequence [2510.22055].

## 4. Supported tasks and evaluation protocol

QuanTemp++ supports two linked tasks: **open-domain evidence retrieval** and **evidence-based claim verification**. In retrieval, given a claim, a system must retrieve relevant evidence from the **165.7k-record** corpus, using either the raw claim or decomposition-generated sub-queries. Retrieval is evaluated with **NDCG@10**, **Recall@10**, **Recall@100**, and **MRR**. In verification, given a claim and retrieved evidence, the system predicts a three-way veracity label and is evaluated with **Accuracy**, **Per-class F1** (**F1-T**, **F1-F**, **F1-C**), **Weighted-F1 (W-F1)**, and **Macro-F1 (M-F1)** [2510.22055].

The experimental pipeline compares several query planning modes: **Claim-Only**, **FCDecomp**, **ClaimDecomp**, **PgmFC**, and **QGen**. For retrieval, the authors compare **BM25**, **ANCE**, **TAS-B**, and **Contriever**, ultimately choosing **Contriever** after validation comparisons. Per-query evidence is aggregated with rank fusion; **CombMAX-Norm** is reported as the best fusion method, while **CombSUM** and **CombMAX** are also better than naive merging [2510.22055].

For verification, the main NLI model is **roberta-large-mnli** fine-tuned for veracity prediction. The paper also evaluates **gpt-3.5-turbo** as an LLM verifier. Retrieved evidence depth is varied over \(k \in \{1,3,5,7,10\}\), and **\(k=3\)** gives the best validation NLI performance. Reported NLI training settings are **Adam**, learning rate \(2 \times 10^{-5}\), and weight decay \(1 \times e^{-5}\) as written in the paper; the LLM verification temperature is **0.1** [2510.22055].

Before full open-domain retrieval is evaluated, the paper first tests whether the collected evidence is intrinsically useful by pairing each claim directly with its linked evidence. This isolates evidence quality from retrieval quality [2510.22055].

## 5. Empirical results and the retrieval–verification gap

The evidence-quality experiment shows that the collected QuanTemp++ evidence is substantially better than claim-only prediction and approaches the “gold justification” upper bound. In particular, **roberta-mnli-qtemp++** reaches **M-F1 59.11**, compared with **M-F1 62.85** for **roberta-mnli-gold**, while **roberta-mnli-claim-no-ev** obtains **M-F1 50.67**. This indicates that the benchmark’s evidence collection procedure yields evidence that is close to human-authored justification evidence in downstream utility [2510.22055].

Open-domain retrieval results show a clear advantage for FCDecomp. **Claim-Only** achieves **NDCG@10 = 0.33**, **Recall@10 = 0.32**, **Recall@100 = 0.51**, and **MRR = 0.54**, whereas **FCDecomp** reaches **NDCG@10 = 0.57**, **Recall@10 = 0.54**, **Recall@100 = 0.82**, and **MRR = 0.68**. The paper treats FCDecomp as an upper bound because it uses justification documents unavailable at deployment time. Temporal filtering lowers some scores slightly but preserves realism [2510.22055].

Downstream verification exhibits a smaller but still important effect. With retrieved evidence, **Claim-Only** reaches **Accuracy 66.53**, **W-F1 64.03**, and **M-F1 55.28**. **FCDecomp** reaches **Accuracy 66.45**, **W-F1 66.8**, and **M-F1 58.92**, and **QGen** reaches **Accuracy 66.25**, **W-F1 65.41**, and **M-F1 57.28**. The improvements in **Macro-F1** for **FCDecomp** and **QGen** over **Claim-Only** are reported as statistically significant at the 0.05 level [2510.22055].

A particularly important result concerns the **Conflicting** class. **Claim-Only** yields **F1-C = 31.02**, whereas **FCDecomp** yields **F1-C = 50.25**. **QGen** and **PgmFC** also improve this class, reaching **F1-C = 36.95** and **F1-C = 37.15** respectively. The paper attributes this to the heterogeneous evidence needs of conflicting claims, whose parts may evaluate differently [2510.22055].

The paper’s most diagnostic conclusion is the **retrieval–verifier gap**. FCDecomp greatly improves retrieval quality—e.g., **Recall@10: 0.54 vs 0.32** and **NDCG@10: 0.57 vs 0.33**—but the downstream verification gain is much smaller, e.g. **M-F1: 58.92 vs 55.28**. The authors explain this gap by the continued weakness of current NLI and LLM verifiers on numerical contextualization, ranges and thresholds, comparison reasoning, temporal relations, and multi-evidence integration [2510.22055].

## 6. Annotation quality, limitations, and subsequent significance

QuanTemp++ is built with weak supervision at scale, followed by targeted qualitative assessment. For qualitative analysis, the authors sample **50 representative claims** and have **3 researchers** annotate generated queries and resulting evidence snippets for **Completeness** (1–5), **Redundancy** (1–5, lower better), and **Relevance**. Inter-annotator agreement by **Fleiss’ \(\kappa\)** is **0.61 / 0.66** for completeness, **0.40 / 0.45** for redundancy, and **0.58 / 0.67** for relevance, for queries and evidence snippets respectively. Average ratings are **90%** query relevance precision, **4.49** query completeness, **2.4** query redundancy, **3.94** evidence completeness, and **2.9** evidence redundancy [2510.22055].

Several limitations are explicit. **FCDecomp itself is not deployable at test time**, because it uses professional fact-check justifications. The practical approximation, **QGen**, still trails the FCDecomp upper bound. Verification models remain weak at numerical reasoning, so retrieval gains are bottlenecked downstream. Dataset construction is weakly supervised with only partial manual quality assessment, and exact class distribution and some temporal-span details are not fully reported [2510.22055].

Within the longer trajectory of QuanTemp-style research, QuanTemp++ sharpens the predecessor’s earlier finding that numerical fact-checking is a retrieval-heavy, temporally grounded, composition-sensitive reasoning problem rather than a conventional NLI task. QuanTemp had already shown strong gains from decomposition-aware retrieval and reported a best **Macro-F1 of 58.32** on its unified evidence setting, with **comparison** and **Conflicting** cases remaining especially difficult [2403.17169]. QuanTemp++ adds a stricter evidence regime and identifies temporal leakage prevention as a first-class evaluation requirement [2510.22055].

Subsequent work on QuanTemp-style pipelines is broadly consistent with this diagnosis. DS@GT at CheckThat! 2025 found that a longer context window did not improve validation macro-F1 and that right-to-left tokenization degraded performance on numerical veracity prediction, while weaker retrieval notably reduced **Conflicting F1**. This suggests that evidence quality remains the dominant bottleneck even when longer-context encoders or number-aware tokenization strategies are introduced [2507.06195].

The benchmark’s intended uses therefore extend beyond end-to-end fact verification. The paper explicitly identifies it as useful for studying decomposition or query planning, temporal retrieval, evidence fusion, numerical reasoning in NLI, and retrieval-aware verifier training, and it suggests future work on iterative decomposition and retrieval, using FCDecomp queries as supervision, and optimizing decomposition for downstream utility rather than only for retrieval relevance [2510.22055].

Source: https://www.emergentmind.com/topics/quantemp