Numerical Claim Detection

• Numerical claim detection is the automated process of identifying and verifying quantitative or temporal assertions in natural language text.
• It employs multi-stage retrieval with BM25 and neural reranking to classify claims as True, False, or Conflicting.
• Applications include financial market analysis and public health, while challenges remain in evidence retrieval and dataset diversity.

Numerical claim detection is the automated identification and verification of statements involving explicit quantities, comparisons, intervals, or temporal references in natural language text. Unlike general claim detection, which seeks any factual assertion, numerical claim detection focuses on claims whose validity hinges upon quantitative or temporal reasoning. This area is distinguished by its heightened susceptibility to the “numeric-truth effect”—the documented phenomenon where readers grant greater initial credibility to numerically-anchored statements.

1. Problem Definition and Scope

Numerical claim detection comprises two subtasks: recognizing the presence and structure of numerical expressions in statements, and determining the veracity of claims grounded in quantities, thresholds, or time frames. Claims in this category exhibit forms such as:

• Simple statistics: “Vaccination coverage in Region B is at least 75%.”
• Comparisons with a temporal reference: “Country A’s unemployment rate has doubled since 2005.”
• Interval and trend analysis: “Between 1990 and 2000, average life expectancy increased by 5 years.”

The CheckThat! 2025 Task 3 constrains the task to three-way veracity classification: True, False, or Conflicting, based on web-sourced fact-checking evidence. “Conflicting” indicates genuine ambiguity or mixed evidential support. By delineating numerical claims from generalized assertions, these studies systematically treat the unique inferential and evidentiary demands of this claim type (Heil et al., 8 Jul 2025).

2. Datasets and Annotation Frameworks

Datasets Tailored for Numerical Claims

The QuanTemp dataset underpins the current state-of-the-art in open-domain numerical fact verification. In its English subset, QuanTemp contains:

• 15,514 labeled claims (18.8% True, 57.9% False, 23.3% Conflicting)
• 432,320 fact-checked evidence snippets sourced from 45 fact-checking organizations
• Four semantic categories: statistical, comparative, interval, temporal

For the financial domain, a large-scale dataset of “numeric-financial” sentences (over 2 million from analyst reports, over 40,000 from earnings call transcripts) has been constructed by filtering documents for the co-occurrence of numerals and financial terminology. Expert annotation splits sentences into in-claim (forecasts/speculations) and out-of-claim (historical/confirmed) classes, with extremely high inter-annotator agreement: 99.21% (analyst reports) and 95.78% (earnings calls) (Shah et al., 2024).

Claim Types and Labeling

Numeric claim identification in financial text is operationalized as:

• “In-claim”—speculative financial forecasts (e.g., future earnings, projected growth)
• “Out-of-claim”—established or historical numeric facts

This binary label, while less granular than the True/False/Conflicting taxonomy, enables downstream analyses of sentiment and market impact (Shah et al., 2024).

3. Evidence Retrieval and Pipeline Design

To construct high-quality evidence sets for numerical verification, recent systems implement multistage retrieval pipelines:

Stage	Methodology	Outputs
Claim Decomposition	GPT-4o-mini	Up to 3 “sub-questions” per input claim
Sparse Retrieval	BM25	Top 50 snippets per sub-question
Neural Reranking	Cross-Encoder	ms-marco-MiniLM-L-12-v2 relevance scores
Evidence Selection	Ablation-dependent	Top 1–3 snippets merged per sub-claim

The initial decomposition uses GPT-based prompting with low temperature settings to break complex statements into tractable queries. BM25 is parameterized with $k_1 \approx 1.2$, $b \approx 0.75$ for sparse scoring:

$$\text{score}_{BM25}(q,d) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{f(t,d) \cdot (k_1+1)}{f(t,d) + k_1 (1 - b + b \cdot |d|/\text{avgdl})}$$

A fine-tuned cross-encoder evaluates claim-evidence pairs jointly for semantic relevance, after which the most highly ranked evidence is selected (1–3 snippets per sub-claim) (Heil et al., 8 Jul 2025).

4. Modeling Strategies and Training

Transformer Architecture and Input Encoding

Veracity classification leverages ModernBERT—a BERT-family transformer supporting up to 8,192 tokens via rotary positional embeddings and a local/global attention mechanism. To analyze the impact of input length and numeric representation, three input conditions are explored:

• Short-Context / L2R: 256 tokens, top 1 evidence, standard left-to-right tokenization
• Long-Context / L2R: 1,024 tokens, top 3 evidences, standard tokenization
• Long-Context / R2L: Same as above, but all digit tokens are reversed prior to byte-pair encoding (right-to-left, R2L)

R2L tokenization is hypothesized to improve handling of fine-grained numeric distinctions by exposing least significant digits first (as found beneficial in arithmetic reasoning tasks).

Loss Functions and Parameter-Efficient Adaptation

• Main loss: categorical cross-entropy over True / False / Conflicting labels
• Focal-loss variant: $\mathcal{L}_{FL} = - (1 - p_t)^\gamma \log p_t$, with $\gamma=2$
• Parameter-efficient fine-tuning (PEFT): LoRA, updating only 1% of model weights

Optimization employs AdamW with $2 \times 10^{-5}$ learning rate and early stopping after two epochs without improvement. Batch sizes are tuned to available GPU memory (RTX 6000 to H100) (Heil et al., 8 Jul 2025).

In the financial claim detection context, fine-tuned transformer PLMs (e.g., RoBERTa-large, BERT-base, domain-specific variants like FinBERT) outperform weakly supervised approaches, yet require labeled data. Weak-supervision exploits 17 expert-designed labeling functions aggregated with a hierarchical, weighted majority-vote scheme; this achieves $F_1 \approx 0.93$ on in-domain binary tasks (Shah et al., 2024).

5. Empirical Results and Bottleneck Analysis

Comparative Performance

Open-domain numerical fact verification experiments reveal:

Configuration	Macro-F1 (Validation)
MathRoBERTa (organizer’s splits)	0.56
Recreated pipeline “Our-Data”	0.52
Short-Context / L2R	0.52
Long-Context / L2R	0.52
Short-Context / R2L	0.45
Long-Context / R2L	0.47
Submission (ModernBERT, L2R)	0.57
LoRA PEFT	0.49
Focal-Loss	0.57

Adding more evidence snippets/longer context does not exceed short-context baselines. While train $F_1$ increases (0.50 → 0.64), validation remains capped at 0.52—diagnosing overfitting to spurious or weak evidence. R2L tokenization, despite prior successes in arithmetic reasoning, degrades macro-F1 by 5–7 points, indicating that NLI-based fact-checking relies primarily on semantic alignment, not digit ordering (Heil et al., 8 Jul 2025).

Error analysis demonstrates that the lower precision for True and Conflicting cases (test macro-F1: False=0.80, Conflicting=0.39, True=0.38) derives from noisy or tangential evidence snippets. Scaling context length simply propagates these retrieval errors into the model.

In the financial detection regime, weak-supervision model performance ($F_1$≈0.93) nearly matches fine-tuned RoBERTa-large ($F_1$=0.96), far surpassing baseline Snorkel and majority-vote aggregators (both $b \approx 0.75$0≈0.43). Zero-shot LLMs underperform (Falcon-7B: $b \approx 0.75$1=0.42, Llama-2-70B: $b \approx 0.75$2=0.73), though few-shot settings provide partial improvements (Shah et al., 2024).

6. Practical Applications and Limitations

Financial Market Analysis

The detection of numerical claims underlies analytic pipelines for financial market intelligence. By tagging in-claim sentences and classifying sentiment, a document-level “optimism” score can be constructed:

$b \approx 0.75$3

where $b \approx 0.75$4 and $b \approx 0.75$5 denote counts of positive/negative in-claim sentences, and $b \approx 0.75$6 is total sentences in document $b \approx 0.75$7.

Regression of post-event returns and earnings surprises on optimism reveals strong, consistent predictive effects ($b \approx 0.75$8). A naive trading strategy (long if optimism below mean, short otherwise) achieves ≈81% directional accuracy over a 60-day window, illustrating the material utility of numerical claim detection for downstream financial modeling (Shah et al., 2024).

Limitations

Outstanding limitations include stagnant retrieval accuracy—identified as the principal bottleneck for veracity classification—domain/geography-limited datasets, unexplored label aggregation frameworks (e.g., COSINE), omission of high-frequency and multimodal sources (audio, news), and the disregard of transaction costs in market strategies.

7. Future Directions and Recommendations

Current findings indicate further gains require:

• Hybridizing sparse and dense retrieval (e.g., BM25, dense vector search) to broaden evidential scope
• Incorporating LLM reranking to better encode nuanced, context-specific numeric entailment
• Normalization of numeric and temporal expressions (e.g., “Q3 2020” $b \approx 0.75$9 date intervals)
• Adopting ensemble NLI classifiers to mitigate individual model biases

In the financial setting, future work will extend to multimodal data fusion (e.g., integrating textual, audio, and visual market signals), experiment with confidence-based or contrastive-augmented weak supervision, and rigorously account for trading frictions and demographic bias.

A plausible implication is that, for both open-domain and domain-specific numerical claim detection, research effort is best directed towards retrieval and evidence normalization, rather than further increases in model input length or alternative numeric tokenizations.

• DS@GT at CheckThat! 2025: Evaluating Context and Tokenization Strategies for Numerical Fact Verification (Heil et al., 8 Jul 2025)
• Numerical Claim Detection in Finance: A New Financial Dataset, Weak-Supervision Model, and Market Analysis (Shah et al., 2024)