CheckThat! 2025 Task 2: Multilingual Claim Normalization

• The paper presents a multilingual claim normalization framework that converts noisy social media posts into precise, check-worthy claims for enhanced fact verification.
• It employs retrieval-first, supervised fine-tuning, and LLM prompting techniques to tackle both supervised and zero-shot multilingual challenges across 20 languages.
• Empirical results demonstrate that hybrid architectures and language-specific tuning boost METEOR scores, though zero-shot conditions remain a significant hurdle.

Claim normalization—the mapping of noisy, informal user-generated content to concise, check-worthy claims—forms a central pillar of automated fact-checking pipelines. In CheckThat! 2025 Task 2, this challenge is extended to a highly multilingual context, targeting both monolingual (supervised) and zero-shot (unseen-language) conditions across 20 languages. The objective is to produce, for a given social media post, a single-sentence normalized claim suitable for downstream evidence retrieval and veracity classification. Below, the problem’s definition, task setup, evaluation regime, system architectures, empirical outcomes, and future research trajectories are systematically analyzed.

1. Formal Task Definition and Dataset Construction

Task 2 of the CLEF-2025 CheckThat! Lab specifies claim normalization as the transformation function $f: \mathcal{P} \rightarrow \mathcal{C}$, converting raw social-media post $p \in \mathcal{P}$ into concise, standalone normalized claim $c \in \mathcal{C}$ (Alam et al., 19 Mar 2025). Noisy inputs typically contain emojis, hashtags, hyperlinks, code-mixed fragments, off-topic remarks, duplicated sentences, and subjective commentary. The output must be a one-sentence, factual summary that preserves and centers on the main verifiable assertion, explicitly removing all redundancy and refusing to introduce new or speculative information (Pramov et al., 24 Aug 2025, Almada et al., 15 Sep 2025).

The official dataset encompasses 20 languages: 13 “in-language” tracks with annotated train/dev/test splits (e.g., English, Arabic, Portuguese, Spanish, Hindi), and 7 zero-shot tracks (e.g., Czech, Greek, Korean, Bengali, Romanian, Dutch, Telugu) with only a test set (Alam et al., 19 Mar 2025, Almada et al., 15 Sep 2025). Gold labels were produced by human annotators, who distilled each post to its central claim. Posts were sourced from the Google Fact-check Explorer API and the ClaimReview schema (Alam et al., 19 Mar 2025). Table 1 (abridged) summarizes available data:

Language	Train	Dev	Test	Zero-Shot
English	11,374	1171	1285	No
Spanish	3458	439	439	No
...	...	...	...	...
Bengali	—	—	81	Yes
Greek	—	—	156	Yes
...	...	...	...	...

A normalized claim is deemed suitable for subsequent IR/veracity modules without further text manipulation (Almada et al., 15 Sep 2025).

2. Official Evaluation Protocol

The sole official evaluation metric is METEOR (Alam et al., 19 Mar 2025, Pramov et al., 24 Aug 2025, Wilder et al., 8 Sep 2025, Almada et al., 15 Sep 2025). For each system output $\hat{c}$ and gold claim $c$:

• Unigram alignments (across exact, stem, synonym, paraphrase) are computed, with precision $P$ and recall $R$:

$$P = \frac{|G \cap S|}{|S|}, \qquad R = \frac{|G \cap S|}{|G|}$$

• The harmonic mean with recall-weight $\alpha=9$:

$F_{\alpha} = \frac{(1+\alpha)PR}{R+\alpha P}$

• The fragmentation penalty $p \in \mathcal{P}$0 is:

$p \in \mathcal{P}$1

• The final METEOR:

$p \in \mathcal{P}$2

Macro-averaged METEOR across the test set determines official ranking; higher scores indicate closer paraphrase-level match with the reference. No alternative metrics (e.g., BERTScore, BLEURT) are considered by the organizers, though participants often track those in development (Pramov et al., 24 Aug 2025, Wilder et al., 8 Sep 2025, Almada et al., 15 Sep 2025).

3. Core System Architectures and Methodological Variants

Three principal system design paradigms dominated CheckThat! 2025 Task 2:

(a) Retrieval-First, LLM-Backed Hybrid Pipelines

DS@GT’s first-place system operationalizes a two-stage pipeline (Pramov et al., 24 Aug 2025):

1. Embedding-based Retrieval: Each test post is encoded by a suitable Sentence Transformer (e.g., multilingual-e5-base, ms-marco-distilbert-base-v3 or Indic-specific models). Cosine similarity is used to index (via e.g., Faiss) into train/dev embeddings, retrieving the top-N nearest posts.
2. Normalization Reuse: If the nearest neighbor’s similarity exceeds a tuned threshold $p \in \mathcal{P}$3, the corresponding gold normalization is copied directly.
3. LLM Fallback: Otherwise, a language-appropriate prompt is constructed comprising the input and the three most similar train/dev (post, normalization) pairs as in-context examples. Generation is handled by GPT-4o-mini.

In zero-shot settings (no in-language train/dev), all instances invoke the LLM with a static English prompt and fixed few-shot examples. See Table below for deployed pretrained models and per-language thresholds.

Lang	k	METEOR	Rank	Model
spa	0.80	0.6077	1	intfloat/multilingual-e5-base
por	0.90	0.5770	1	intfloat/multilingual-e5-base
...	...	...	...	...

(b) Supervised Fine-Tuning of Small LLMs (SLMs)

AKCIT-FN and others fine-tune T5-variants or related encoder-decoder SLMs on language-specific or multilingual data (Almada et al., 15 Sep 2025):

• For each instance: input is formatted as "normalize: {post}", target as the normalized claim.
• Models include PTT5, AraT5, T5-French, Varta-T5 (Indic), PLT5, T5S, ThaiT5-Instruct, Flan-T5, mBART, UMT5.
• Standard cross-entropy loss is minimized:

$p \in \mathcal{P}$4

• Hyperparameters (epochs, learning rate, optimizer, batch size, gen length, beam size) are tuned per language.

Zero-shot tracks use direct LLM prompting (Qwen 2.5 Instruct, GPT-4o-mini, etc.) with templated prompts translated as appropriate.

(c) Prompting and Fine-Tuning with LLMs

The UNH team explored prompt engineering for various LLMs (GPT-4.1-nano, Gemini-2.0-Flash, LLaMA 3.3-70B) and classic sequence-to-sequence fine-tuning (Flan-T5 Large, LoRA on Flan-T5 Base) (Wilder et al., 8 Sep 2025):

• Prompts ranged from zero-shot (“Identify the decontextualized, stand-alone, and verifiable central claim in the given post: ...”) to few-shot (random or keyword-based similarity), with or without “Let’s think step by step” chain-of-thought triggers.
• A self-refinement stage—where the model critiques and refines its own output—often yielded additional gains.
• Full-model fine-tuning on Flan-T5-Large achieved the highest METEOR, but prompted LLMs sometimes produced richer or more contextually faithful normalizations even if scoring lower on METEOR.

4. Empirical Results and Comparative Performance

Monolingual (supervised) tracks consistently favor SLM fine-tuning or hybrid retrieval+LLM pipelines. DS@GT’s approach achieved first place on 7/13 supervised tracks and near-top marks on the remainder (Pramov et al., 24 Aug 2025). AKCIT-FN’s approach consistently placed in the top three in 15 of 20 languages, with SLM fine-tuning dominating in supervised conditions (Almada et al., 15 Sep 2025). UNH reports best English-only leaderboard METEOR from Flan-T5 Large fine-tuning, again highlighting direct exposure to in-domain data as the critical success factor (Wilder et al., 8 Sep 2025).

Zero-shot languages present a much more challenging regime. Across teams, zero-shot mean METEOR is ~0.215 compared to ~0.46 for supervised tracks (Pramov et al., 24 Aug 2025). LLM prompting without language-specific exemplars often produces less fluent or semantically faithful outputs, with frequent hallucinations, omission of key morphological elements, or syntactic disfluency.

Aggregated language-wise results illustrate this pattern:

Language	Strategy	METEOR	Rank
Spanish	SLM fine-tuning	0.5213	3
Tamil	SLM fine-tuning	0.5197	2
Telugu	LLM prompting	0.5176	2
Bengali	LLM prompting	0.2916	3
Romanian	LLM prompting	0.2516	2

5. Error Modes, Analytical Insights, and Ablations

Central observations and error findings include:

• In high-resource languages, cosine-based retrieval suffices for direct reuse of normalizations in 40–60% of cases, with METEOR drops of up to 20 points when relying on LLM generation alone (Pramov et al., 24 Aug 2025).
• Zero-shot LLM prompting, even with chain-of-thought or carefully constructed instructions, struggles with typologically distant targets, especially for agglutinative or morphologically complex scripts (Pramov et al., 24 Aug 2025, Almada et al., 15 Sep 2025).
• Performance in zero-shot tracks is highly sensitive to the language and style of in-context examples. The use of English-only context does not transfer well, and omission/hallucination errors are frequent (Pramov et al., 24 Aug 2025).
• For systems relying on fine-tuning, hyperparameter sensitivity (LR, epochs) is pronounced, and cross-split deduplication is required to prevent data-leak overfitting (Almada et al., 15 Sep 2025). Seq2seq systems overfitted when posts overlapped between splits (Pramov et al., 24 Aug 2025).
• Metrics: While METEOR is robust to paraphrase, it may penalize useful semantic variations or more contextually informative normalizations. BERTScore provides helpful auxiliary signal but was not a ranking metric (Wilder et al., 8 Sep 2025).

6. Systematic Limitations and Recommendations for Future Research

Structural limitations shape ongoing and future work:

• Strong dependence on training data coverage and potential overlap between train/dev/test splits may inflate performance unless rigorously stratified (Pramov et al., 24 Aug 2025, Almada et al., 15 Sep 2025).
• Zero-shot performance remains fundamentally limited by the lack of in-language context; introducing multilingual few-shot or lightweight in-language fine-tuning is strongly recommended (Pramov et al., 24 Aug 2025, Almada et al., 15 Sep 2025).
• Dedicated language-tailored Transformers or adapters are necessary to improve retrieval in underperforming languages, particularly among under-resourced Indo-Aryan or Dravidian families (Pramov et al., 24 Aug 2025, Almada et al., 15 Sep 2025).
• Chain-of-thought and prompt refinement techniques show promise but require better evaluation metrics that balance lexical match with factual informativeness (Wilder et al., 8 Sep 2025).
• Official evaluation beyond METEOR (e.g., BERTScore, BLEURT, or human annotation) would allow better coverage of semantically accurate, factually rich outputs (Alam et al., 19 Mar 2025, Wilder et al., 8 Sep 2025).

Key recommendations proposed:

• Develop multilingual or typologically adaptive in-context prompting regimes for zero-shot tracks (Pramov et al., 24 Aug 2025, Almada et al., 15 Sep 2025).
• Incorporate human-in-the-loop error analysis and exemplar selection for more robust normalization, especially in morphologically complex or code-mixed contexts (Almada et al., 15 Sep 2025).
• Refine prompt engineering (e.g., chain-of-thought, multi-prompt ensembles) to improve factual and semantic fidelity (Almada et al., 15 Sep 2025, Wilder et al., 8 Sep 2025).
• Ensure rigorous stratification and deduplication in dataset construction to avoid evaluation leakage (Pramov et al., 24 Aug 2025).

7. Integration and Pipeline Design in Multilingual Fact-Checking

Claim normalization forms the linchpin connecting subjectivity filtering (Task 1) with downstream retrieval (Task 3) and computational veracity assessment (Task 4) (Alam et al., 19 Mar 2025). The best-performing Task 2 systems rely on modular, resource-adaptive pipelines: SLM fine-tuning for data-rich settings, retrieval-augmented or pure LLM prompting for resource-poor conditions (Pramov et al., 24 Aug 2025, Almada et al., 15 Sep 2025). The modular design allows quick expansion to new languages and domains, contingent on the continual improvement of both monolingual sequence-to-sequence models and massively multilingual LLMs.

A plausible implication is that hybrid architectures—combining fast retrieval with LLM-based generation and robust stratification for error-prone regimes—are likely to set new benchmarks for scalable, semi-automated claim normalization in veracity pipelines. Further, as evaluation metrics evolve, systems optimizing for both paraphrase-level match and factual informativeness will more closely align with the needs of human fact-checkers (Wilder et al., 8 Sep 2025).

---

References:

(Alam et al., 19 Mar 2025, Pramov et al., 24 Aug 2025, Wilder et al., 8 Sep 2025, Almada et al., 15 Sep 2025)