---
title: 'CheckThat! 2025 Task 2: Multilingual Claim Normalization'
url: https://www.emergentmind.com/topics/checkthat-2025-task-2
type: topic
---

# CheckThat! 2025 Task 2: Multilingual Claim Normalization

Claim normalization—the mapping of noisy, informal user-generated content to concise, check-worthy claims—forms a central pillar of automated fact-checking pipelines. In CheckThat! 2025 Task 2, this challenge is extended to a highly multilingual context, targeting both monolingual (supervised) and zero-shot (unseen-language) conditions across 20 languages. The objective is to produce, for a given social media post, a single-sentence normalized claim suitable for downstream evidence retrieval and veracity classification. Below, the problem’s definition, task setup, evaluation regime, system architectures, empirical outcomes, and future research trajectories are systematically analyzed.

## 1. Formal Task Definition and Dataset Construction

Task 2 of the CLEF-2025 CheckThat! Lab specifies claim normalization as the transformation function $f: \mathcal{P} \rightarrow \mathcal{C}$, converting raw social-media post $p \in \mathcal{P}$ into concise, standalone normalized claim $c \in \mathcal{C}$ [2503.14828]. Noisy inputs typically contain emojis, hashtags, hyperlinks, code-mixed fragments, off-topic remarks, duplicated sentences, and subjective commentary. The output must be a one-sentence, factual summary that preserves and centers on the main verifiable assertion, explicitly removing all redundancy and refusing to introduce new or speculative information [2508.17402, 2509.11496].

The official dataset encompasses 20 languages: 13 “in-language” tracks with annotated train/dev/test splits (e.g., English, Arabic, Portuguese, Spanish, Hindi), and 7 zero-shot tracks (e.g., Czech, Greek, Korean, Bengali, Romanian, Dutch, Telugu) with only a test set [2503.14828, 2509.11496]. Gold labels were produced by human annotators, who distilled each post to its central claim. Posts were sourced from the Google Fact-check Explorer API and the ClaimReview schema [2503.14828]. Table 1 (abridged) summarizes available data:

| Language | Train | Dev | Test | Zero-Shot | 
|----------|-------|-----|------|-----------|
| English  | 11,374|1171 |1285  | No        |
| Spanish  | 3458  | 439 | 439  | No        |
| ...      | ...   | ... | ...  | ...       |
| Bengali  |  —    | —   | 81   | Yes       |
| Greek    |  —    | —   | 156  | Yes       |
| ...      | ...   | ... | ...  | ...       |

A normalized claim is deemed suitable for subsequent IR/veracity modules without further text manipulation [2509.11496].

## 2. Official Evaluation Protocol

The sole official evaluation metric is METEOR [2503.14828, 2508.17402, 2509.06883, 2509.11496]. For each system output $\hat{c}$ and gold claim $c$:

- Unigram alignments (across exact, stem, synonym, paraphrase) are computed, with precision $P$ and recall $R$:

  $$
  P = \frac{|G \cap S|}{|S|}, \qquad R = \frac{|G \cap S|}{|G|} 
  $$

- The harmonic mean with recall-weight $\alpha=9$:

  $$
  F_{\alpha} = \frac{(1+\alpha)PR}{R+\alpha P}
  $$

- The fragmentation penalty $\operatorname{Pen}$ is:

  $$
  \operatorname{Pen} = \gamma \left( \frac{\text{chunks}}{m} \right)^{\beta}
  $$

- The final METEOR:

  $$
  \mathrm{METEOR} = (1-\operatorname{Pen}) F_{\alpha}
  $$

Macro-averaged METEOR across the test set determines official ranking; higher scores indicate closer paraphrase-level match with the reference. No alternative metrics (e.g., BERTScore, BLEURT) are considered by the organizers, though participants often track those in development [2508.17402, 2509.06883, 2509.11496].

## 3. Core System Architectures and Methodological Variants

Three principal system design paradigms dominated CheckThat! 2025 Task 2:

**(a) Retrieval-First, LLM-Backed Hybrid Pipelines**

DS@GT’s first-place system operationalizes a two-stage pipeline [2508.17402]:

1. **Embedding-based Retrieval**: Each test post is encoded by a suitable Sentence Transformer (e.g., multilingual-e5-base, ms-marco-distilbert-base-v3 or Indic-specific models). Cosine similarity is used to index (via e.g., Faiss) into train/dev embeddings, retrieving the top-N nearest posts.
2. **Normalization Reuse**: If the nearest neighbor’s similarity exceeds a tuned threshold $k_{\text{lang}}$, the corresponding gold normalization is copied directly.
3. **LLM Fallback**: Otherwise, a language-appropriate prompt is constructed comprising the input and the three most similar train/dev (post, normalization) pairs as in-context examples. Generation is handled by GPT-4o-mini.

In zero-shot settings (no in-language train/dev), all instances invoke the LLM with a static English prompt and fixed few-shot examples. See Table below for deployed pretrained models and per-language thresholds.

| Lang | k | METEOR | Rank | Model                           |
|------|---|--------|------|---------------------------------|
| spa  |0.80|0.6077 |  1   | intfloat/multilingual-e5-base   |
| por  |0.90|0.5770 |  1   | intfloat/multilingual-e5-base   |
| ...  | ...| ...   | ...  | ...                             |

**(b) Supervised Fine-Tuning of Small Language Models (SLMs)**

AKCIT-FN and others fine-tune T5-variants or related encoder-decoder SLMs on language-specific or multilingual data [2509.11496]:

- For each instance: input is formatted as "normalize: {post}", target as the normalized claim.
- Models include PTT5, AraT5, T5-French, Varta-T5 (Indic), PLT5, T5S, ThaiT5-Instruct, Flan-T5, mBART, UMT5.
- Standard cross-entropy loss is minimized:

  $$
  \mathcal{L}_{\mathrm{CE}} = -\sum_{t=1}^{T} \log P_\theta(y_t|y_{<t}, x)
  $$

- Hyperparameters (epochs, learning rate, optimizer, batch size, gen length, beam size) are tuned per language.

Zero-shot tracks use direct LLM prompting (Qwen 2.5 Instruct, GPT-4o-mini, etc.) with templated prompts translated as appropriate.

**(c) Prompting and Fine-Tuning with LLMs**

The UNH team explored prompt engineering for various LLMs (GPT-4.1-nano, Gemini-2.0-Flash, LLaMA 3.3-70B) and classic sequence-to-sequence fine-tuning (Flan-T5 Large, LoRA on Flan-T5 Base) [2509.06883]:

- Prompts ranged from zero-shot (“Identify the decontextualized, stand-alone, and verifiable central claim in the given post: ...”) to few-shot (random or keyword-based similarity), with or without “Let’s think step by step” chain-of-thought triggers.
- A self-refinement stage—where the model critiques and refines its own output—often yielded additional gains.
- Full-model fine-tuning on Flan-T5-Large achieved the highest METEOR, but prompted LLMs sometimes produced richer or more contextually faithful normalizations even if scoring lower on METEOR.

## 4. Empirical Results and Comparative Performance

Monolingual (supervised) tracks consistently favor SLM fine-tuning or hybrid retrieval+LLM pipelines. DS@GT’s approach achieved first place on 7/13 supervised tracks and near-top marks on the remainder [2508.17402]. AKCIT-FN’s approach consistently placed in the top three in 15 of 20 languages, with SLM fine-tuning dominating in supervised conditions [2509.11496]. UNH reports best English-only leaderboard METEOR from Flan-T5 Large fine-tuning, again highlighting direct exposure to in-domain data as the critical success factor [2509.06883].

Zero-shot languages present a much more challenging regime. Across teams, zero-shot mean METEOR is ~0.215 compared to ~0.46 for supervised tracks [2508.17402]. LLM prompting without language-specific exemplars often produces less fluent or semantically faithful outputs, with frequent hallucinations, omission of key morphological elements, or syntactic disfluency.

Aggregated language-wise results illustrate this pattern:

| Language   | Strategy          | METEOR | Rank |
|------------|-------------------|--------|------|
| Spanish    | SLM fine-tuning   | 0.5213 | 3    |
| Tamil      | SLM fine-tuning   | 0.5197 | 2    |
| Telugu     | LLM prompting     | 0.5176 | 2    |
| Bengali    | LLM prompting     | 0.2916 | 3    |
| Romanian   | LLM prompting     | 0.2516 | 2    |

## 5. Error Modes, Analytical Insights, and Ablations

Central observations and error findings include:

- In high-resource languages, cosine-based retrieval suffices for direct reuse of normalizations in 40–60% of cases, with METEOR drops of up to 20 points when relying on LLM generation alone [2508.17402].
- Zero-shot LLM prompting, even with chain-of-thought or carefully constructed instructions, struggles with typologically distant targets, especially for agglutinative or morphologically complex scripts [2508.17402, 2509.11496].
- Performance in zero-shot tracks is highly sensitive to the language and style of in-context examples. The use of English-only context does not transfer well, and omission/hallucination errors are frequent [2508.17402].
- For systems relying on fine-tuning, hyperparameter sensitivity (LR, epochs) is pronounced, and cross-split deduplication is required to prevent data-leak overfitting [2509.11496]. Seq2seq systems overfitted when posts overlapped between splits [2508.17402].
- Metrics: While METEOR is robust to paraphrase, it may penalize useful semantic variations or more contextually informative normalizations. BERTScore provides helpful auxiliary signal but was not a ranking metric [2509.06883].

## 6. Systematic Limitations and Recommendations for Future Research

Structural limitations shape ongoing and future work:

- Strong dependence on training data coverage and potential overlap between train/dev/test splits may inflate performance unless rigorously stratified [2508.17402, 2509.11496].
- Zero-shot performance remains fundamentally limited by the lack of in-language context; introducing multilingual few-shot or lightweight in-language fine-tuning is strongly recommended [2508.17402, 2509.11496].
- Dedicated language-tailored Transformers or adapters are necessary to improve retrieval in underperforming languages, particularly among under-resourced Indo-Aryan or Dravidian families [2508.17402, 2509.11496].
- Chain-of-thought and prompt refinement techniques show promise but require better evaluation metrics that balance lexical match with factual informativeness [2509.06883].
- Official evaluation beyond METEOR (e.g., BERTScore, BLEURT, or human annotation) would allow better coverage of semantically accurate, factually rich outputs [2503.14828, 2509.06883].

Key recommendations proposed:

- Develop multilingual or typologically adaptive in-context prompting regimes for zero-shot tracks [2508.17402, 2509.11496].
- Incorporate human-in-the-loop error analysis and exemplar selection for more robust normalization, especially in morphologically complex or code-mixed contexts [2509.11496].
- Refine prompt engineering (e.g., chain-of-thought, multi-prompt ensembles) to improve factual and semantic fidelity [2509.11496, 2509.06883].
- Ensure rigorous stratification and deduplication in dataset construction to avoid evaluation leakage [2508.17402].

## 7. Integration and Pipeline Design in Multilingual Fact-Checking

Claim normalization forms the linchpin connecting subjectivity filtering (Task 1) with downstream retrieval (Task 3) and computational veracity assessment (Task 4) [2503.14828]. The best-performing Task 2 systems rely on modular, resource-adaptive pipelines: SLM fine-tuning for data-rich settings, retrieval-augmented or pure LLM prompting for resource-poor conditions [2508.17402, 2509.11496]. The modular design allows quick expansion to new languages and domains, contingent on the continual improvement of both monolingual sequence-to-sequence models and massively multilingual LLMs.

A plausible implication is that hybrid architectures—combining fast retrieval with LLM-based generation and robust stratification for error-prone regimes—are likely to set new benchmarks for scalable, semi-automated claim normalization in veracity pipelines. Further, as evaluation metrics evolve, systems optimizing for both paraphrase-level match and factual informativeness will more closely align with the needs of human fact-checkers [2509.06883].

---

**References**:  
[2503.14828], [2508.17402], [2509.06883], [2509.11496]

Source: https://www.emergentmind.com/topics/checkthat-2025-task-2