---
title: 'SemEval-2025 Task 7: Fact-Checked Claim Retrieval'
url: https://www.emergentmind.com/topics/semeval-2025-task-7
type: topic
---

# SemEval-2025 Task 7: Fact-Checked Claim Retrieval

SemEval-2025 Task 7 is a shared task on **multilingual and crosslingual fact-checked claim retrieval**: given a new social media post, a system must retrieve previously fact-checked claims so that a relevant prior fact-check appears near the top of the ranking. The task is built on **MultiClaim**, extends it with a new test set, and models a realistic fact-checking workflow in which a post may repeat, paraphrase, or translate a claim that was already checked elsewhere. It includes a **monolingual track**, where the post and relevant claim are in the same language, and a **crosslingual track**, where the matching claim may be in another language. The shared task drew **179 registered participants**, produced **52 test submissions**, and received system papers from **23 out of 31 teams** [2505.10740]. As with other SemEval editions, the task number is edition-specific and unrelated to **SemEval-2020 Task 7**, which concerned humor assessment in edited news headlines [2008.00304].

## 1. Task definition and operational setting

SemEval-2025 Task 7 addresses the problem of finding whether a newly encountered social media post corresponds to a claim that has **already been fact-checked** by a professional fact-checking organization. The task is formulated as an **information retrieval** problem rather than as a binary verification problem: systems receive a post as a query, search a database of fact-checked claims, and return the **top 10 claims** for that post [2505.10740].

The task is motivated by a concrete workflow. A fact-checker receives a post, suspects that its claim may already have been checked, and searches a multilingual repository of prior fact-checks. This is operationally important because manual fact-checking does not scale, misinformation is often repeated, and cross-language spread is common. The benchmark therefore treats multilingual retrieval not as an auxiliary capability but as a central requirement for automated fact-checking systems [2505.10740].

Two retrieval settings are defined. In the **monolingual track**, the post and the matching claim are always in the same language, and the candidate pool is language-specific to the post. In the **crosslingual track**, the candidate pool is not restricted by language, and the matched claim may be in a different language from the post. In both tracks, the system output is a ranked list of claim IDs rather than a standalone truth label [2505.10740].

## 2. Benchmark construction and data resources

The shared task uses training and development data from **MultiClaim** and augments them with a newly collected test set following the same methodology. The original MultiClaim dataset contains **205,751 fact-checked claims** extracted from fact-checks created by **142 fact-checking organizations**, and **28,092 social media posts** spanning **27 languages**, collected from **Facebook, Instagram, and Twitter**. It established **31,305 SMP-Claim pairs**, where the links were derived from **ClaimReview JSON schema links** and **Facebook/Instagram fact-checking warnings** [2505.10740].

The construction procedure matters because the task is not based on approximate semantic matching heuristics alone. The paper states that these pairing signals ensure that a post–claim connection is correct because a fact-checker left an explicit signal confirming it. Manual inspection by **three authors** on a random sample of **100 pairs** confirmed the **absence of false positives**. A second inspection of claims retrieved for a sample of **87 posts** showed that many valid SMP-Claim links were missing, especially for crosslingual pairs. To mitigate this incompleteness, the organizers added **3,351 additional SMP-Claim pairs** using **transitive connections** [2505.10740].

The task is also constrained by partial multimodality. Roughly **15% of the connections rely on visual information** present in attached images or videos, which creates an inherent ceiling for text-only systems [2505.10740]. This is a central property of the benchmark rather than an incidental annotation issue.

For the **monolingual track**, training and development cover **8 languages**: Arabic, English, French, German, Malay, Portuguese, Spanish, and Thai. The **monolingual test set** adds **2 unannounced languages**, Polish and Turkish, for a total of **10 languages**. Official monolingual totals are **17,016 train posts**, **1,891 dev posts**, and **4,276 test posts**; **153,743 claims** in train and dev; **260,880 claims** in test; and **19,956 train pairs**, **2,259 dev pairs**, and **4,959 test pairs** [2505.10740].

For the **crosslingual track**, training and development use **8 claim languages**, **14 post languages**, and **52 language combinations**; the test set expands to **11 claim languages**, **14 post languages**, and **60 language combinations**. A system paper describing the official setup reports **4,972/552/4,000 posts** for train/dev/test and **153,743 fact-check entries** in train+dev and **272,447** in test [2506.17272]. Most crosslingual pairs still involve **English** on one side, which affects the interpretation of crosslingual generalization [2505.10740].

## 3. Retrieval tracks, inputs, and evaluation

The task is defined by a ranked-retrieval objective with a single official metric, **success@10**. A system succeeds for a query post if **at least one relevant claim** appears in the returned top-10 list [2505.10740].

| Track | Candidate pool | Official metric |
|---|---|---|
| Monolingual | Claims in the same language as the post | success@10 |
| Crosslingual | Multilingual claim pool; relevant claim may be in another language | success@10 |

The official metric is defined as:

$$
\text{success@10} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}(\text{relevant claim in top10})
$$

where \(N\) is the total number of examples and the indicator is 1 when at least one relevant claim is retrieved in the top 10 for post \(i\) [2505.10740].

This formulation is significant because the task allows **multiple relevant claims per post**. The metric therefore evaluates whether retrieval places at least one correct item within the top 10, rather than requiring exact ranking of every relevant claim [2505.10740]. In the monolingual track, systems are ranked by **average performance across all languages**; in the crosslingual track, systems are ranked by overall **success@10** [2505.10740].

The organizers released four baselines: **BM25**, **GTR-T5-Large**, **Paraphrase-Multilingual-MPNet-Base-v2**, and **Multilingual-E5-Large**. On the **monolingual** track, **E5-Large** was the strongest baseline at **0.8589** average success@10, ahead of **BM25 0.8123**, **GTR-T5-Large 0.7299**, and **Paraphrase-Multi-v2 0.5683**. On the **crosslingual** track, **GTR-T5-Large** was strongest at **0.70525**, ahead of **E5-Large 0.63725**, **BM25 0.60275**, and **Paraphrase-Multi-v2 0.41075** [2505.10740].

## 4. Dominant modeling strategies

The official overview identifies the dominant methodological pattern as **dense retrieval with multilingual or translated English embeddings**, often strengthened by **contrastive fine-tuning, hard negatives, reranking, and ensemble or voting strategies** [2505.10740]. Within that general pattern, system papers reveal several distinct design families.

A classical sparse baseline remained competitive enough to be instructive. The **Duluth** system used a deliberately simple **TF-IDF** pipeline for the **monolingual track** only, tuned over `max_features` and `analyzer` in `TfidfVectorizer`. Its best configuration used **word-level tokenization** with **15,000 features**, improving development average success@10 from **0.7630** at **10K** features to **0.7757** at **15K**, while `char_wb` and `char` analyzers were much weaker at **0.4528** and **0.4637** [2505.12616].

Several strong systems treated English as a pivot language. **UWBa** used a fully **zero-shot** dense-retrieval system based on pretrained text embeddings and nearest-neighbor search by cosine similarity. It concatenated post text with OCR text, concatenated fact-check title and claim text, and used the provided **English translations** for all models except mGTE because multilingual models on original-language texts **achieved worse results**. Among five embedding models, **NVIDIA NV-Embed-v2** was best in both monolingual and cross-lingual development experiments [2508.09517].

The **QUST_NLP** submission implemented a more elaborate **three-stage retrieval framework**: initial retrieval, reranking, and weighted voting. In the first stage it compared several dense retrievers, including **mul-e5-large**, **mul-e5-large-instruct**, **e5-mistral-7b-instruct**, **bge-mul-gemma2**, **gte-Qwen2-7B-instruct**, and **NV-Embed-v2**. In the second stage it reranked the **100** retrieved candidates using models from the **BAAI/bge-reranker** family. In the third stage it aggregated reranked outputs by **weighted voting** based on validation-set performance. The paper reports that combining original and machine-translated text generally helps **monolingual** retrieval, whereas **machine-translated input** is preferred in the **crosslingual** setting [2506.17272].

The **Word2winners** system used a dense **bi-encoder** retrieval architecture and compared multilingual shared-space retrieval with an **English-translation pivot**. It reported that multilingual models generally work best on **original-language text** for **monolingual** retrieval, while **translating inputs to English improves crosslingual retrieval**. It also tested an LLM-assisted summarization stage and found that summarization **reduced S@10 substantially**, indicating that compression removed distinctions needed to separate near-duplicate claims [2503.09011].

The **fact check AI** submission framed the task explicitly as a **Learning-to-Rank** problem with a shared-weight **bi-encoder** fine-tuned from sentence-embedding transformers. It trained on **source languages and English translations** for multilingual retrieval, and on **English translations only** for cross-lingual retrieval. The paper reports **Multiple Negatives Ranking loss**, a symmetric cross-entropy over the similarity matrix, and five-fold ensembling for cross-lingual retrieval [2508.03475].

A later system paper, **MultiMind**, proposed **TriAligner**, a dual-encoder architecture with three similarity channels: native-language, English-translated, and fused native+English representations. It combined these channels with trainable coefficients, trained with a symmetric contrastive objective, added hard negatives, and optionally used **GPT-4o** for post rewriting and reranking. The paper positions this as explicit **multi-source alignment** rather than single-space multilingual retrieval [2512.20950].

## 5. Official results and representative submissions

Official leaderboard results show that monolingual retrieval was already very strong for the top systems, while crosslingual retrieval remained substantially harder. In the **monolingual track**, the top five systems were **PINGAN AI** at **0.9601**, **PALI** at **0.9472**, **TIFIN India** at **0.9383**, **RACAI** at **0.9377**, and **QUST_NLP** at **0.9365** average success@10. In the **crosslingual track**, the top five were **PINGAN AI** at **0.85875**, **PALI** at **0.82675**, **RACAI** at **0.82450**, **TIFIN India** at **0.81025**, and **fact check AI** at **0.79750** [2505.10740].

Language-wise variation was substantial even within the best monolingual systems. The best reported success@10 values by language were **0.9160** for English, **0.9720** for French, **0.9580** for German, **0.9260** for Portuguese, **0.9740** for Spanish, **1.0000** for Thai, **1.0000** for Malay, **0.9860** for Arabic, **0.9480** for Turkish, and **0.9260** for Polish [2505.10740]. These numbers indicate that the benchmark is not uniformly difficult across languages or candidate pools.

Selected system-description papers illustrate the range of submitted architectures:

| System | Core design | Reported outcome |
|---|---|---|
| Duluth | Word-level TF-IDF, 15K features, monolingual only [2505.12616] | **0.6883** test average, **23rd/28** |
| QUST_NLP | Three-stage dense retrieval, reranking, weighted voting [2506.17272] | **93.65** monolingual, **79.25** crosslingual; **5th/28**, **7th/29** |
| UWBa | Zero-shot embeddings with English translations; NV-Embed-v2 best [2508.09517] | **0.927** monolingual, **0.783** cross-lingual; **7th**, **9th** |
| fact check AI | Bi-encoder learning-to-rank with translation-mediated training [2508.03475] | **0.923178** monolingual, **0.7975** cross-lingual; **10th**, **5th** |
| Word2winners | Dense bi-encoder, translation, fine-tuning, weighted ensemble [2503.09011] | **0.92** overall, **0.85** crosslingual |

The gap between development-time architectural promise and official test ranking was also visible. **MultiMind** reported strong development-set gains from native+English multi-source alignment and reranking, but its official test submission ranked **21st** in monolingual and **24th** in crosslingual because the model was run **without the reranker** at test time due to compute constraints [2512.20950]. This highlights that the task rewards not only representation quality but also deployable retrieval pipelines under large candidate pools.

## 6. Challenges, limitations, and research significance

The organizers identify several benchmark-specific difficulties that structure the research agenda. First, **crosslingual retrieval is substantially harder**: performance in crosslingual is worse than in monolingual by **more than 10 percentage points** [2505.10740]. Second, **unseen languages** remain difficult: **Turkish** and **Polish** were unannounced monolingual test languages and generally showed worse performance than seen languages [2505.10740]. Third, the benchmark is affected by **missing relevance labels**: manual inspection suggested that many valid SMP-Claim links are absent, especially in the crosslingual setting [2505.10740].

A further limitation is that the crosslingual benchmark is not fully symmetric across languages. The overview paper states that most crosslingual pairs still contain **English** on one side [2505.10740]. This suggests that high crosslingual success@10 does not automatically imply equally strong non-English-to-non-English transfer. The system papers reinforce this point from different angles. **QUST_NLP** attributes crosslingual difficulty partly to **translation inconsistency**, especially for low-resource languages [2506.17272]. **UWBa** reports that using high-quality **English translations** with strong English-centric embedding models outperformed direct multilingual embedding in its zero-shot setting [2508.09517]. **Word2winners** reaches a similar conclusion for its experiments: translation to English is highly effective for crosslingual retrieval, even when multilingual encoders remain strong in monolingual settings [2503.09011].

The benchmark also exposes a tension between lexical matching and semantic abstraction. The **Duluth** results show that a carefully tuned sparse lexical baseline can remain a meaningful point of comparison under limited compute [2505.12616], whereas the strongest overall systems add contrastive fine-tuning, reranking, and list fusion on top of dense retrieval [2505.10740]. This suggests that fact-checked claim retrieval is neither a pure semantic-search problem nor a pure lexical-overlap problem.

In a broader perspective, SemEval-2025 Task 7 established a large, operationally motivated benchmark for multilingual automated fact-checking. It showed that **dense bi-encoder retrieval** is the dominant substrate, that **translation-to-English pipelines** remain highly competitive, and that **reranking, hard negatives, and ensemble or voting strategies** are often decisive at the top of the leaderboard [2505.10740]. A plausible implication is that future progress will depend less on replacing retrieval with end-to-end generation than on improving multilingual alignment, zero-shot generalization to unseen languages, non-English-to-non-English retrieval, and integration of the **visual information** on which roughly **15%** of links depend [2505.10740].

Source: https://www.emergentmind.com/topics/semeval-2025-task-7