SemEval-2025 Task 7: Fact-Checked Claim Retrieval
- SemEval-2025 Task 7 is a benchmark task that defines an IR problem where systems retrieve fact-checked claims matching social media posts using multilingual and crosslingual methods.
- It leverages the MultiClaim dataset, augmented with a new test set, to evaluate performance with the success@10 metric across diverse language settings.
- The task spurred diverse approaches such as dense retrieval, translation pivoting, contrastive fine-tuning, and ensemble methods, highlighting challenges in crosslingual transfer and missing relevance labels.
SemEval-2025 Task 7 is a shared task on multilingual and crosslingual fact-checked claim retrieval: given a new social media post, a system must retrieve previously fact-checked claims so that a relevant prior fact-check appears near the top of the ranking. The task is built on MultiClaim, extends it with a new test set, and models a realistic fact-checking workflow in which a post may repeat, paraphrase, or translate a claim that was already checked elsewhere. It includes a monolingual track, where the post and relevant claim are in the same language, and a crosslingual track, where the matching claim may be in another language. The shared task drew 179 registered participants, produced 52 test submissions, and received system papers from 23 out of 31 teams (Peng et al., 15 May 2025). As with other SemEval editions, the task number is edition-specific and unrelated to SemEval-2020 Task 7, which concerned humor assessment in edited news headlines (Hossain et al., 2020).
1. Task definition and operational setting
SemEval-2025 Task 7 addresses the problem of finding whether a newly encountered social media post corresponds to a claim that has already been fact-checked by a professional fact-checking organization. The task is formulated as an information retrieval problem rather than as a binary verification problem: systems receive a post as a query, search a database of fact-checked claims, and return the top 10 claims for that post (Peng et al., 15 May 2025).
The task is motivated by a concrete workflow. A fact-checker receives a post, suspects that its claim may already have been checked, and searches a multilingual repository of prior fact-checks. This is operationally important because manual fact-checking does not scale, misinformation is often repeated, and cross-language spread is common. The benchmark therefore treats multilingual retrieval not as an auxiliary capability but as a central requirement for automated fact-checking systems (Peng et al., 15 May 2025).
Two retrieval settings are defined. In the monolingual track, the post and the matching claim are always in the same language, and the candidate pool is language-specific to the post. In the crosslingual track, the candidate pool is not restricted by language, and the matched claim may be in a different language from the post. In both tracks, the system output is a ranked list of claim IDs rather than a standalone truth label (Peng et al., 15 May 2025).
2. Benchmark construction and data resources
The shared task uses training and development data from MultiClaim and augments them with a newly collected test set following the same methodology. The original MultiClaim dataset contains 205,751 fact-checked claims extracted from fact-checks created by 142 fact-checking organizations, and 28,092 social media posts spanning 27 languages, collected from Facebook, Instagram, and Twitter. It established 31,305 SMP-Claim pairs, where the links were derived from ClaimReview JSON schema links and Facebook/Instagram fact-checking warnings (Peng et al., 15 May 2025).
The construction procedure matters because the task is not based on approximate semantic matching heuristics alone. The paper states that these pairing signals ensure that a post–claim connection is correct because a fact-checker left an explicit signal confirming it. Manual inspection by three authors on a random sample of 100 pairs confirmed the absence of false positives. A second inspection of claims retrieved for a sample of 87 posts showed that many valid SMP-Claim links were missing, especially for crosslingual pairs. To mitigate this incompleteness, the organizers added 3,351 additional SMP-Claim pairs using transitive connections (Peng et al., 15 May 2025).
The task is also constrained by partial multimodality. Roughly 15% of the connections rely on visual information present in attached images or videos, which creates an inherent ceiling for text-only systems (Peng et al., 15 May 2025). This is a central property of the benchmark rather than an incidental annotation issue.
For the monolingual track, training and development cover 8 languages: Arabic, English, French, German, Malay, Portuguese, Spanish, and Thai. The monolingual test set adds 2 unannounced languages, Polish and Turkish, for a total of 10 languages. Official monolingual totals are 17,016 train posts, 1,891 dev posts, and 4,276 test posts; 153,743 claims in train and dev; 260,880 claims in test; and 19,956 train pairs, 2,259 dev pairs, and 4,959 test pairs (Peng et al., 15 May 2025).
For the crosslingual track, training and development use 8 claim languages, 14 post languages, and 52 language combinations; the test set expands to 11 claim languages, 14 post languages, and 60 language combinations. A system paper describing the official setup reports 4,972/552/4,000 posts for train/dev/test and 153,743 fact-check entries in train+dev and 272,447 in test (Liu et al., 12 Jun 2025). Most crosslingual pairs still involve English on one side, which affects the interpretation of crosslingual generalization (Peng et al., 15 May 2025).
3. Retrieval tracks, inputs, and evaluation
The task is defined by a ranked-retrieval objective with a single official metric, success@10. A system succeeds for a query post if at least one relevant claim appears in the returned top-10 list (Peng et al., 15 May 2025).
| Track | Candidate pool | Official metric |
|---|---|---|
| Monolingual | Claims in the same language as the post | success@10 |
| Crosslingual | Multilingual claim pool; relevant claim may be in another language | success@10 |
The official metric is defined as:
where is the total number of examples and the indicator is 1 when at least one relevant claim is retrieved in the top 10 for post (Peng et al., 15 May 2025).
This formulation is significant because the task allows multiple relevant claims per post. The metric therefore evaluates whether retrieval places at least one correct item within the top 10, rather than requiring exact ranking of every relevant claim (Peng et al., 15 May 2025). In the monolingual track, systems are ranked by average performance across all languages; in the crosslingual track, systems are ranked by overall success@10 (Peng et al., 15 May 2025).
The organizers released four baselines: BM25, GTR-T5-Large, Paraphrase-Multilingual-MPNet-Base-v2, and Multilingual-E5-Large. On the monolingual track, E5-Large was the strongest baseline at 0.8589 average success@10, ahead of BM25 0.8123, GTR-T5-Large 0.7299, and Paraphrase-Multi-v2 0.5683. On the crosslingual track, GTR-T5-Large was strongest at 0.70525, ahead of E5-Large 0.63725, BM25 0.60275, and Paraphrase-Multi-v2 0.41075 (Peng et al., 15 May 2025).
4. Dominant modeling strategies
The official overview identifies the dominant methodological pattern as dense retrieval with multilingual or translated English embeddings, often strengthened by contrastive fine-tuning, hard negatives, reranking, and ensemble or voting strategies (Peng et al., 15 May 2025). Within that general pattern, system papers reveal several distinct design families.
A classical sparse baseline remained competitive enough to be instructive. The Duluth system used a deliberately simple TF-IDF pipeline for the monolingual track only, tuned over max_features and analyzer in TfidfVectorizer. Its best configuration used word-level tokenization with 15,000 features, improving development average success@10 from 0.7630 at 10K features to 0.7757 at 15K, while char_wb and char analyzers were much weaker at 0.4528 and 0.4637 (Syed et al., 19 May 2025).
Several strong systems treated English as a pivot language. UWBa used a fully zero-shot dense-retrieval system based on pretrained text embeddings and nearest-neighbor search by cosine similarity. It concatenated post text with OCR text, concatenated fact-check title and claim text, and used the provided English translations for all models except mGTE because multilingual models on original-language texts achieved worse results. Among five embedding models, NVIDIA NV-Embed-v2 was best in both monolingual and cross-lingual development experiments (Lenc et al., 13 Aug 2025).
The QUST_NLP submission implemented a more elaborate three-stage retrieval framework: initial retrieval, reranking, and weighted voting. In the first stage it compared several dense retrievers, including mul-e5-large, mul-e5-large-instruct, e5-mistral-7b-instruct, bge-mul-gemma2, gte-Qwen2-7B-instruct, and NV-Embed-v2. In the second stage it reranked the 100 retrieved candidates using models from the BAAI/bge-reranker family. In the third stage it aggregated reranked outputs by weighted voting based on validation-set performance. The paper reports that combining original and machine-translated text generally helps monolingual retrieval, whereas machine-translated input is preferred in the crosslingual setting (Liu et al., 12 Jun 2025).
The Word2winners system used a dense bi-encoder retrieval architecture and compared multilingual shared-space retrieval with an English-translation pivot. It reported that multilingual models generally work best on original-language text for monolingual retrieval, while translating inputs to English improves crosslingual retrieval. It also tested an LLM-assisted summarization stage and found that summarization reduced S@10 substantially, indicating that compression removed distinctions needed to separate near-duplicate claims (Azadi et al., 12 Mar 2025).
The fact check AI submission framed the task explicitly as a Learning-to-Rank problem with a shared-weight bi-encoder fine-tuned from sentence-embedding transformers. It trained on source languages and English translations for multilingual retrieval, and on English translations only for cross-lingual retrieval. The paper reports Multiple Negatives Ranking loss, a symmetric cross-entropy over the similarity matrix, and five-fold ensembling for cross-lingual retrieval (Rastogi, 5 Aug 2025).
A later system paper, MultiMind, proposed TriAligner, a dual-encoder architecture with three similarity channels: native-language, English-translated, and fused native+English representations. It combined these channels with trainable coefficients, trained with a symmetric contrastive objective, added hard negatives, and optionally used GPT-4o for post rewriting and reranking. The paper positions this as explicit multi-source alignment rather than single-space multilingual retrieval (Abootorabi et al., 24 Dec 2025).
5. Official results and representative submissions
Official leaderboard results show that monolingual retrieval was already very strong for the top systems, while crosslingual retrieval remained substantially harder. In the monolingual track, the top five systems were PINGAN AI at 0.9601, PALI at 0.9472, TIFIN India at 0.9383, RACAI at 0.9377, and QUST_NLP at 0.9365 average success@10. In the crosslingual track, the top five were PINGAN AI at 0.85875, PALI at 0.82675, RACAI at 0.82450, TIFIN India at 0.81025, and fact check AI at 0.79750 (Peng et al., 15 May 2025).
Language-wise variation was substantial even within the best monolingual systems. The best reported success@10 values by language were 0.9160 for English, 0.9720 for French, 0.9580 for German, 0.9260 for Portuguese, 0.9740 for Spanish, 1.0000 for Thai, 1.0000 for Malay, 0.9860 for Arabic, 0.9480 for Turkish, and 0.9260 for Polish (Peng et al., 15 May 2025). These numbers indicate that the benchmark is not uniformly difficult across languages or candidate pools.
Selected system-description papers illustrate the range of submitted architectures:
| System | Core design | Reported outcome |
|---|---|---|
| Duluth | Word-level TF-IDF, 15K features, monolingual only (Syed et al., 19 May 2025) | 0.6883 test average, 23rd/28 |
| QUST_NLP | Three-stage dense retrieval, reranking, weighted voting (Liu et al., 12 Jun 2025) | 93.65 monolingual, 79.25 crosslingual; 5th/28, 7th/29 |
| UWBa | Zero-shot embeddings with English translations; NV-Embed-v2 best (Lenc et al., 13 Aug 2025) | 0.927 monolingual, 0.783 cross-lingual; 7th, 9th |
| fact check AI | Bi-encoder learning-to-rank with translation-mediated training (Rastogi, 5 Aug 2025) | 0.923178 monolingual, 0.7975 cross-lingual; 10th, 5th |
| Word2winners | Dense bi-encoder, translation, fine-tuning, weighted ensemble (Azadi et al., 12 Mar 2025) | 0.92 overall, 0.85 crosslingual |
The gap between development-time architectural promise and official test ranking was also visible. MultiMind reported strong development-set gains from native+English multi-source alignment and reranking, but its official test submission ranked 21st in monolingual and 24th in crosslingual because the model was run without the reranker at test time due to compute constraints (Abootorabi et al., 24 Dec 2025). This highlights that the task rewards not only representation quality but also deployable retrieval pipelines under large candidate pools.
6. Challenges, limitations, and research significance
The organizers identify several benchmark-specific difficulties that structure the research agenda. First, crosslingual retrieval is substantially harder: performance in crosslingual is worse than in monolingual by more than 10 percentage points (Peng et al., 15 May 2025). Second, unseen languages remain difficult: Turkish and Polish were unannounced monolingual test languages and generally showed worse performance than seen languages (Peng et al., 15 May 2025). Third, the benchmark is affected by missing relevance labels: manual inspection suggested that many valid SMP-Claim links are absent, especially in the crosslingual setting (Peng et al., 15 May 2025).
A further limitation is that the crosslingual benchmark is not fully symmetric across languages. The overview paper states that most crosslingual pairs still contain English on one side (Peng et al., 15 May 2025). This suggests that high crosslingual success@10 does not automatically imply equally strong non-English-to-non-English transfer. The system papers reinforce this point from different angles. QUST_NLP attributes crosslingual difficulty partly to translation inconsistency, especially for low-resource languages (Liu et al., 12 Jun 2025). UWBa reports that using high-quality English translations with strong English-centric embedding models outperformed direct multilingual embedding in its zero-shot setting (Lenc et al., 13 Aug 2025). Word2winners reaches a similar conclusion for its experiments: translation to English is highly effective for crosslingual retrieval, even when multilingual encoders remain strong in monolingual settings (Azadi et al., 12 Mar 2025).
The benchmark also exposes a tension between lexical matching and semantic abstraction. The Duluth results show that a carefully tuned sparse lexical baseline can remain a meaningful point of comparison under limited compute (Syed et al., 19 May 2025), whereas the strongest overall systems add contrastive fine-tuning, reranking, and list fusion on top of dense retrieval (Peng et al., 15 May 2025). This suggests that fact-checked claim retrieval is neither a pure semantic-search problem nor a pure lexical-overlap problem.
In a broader perspective, SemEval-2025 Task 7 established a large, operationally motivated benchmark for multilingual automated fact-checking. It showed that dense bi-encoder retrieval is the dominant substrate, that translation-to-English pipelines remain highly competitive, and that reranking, hard negatives, and ensemble or voting strategies are often decisive at the top of the leaderboard (Peng et al., 15 May 2025). A plausible implication is that future progress will depend less on replacing retrieval with end-to-end generation than on improving multilingual alignment, zero-shot generalization to unseen languages, non-English-to-non-English retrieval, and integration of the visual information on which roughly 15% of links depend (Peng et al., 15 May 2025).