---
title: 'SemEval-2025 Task 3: Mu-SHROOM Benchmark'
url: https://www.emergentmind.com/topics/semeval-2025-task-3
type: topic
---

# SemEval-2025 Task 3: Mu-SHROOM Benchmark

SemEval-2025 Task 3 is **Mu-SHROOM**, the **Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakes**, a SemEval benchmark devoted to detecting unsupported content in the outputs of **instruction-tuned large language models**. Mu-SHROOM addresses **general-purpose LLMs** in **14 languages** and frames hallucination detection as a **span-labeling task** over generated answers: the objective is to identify *where* unsupported content occurs, not merely whether an answer is globally wrong. In the official formulation, the annotation target is the **characters/tokens in the generated answer** that should be considered hallucinated or overgenerated relative to a **Wikipedia reference page**, and the main ranking criterion is **character-level Intersection-over-Union (IoU)**, complemented by a correlation metric that models annotator uncertainty [2504.11975].

## 1. Task identity and problem formulation

Mu-SHROOM extends the earlier **SHROOM** shared task from SemEval-2024. The 2025 edition broadens the scope in three explicit ways: it is **multilingual**, it addresses **general-purpose LLMs**, and it introduces **token/character-level annotations** for precise localization. Each datapoint consists of a **question**, an **LLM-generated answer**, and a **Wikipedia reference page from which the question was derived**. Participants must identify the spans in the answer that contain unsupported factual content. The organizers define hallucination as **“content that contains or describes facts that are not supported by a provided reference”**, and the annotation instructions sharpen this further: hallucinations are cases where the answer is **“more specific than it should be, given the information available in the Wikipedia page.”** Annotators were instructed to highlight the **minimum number of characters** that would need to be edited or deleted to make the answer correct, while preferring **content words** over function words where possible [2504.11975].

This formulation makes Mu-SHROOM a fine-grained factuality benchmark rather than a document-level or sentence-level classification task. The benchmark is centered on **question answering**, but the actual prediction object is the generated answer’s character sequence. For Chinese, the unit was described as **“character components,”** although evaluation remained character-based. The task also explicitly includes **related observable overgeneration mistakes**, not only canonical hallucinations, reflecting the organizers’ view that unsupported over-specificity is an operationally important failure mode for instruction-tuned models [2504.11975].

A recurrent source of confusion is the phrase **“SemEval Task 3.”** In SemEval, task numbers are year-specific rather than persistent research tracks. **SemEval-2024 Task 3** was **“Multimodal Emotion Cause Analysis in Conversations,”** and **SemEval-2023 Task 3** was **“Detecting the Category, the Framing, and the Persuasion Techniques in Online News in a Multi-lingual Setup,”** so **SemEval-2025 Task 3** is not a continuation of either of those problems despite sharing the same ordinal label [2405.13049] [2305.03336].

## 2. Corpus construction, language coverage, and annotation

Mu-SHROOM covers **14 languages**. The **10 languages with validation and test sets** are **Modern Standard Arabic (AR), German (DE), English (EN), Spanish (ES), Finnish (FI), French (FR), Hindi (HI), Italian (IT), Swedish (SV), and Mandarin Chinese (ZH)**. The **4 test-only surprise languages** are **Catalan (CA), Czech (CS), Basque (EU), and Farsi (FA)**. In addition, the organizers released **unlabeled training data** for **English, Spanish, French, and Chinese**, including **prompts, model outputs, logits, and model identifiers** [2504.11975].

| Dataset component | Languages | Split information |
|---|---|---|
| Main languages | AR, DE, EN, ES, FI, FR, HI, IT, SV, ZH | **50 validation datapoints** and **150 test datapoints** per language |
| Surprise languages | CA, CS, EU, FA | **about 100 test datapoints** per language |
| Unlabeled training data | EN, ES, FR, ZH | prompts, model outputs, logits, model identifiers |

The data creation process began with automatic extraction of about **400 Wikipedia pages**, later expanded to **762 links**. For each main language, an organizer proficient in that language manually selected and read **200 Wikipedia pages**; for surprise languages, the number was **100 pages**. Each page received one question, and questions had to be **factual** and **closed**, meaning answerable by a constrained set of answers such as names, places, dates, or numbers. The page/question sets vary by language because Wikipedia coverage differs; the organizers note, for example, that about **75% of Hindi datapoints have no equivalent in other languages**, and Czech was likewise built from pages with no equivalent in other languages [2504.11975].

The answers to be annotated were generated from openly available instruction-tuned LLMs while varying **top-\(p\)**, **top-\(k\)**, and **temperature**. For each question, organizers manually selected **one output per question** for annotation, preferring answers that were **fluent and in the target language**, **relevant to the question**, and **appeared to contain hallucinations or otherwise interesting content to annotate**. A subset of the non-selected outputs became the unlabeled training material. The final annotated dataset covers outputs from **38 LLMs**, though these are not balanced across languages; **over 85% of the Hindi test set** comes from `nickmalhotra/ProjectIndus` [2504.11975].

Annotation was performed through a custom platform showing the **question**, the **model answer**, and the **source Wikipedia page**. Annotators were asked to highlight all spans unsupported by the page and were restricted to using **Wikipedia only** as an evidence source, although they could browse other Wikipedia pages if needed and record those in comments. All selected outputs were annotated by **at least three annotators**, usually the same three people for all **200 datapoints** in that language. English and Chinese used larger annotator pools: English datapoints could have **up to 12 annotators**, Chinese **up to 6**, and Spanish test data had **6 annotators**. The organizers emphasize that agreement is only moderate and highly language-dependent. Representative agreement values include **0.45** for English validation, **0.85** for Italian validation, **0.49** for English test, **0.51** for Spanish test, **0.58** for Chinese test, and **0.87** for Italian test, with English, Spanish, and Chinese showing notably lower overlap than several other languages [2504.11975].

## 3. Official evaluation and the role of annotator disagreement

Although Mu-SHROOM is described as a span-labeling task, the official scoring formalization is **character-based**. Let \(C_n\) be the set of character indices marked as hallucinated by annotator \(n\). Human annotations are binarized by majority vote at the character level, producing

\[
C_{\mathrm{bin}} = \left\{ c_i \mid 0.5 < \frac{1}{n}\sum \mathbf{1}\{c_i \in C_n\} \right\}.
\]

Given a system prediction \(\hat C_{\mathrm{bin}}\), the main ranking metric is

\[
\mathrm{IoU} = \frac{|C_{\mathrm{bin}} \cap \hat C_{\mathrm{bin}}|}{|C_{\mathrm{bin}} \cup \hat C_{\mathrm{bin}}|}.
\]

The second official metric is a calibration-oriented correlation score. For a datapoint of length \(k\), the empirical hallucination probability for each character is

\[
\Pr c_i = \frac{1}{n}\sum \mathbf{1}\{c_i \in C_n\},
\]

and the system predicts a probabilistic vector

\[
\hat c = (p(c_1 \mid O), \ldots, p(c_k \mid O)).
\]

The correlation metric is then

\[
\rho = \mathrm{Spearman}(c, \hat c).
\]

Official rankings were based on the **highest IoU**, with ties broken by **\(\rho\)** [2504.11975].

This dual-metric design is a direct response to annotation ambiguity. IoU presupposes a majority-vote gold standard, while \(\rho\) rewards systems whose per-character confidence aligns with the distribution of human judgments. Participants could submit **binary predictions**, **continuous predictions**, or both. If one form was missing, the organizers generated it heuristically:

\[
\hat C_{\mathrm{bin}} = \{ c_i \mid p(c_i \mid O) > 0.5 \}
\]

and

\[
\hat c = (\mathbf{1}\{c_1 \in \hat C_{\mathrm{bin}}\}, \ldots, \mathbf{1}\{c_k \in \hat C_{\mathrm{bin}}\}).
\]

Edge cases were also specified. For datapoints with no hallucinations, **IoU is 1** if the system predicts no hallucinated characters and **0** otherwise; **\(\rho\) is 1** if the model assigns the same probability to all tokens and **0** otherwise [2504.11975].

A central implication of this evaluation design is that span localization and calibration are separable competencies. The organizers explicitly interpret \(\rho\) as a measure of **human calibration**, and later analyses show that many systems obtain **IoU \(>\rho\)**, indicating that making binary selections is often easier than matching human gradience. Conversely, some systems achieve the reverse pattern, suggesting plausible ranking of characters without accurate hard-span commitment [2504.11975].

## 4. Participation, baselines, and the method landscape

Mu-SHROOM drew unusually strong participation. The task received **2,618 submissions** from **43 teams**, and **35 teams** submitted system papers. On average, **27.2 teams** participated per language. English had the highest participation with **41 teams**, followed by Spanish (**32**) and French (**30**), while the lowest participation occurred in the surprise languages, with **21 teams** for Catalan and **23 teams** each for Czech, Basque, and Farsi [2504.11975].

The official baselines were intentionally simple. They consisted of a fine-tuned **XLM-R** token classifier (`FacebookAI/xlm-roberta-base`), trained for **5 epochs** with learning rate **\(2e{-5}\)**, plus two trivial heuristics: **mark-all**, which labels every character as hallucinated, and **mark-none**, which labels none. The organizers emphasize that **mark-all** is surprisingly nontrivial because the dataset is intentionally enriched for hallucination-containing outputs. Most teams nevertheless outperformed the baselines [2504.11975].

The methodological space was highly diverse. The overview paper explicitly lists **fine-tuned token-level classifiers**, **NER-based or QA-based formulations**, **prompting and zero-shot methods**, **retrieval-augmented generation (RAG)**, **ensembles**, **multistage pipelines**, **synthetic data augmentation**, **cross-lingual transfer via translation**, **white-box methods using logits, hidden states, or uncertainty**, and methods comparing outputs against **external references** or against **larger LLMs**. Two aggregate findings are especially notable. First, **52.60%** of submissions used **RAG**, and these achieved significantly higher scores on both metrics, with **IoU: Mann-Whitney \(p < 10^{-59}\), common-language effect size \(f = 69.80\%\)** and **\(\rho: p < 10^{-39}, f = 66.16\%\)**. Second, **36.98%** of submissions were prompt-based, and prompt-only methods did **not** have statistically distinct IoU from non-prompt systems, but they had significantly **lower \(\rho\)**, with **\(p < 0.002\)** and **\(f = 45.96\%\)**. Submissions relying mainly on the provided Mu-SHROOM data also tended to score worse: **34.88%** of submissions primarily used the provided data, and these had lower **IoU (\(p < 10^{-21}, f = 37.60\%\))** and lower **\(\rho\) (\(p < 10^{-22}, f = 37.28\%\))** [2504.11975].

The organizers also reported a model-family analysis based on self-reported components. **DeepSeek** was used in only **2.32%** of submissions but was associated with a **77.87%** chance of outranking a randomly chosen non-DeepSeek submission on IoU and **80.15%** on \(\rho\). **Claude**, **GPT**, and **Qwen** were also associated with stronger performance, whereas **Llama**, **Flan-T5**, and **BERT** tended to underperform relative to others. This suggests that, within Mu-SHROOM, external evidence and calibration mechanisms mattered at least as much as raw classifier architecture [2504.11975].

## 5. Representative system designs

The published system papers show that Mu-SHROOM supported several distinct solution families rather than a single dominant architecture. One prominent family was **retrieval-grounded direct extraction**. The **UCSC** system used a three-stage pipeline—**context retrieval**, **hallucinated content detection**, and **span mapping**—then added **prompt optimization with MiPROv2** and a **multi-system combination** for soft labels. Retrieval was performed with **Perplexity Sonar Pro**, and the strongest configuration relied on a direct extraction prompt rather than knowledge-graph verification. UCSC achieved the **highest overall performance**, ranking **#1 in average position across all languages** [2505.03030].

A second family combined **prompt-based weak labeling** with **LLM adjudication**. The **MSA** system treated one LLM as a **Span Extractor Model** and the other three as **Voting Adjudicator Models**, using **Gemini-2.0-Flash-Exp**, **Qwen-2.5-Max**, **GPT-4o**, and **DeepSeek-V3**. Candidate spans were accepted when the mean verifier score satisfied \(p_i \geq 0.7\), after which **fuzzy matching** with threshold **0.9** refined alignment. This system ranked **1st in Arabic and Basque**, **2nd in German, Swedish, and Finnish**, and **3rd in Czech, Farsi, and French** [2505.20880].

A third family exploited **black-box self-disagreement** rather than external retrieval. The **keepitsimple** submission proposed a **training-free** multilingual detector based on the variability of **stochastically sampled alternative responses**. Using **Llama-3.2-3B-Instruct**, the system generated **20 candidate responses** per input, compared local spans from the original answer to matched spans in the samples, and aggregated **semantic entropy**, **lexical entropy**, and a **frequency-based mismatch score**. It ranked **18th on average among 40+ submissions**, with best rank **10th in Chinese**, and illustrates that Mu-SHROOM could also be approached as an uncertainty-estimation problem rather than a retrieval problem [2505.17485].

Other competitive systems hybridized retrieval with supervised or semi-supervised learning. **NCL-UoR** adapted **RefChecker** and **SelfCheckGPT** into **Modified-RefChecker (MRC)** and **Modified-SelfCheckGPT-H (MSCGH)**, adding external knowledge from **Google CSE** and **Wikipedia**, multilingual keyword extraction, repeated prompting with **\(N=5\)** runs, overlap-based interval merging, and thresholding at **0.5**. Their best submitted system reported **average IoU = 0.5310** and **average COR = 0.5669** [2503.01921]. **ATLANTIS** explored both **with-context** and **without-context** approaches, using **Gemini 1.5 Pro**, an **XLM-RoBERTa large** token classifier, and a **LoRA**-tuned **Llama-3.2-3B-Instruct**, all trained or evaluated with synthetic data built from **MKQA**. This team achieved **1st in Spanish**, **3rd in English**, **5th in German**, and **11th in French**, while emphasizing that adding relevant external context improved performance for every method they tested [2508.05179]. **TUM-MiKaNi** proposed a **three-component architecture**—a **Retrieval-based Fact Verification Model (RFVM)**, a **BERT-based Model (BM)**, and an **SVR** combiner—targeting **token-level multilingual hallucination detection** with explicit fact verification against **English Wikipedia**. It reached **top-10 results in eight languages**, including English [2507.00579].

Taken together, these systems indicate that Mu-SHROOM rewarded at least four distinct capabilities: high-quality external evidence retrieval, robust multilingual prompting, calibration of soft hallucination probabilities, and precise projection of factual judgments back to character spans. This suggests that the benchmark is as much about **evidence alignment and localization** as about factual verification per se.

## 6. Results, language effects, and long-term significance

The official winners by IoU illustrate a geographically and methodologically heterogeneous leaderboard. **MSA** won **Arabic** and **Basque**. **UCSC** won **Catalan, German, Finnish, Italian, and Swedish** and appeared in the top 3 for **11 languages**. **AILS-NTUA** won **Czech** and **Farsi**. **iai_MSU** won **English**. **ATLANTIS** won **Spanish**. **Deloitte** won **French**. **CCNU** won **Hindi**. **YNU-HPCC** won **Chinese**. The language-winning IoU values ranged from **0.5311** in Spanish to **0.7872** in Italian, with the strongest single leaderboard entry reported for **Italian** [2504.11975].

The organizers also analyzed difficulty by averaging non-baseline system scores per language. Ranked by mean IoU, the languages were led by **Italian** (**0.51**) and **Hindi** (**0.50**), followed by **Catalan** (**0.49**) and **Finnish** (**0.48**). The most difficult languages on average were **Spanish** (**0.31**), **Chinese** (**0.37**), and **Czech** (**0.37**). Spanish was especially revealing: **ATLANTIS** won by IoU in Spanish, but its **\(\rho\)** was reported as nearly zero, which the organizers treat as a concrete example of strong binary decisions with weak probabilistic ranking [2504.11975].

Several broader findings temper the raw leaderboard narrative. Bootstrap analysis showed that **most rank differences are not statistically stable**: in many cases, the probability that a system truly outranks the one below it under resampling is well under **0.95**. At the item level, every language showed a **low to moderate positive Spearman correlation** between inter-annotator agreement and performance, with correlations for IoU roughly between **0.16 and 0.37** and for \(\rho\) between **0.04 and 0.27**. The organizers therefore argue that low human agreement is a genuine contributor to task difficulty, but not the only one [2504.11975].

The benchmark’s long-term significance lies in the combination of multilinguality, fine-grained span annotation, and explicit treatment of annotation ambiguity. Mu-SHROOM established that **retrieval matters a great deal**, that **prompt-only systems are not sufficient for strong calibration**, and that **character-boundary disagreement** is itself a first-order research problem. The organizers explicitly state that even top-scoring systems can perform **at random on the most challenging items**, and that even the best systems still fail to detect **20% or more** of hallucination spans. This suggests that future progress will likely depend not only on stronger verifiers, but also on better reference retrieval, better multilingual evidence alignment, and better modeling of uncertain or disputed boundaries [2504.11975].

Source: https://www.emergentmind.com/topics/semeval-2025-task-3