SemEval-2025 Task 3: Mu-SHROOM Benchmark

• The paper introduces Mu-SHROOM, a multilingual benchmark that precisely localizes hallucinations in LLM outputs via token/character-level span labeling.
• SemEval-2025 Task 3 is designed to detect unsupported content by comparing generated answers with Wikipedia references across 14 languages.
• The evaluation framework combines character-level IoU with a calibration metric to effectively address annotation ambiguities in detecting overgeneration errors.

SemEval-2025 Task 3 is Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakes, a SemEval benchmark devoted to detecting unsupported content in the outputs of instruction-tuned LLMs. Mu-SHROOM addresses general-purpose LLMs in 14 languages and frames hallucination detection as a span-labeling task over generated answers: the objective is to identify where unsupported content occurs, not merely whether an answer is globally wrong. In the official formulation, the annotation target is the characters/tokens in the generated answer that should be considered hallucinated or overgenerated relative to a Wikipedia reference page, and the main ranking criterion is character-level Intersection-over-Union (IoU), complemented by a correlation metric that models annotator uncertainty (Vázquez et al., 16 Apr 2025).

1. Task identity and problem formulation

Mu-SHROOM extends the earlier SHROOM shared task from SemEval-2024. The 2025 edition broadens the scope in three explicit ways: it is multilingual, it addresses general-purpose LLMs, and it introduces token/character-level annotations for precise localization. Each datapoint consists of a question, an LLM-generated answer, and a Wikipedia reference page from which the question was derived. Participants must identify the spans in the answer that contain unsupported factual content. The organizers define hallucination as “content that contains or describes facts that are not supported by a provided reference”, and the annotation instructions sharpen this further: hallucinations are cases where the answer is “more specific than it should be, given the information available in the Wikipedia page.” Annotators were instructed to highlight the minimum number of characters that would need to be edited or deleted to make the answer correct, while preferring content words over function words where possible (Vázquez et al., 16 Apr 2025).

This formulation makes Mu-SHROOM a fine-grained factuality benchmark rather than a document-level or sentence-level classification task. The benchmark is centered on question answering, but the actual prediction object is the generated answer’s character sequence. For Chinese, the unit was described as “character components,” although evaluation remained character-based. The task also explicitly includes related observable overgeneration mistakes, not only canonical hallucinations, reflecting the organizers’ view that unsupported over-specificity is an operationally important failure mode for instruction-tuned models (Vázquez et al., 16 Apr 2025).

A recurrent source of confusion is the phrase “SemEval Task 3.” In SemEval, task numbers are year-specific rather than persistent research tracks. SemEval-2024 Task 3 was “Multimodal Emotion Cause Analysis in Conversations,” and SemEval-2023 Task 3 was “Detecting the Category, the Framing, and the Persuasion Techniques in Online News in a Multi-lingual Setup,” so SemEval-2025 Task 3 is not a continuation of either of those problems despite sharing the same ordinal label (Wang et al., 2024, Hasanain et al., 2023).

2. Corpus construction, language coverage, and annotation

Mu-SHROOM covers 14 languages. The 10 languages with validation and test sets are Modern Standard Arabic (AR), German (DE), English (EN), Spanish (ES), Finnish (FI), French (FR), Hindi (HI), Italian (IT), Swedish (SV), and Mandarin Chinese (ZH). The 4 test-only surprise languages are Catalan (CA), Czech (CS), Basque (EU), and Farsi (FA). In addition, the organizers released unlabeled training data for English, Spanish, French, and Chinese, including prompts, model outputs, logits, and model identifiers (Vázquez et al., 16 Apr 2025).

Dataset component	Languages	Split information
Main languages	AR, DE, EN, ES, FI, FR, HI, IT, SV, ZH	50 validation datapoints and 150 test datapoints per language
Surprise languages	CA, CS, EU, FA	about 100 test datapoints per language
Unlabeled training data	EN, ES, FR, ZH	prompts, model outputs, logits, model identifiers

The data creation process began with automatic extraction of about 400 Wikipedia pages, later expanded to 762 links. For each main language, an organizer proficient in that language manually selected and read 200 Wikipedia pages; for surprise languages, the number was 100 pages. Each page received one question, and questions had to be factual and closed, meaning answerable by a constrained set of answers such as names, places, dates, or numbers. The page/question sets vary by language because Wikipedia coverage differs; the organizers note, for example, that about 75% of Hindi datapoints have no equivalent in other languages, and Czech was likewise built from pages with no equivalent in other languages (Vázquez et al., 16 Apr 2025).

The answers to be annotated were generated from openly available instruction-tuned LLMs while varying top-$p$, top-$k$, and temperature. For each question, organizers manually selected one output per question for annotation, preferring answers that were fluent and in the target language, relevant to the question, and appeared to contain hallucinations or otherwise interesting content to annotate. A subset of the non-selected outputs became the unlabeled training material. The final annotated dataset covers outputs from 38 LLMs, though these are not balanced across languages; over 85% of the Hindi test set comes from nickmalhotra/ProjectIndus (Vázquez et al., 16 Apr 2025).

Annotation was performed through a custom platform showing the question, the model answer, and the source Wikipedia page. Annotators were asked to highlight all spans unsupported by the page and were restricted to using Wikipedia only as an evidence source, although they could browse other Wikipedia pages if needed and record those in comments. All selected outputs were annotated by at least three annotators, usually the same three people for all 200 datapoints in that language. English and Chinese used larger annotator pools: English datapoints could have up to 12 annotators, Chinese up to 6, and Spanish test data had 6 annotators. The organizers emphasize that agreement is only moderate and highly language-dependent. Representative agreement values include 0.45 for English validation, 0.85 for Italian validation, 0.49 for English test, 0.51 for Spanish test, 0.58 for Chinese test, and 0.87 for Italian test, with English, Spanish, and Chinese showing notably lower overlap than several other languages (Vázquez et al., 16 Apr 2025).

3. Official evaluation and the role of annotator disagreement

Although Mu-SHROOM is described as a span-labeling task, the official scoring formalization is character-based. Let $C_n$ be the set of character indices marked as hallucinated by annotator $n$. Human annotations are binarized by majority vote at the character level, producing

$$C_{\mathrm{bin}} = \left\{ c_i \mid 0.5 < \frac{1}{n}\sum \mathbf{1}\{c_i \in C_n\} \right\}.$$

Given a system prediction $\hat C_{\mathrm{bin}}$, the main ranking metric is

$$\mathrm{IoU} = \frac{|C_{\mathrm{bin}} \cap \hat C_{\mathrm{bin}}|}{|C_{\mathrm{bin}} \cup \hat C_{\mathrm{bin}}|}.$$

The second official metric is a calibration-oriented correlation score. For a datapoint of length $k$, the empirical hallucination probability for each character is

$$\Pr c_i = \frac{1}{n}\sum \mathbf{1}\{c_i \in C_n\},$$

and the system predicts a probabilistic vector

$\hat c = (p(c_1 \mid O), \ldots, p(c_k \mid O)).$

The correlation metric is then

$k$0

Official rankings were based on the highest IoU, with ties broken by $k$1 (Vázquez et al., 16 Apr 2025).

This dual-metric design is a direct response to annotation ambiguity. IoU presupposes a majority-vote gold standard, while $k$2 rewards systems whose per-character confidence aligns with the distribution of human judgments. Participants could submit binary predictions, continuous predictions, or both. If one form was missing, the organizers generated it heuristically:

$k$3

and

$k$4

Edge cases were also specified. For datapoints with no hallucinations, IoU is 1 if the system predicts no hallucinated characters and 0 otherwise; $k$5 is 1 if the model assigns the same probability to all tokens and 0 otherwise (Vázquez et al., 16 Apr 2025).

A central implication of this evaluation design is that span localization and calibration are separable competencies. The organizers explicitly interpret $k$6 as a measure of human calibration, and later analyses show that many systems obtain IoU $k$7, indicating that making binary selections is often easier than matching human gradience. Conversely, some systems achieve the reverse pattern, suggesting plausible ranking of characters without accurate hard-span commitment (Vázquez et al., 16 Apr 2025).

4. Participation, baselines, and the method landscape

Mu-SHROOM drew unusually strong participation. The task received 2,618 submissions from 43 teams, and 35 teams submitted system papers. On average, 27.2 teams participated per language. English had the highest participation with 41 teams, followed by Spanish (32) and French (30), while the lowest participation occurred in the surprise languages, with 21 teams for Catalan and 23 teams each for Czech, Basque, and Farsi (Vázquez et al., 16 Apr 2025).

The official baselines were intentionally simple. They consisted of a fine-tuned XLM-R token classifier (FacebookAI/xlm-roberta-base), trained for 5 epochs with learning rate $k$8, plus two trivial heuristics: mark-all, which labels every character as hallucinated, and mark-none, which labels none. The organizers emphasize that mark-all is surprisingly nontrivial because the dataset is intentionally enriched for hallucination-containing outputs. Most teams nevertheless outperformed the baselines (Vázquez et al., 16 Apr 2025).

The methodological space was highly diverse. The overview paper explicitly lists fine-tuned token-level classifiers, NER-based or QA-based formulations, prompting and zero-shot methods, retrieval-augmented generation (RAG), ensembles, multistage pipelines, synthetic data augmentation, cross-lingual transfer via translation, white-box methods using logits, hidden states, or uncertainty, and methods comparing outputs against external references or against larger LLMs. Two aggregate findings are especially notable. First, 52.60% of submissions used RAG, and these achieved significantly higher scores on both metrics, with IoU: Mann-Whitney $k$9, common-language effect size $C_n$0 and $C_n$1. Second, 36.98% of submissions were prompt-based, and prompt-only methods did not have statistically distinct IoU from non-prompt systems, but they had significantly lower $C_n$2, with $C_n$3 and $C_n$4. Submissions relying mainly on the provided Mu-SHROOM data also tended to score worse: 34.88% of submissions primarily used the provided data, and these had lower IoU ($C_n$5) and lower $C_n$6 ($C_n$7) (Vázquez et al., 16 Apr 2025).

The organizers also reported a model-family analysis based on self-reported components. DeepSeek was used in only 2.32% of submissions but was associated with a 77.87% chance of outranking a randomly chosen non-DeepSeek submission on IoU and 80.15% on $C_n$8. Claude, GPT, and Qwen were also associated with stronger performance, whereas Llama, Flan-T5, and BERT tended to underperform relative to others. This suggests that, within Mu-SHROOM, external evidence and calibration mechanisms mattered at least as much as raw classifier architecture (Vázquez et al., 16 Apr 2025).

5. Representative system designs

The published system papers show that Mu-SHROOM supported several distinct solution families rather than a single dominant architecture. One prominent family was retrieval-grounded direct extraction. The UCSC system used a three-stage pipeline—context retrieval, hallucinated content detection, and span mapping—then added prompt optimization with MiPROv2 and a multi-system combination for soft labels. Retrieval was performed with Perplexity Sonar Pro, and the strongest configuration relied on a direct extraction prompt rather than knowledge-graph verification. UCSC achieved the highest overall performance, ranking #1 in average position across all languages (Huang et al., 5 May 2025).

A second family combined prompt-based weak labeling with LLM adjudication. The MSA system treated one LLM as a Span Extractor Model and the other three as Voting Adjudicator Models, using Gemini-2.0-Flash-Exp, Qwen-2.5-Max, GPT-4o, and DeepSeek-V3. Candidate spans were accepted when the mean verifier score satisfied $C_n$9, after which fuzzy matching with threshold 0.9 refined alignment. This system ranked 1st in Arabic and Basque, 2nd in German, Swedish, and Finnish, and 3rd in Czech, Farsi, and French (Hikal et al., 27 May 2025).

A third family exploited black-box self-disagreement rather than external retrieval. The keepitsimple submission proposed a training-free multilingual detector based on the variability of stochastically sampled alternative responses. Using Llama-3.2-3B-Instruct, the system generated 20 candidate responses per input, compared local spans from the original answer to matched spans in the samples, and aggregated semantic entropy, lexical entropy, and a frequency-based mismatch score. It ranked 18th on average among 40+ submissions, with best rank 10th in Chinese, and illustrates that Mu-SHROOM could also be approached as an uncertainty-estimation problem rather than a retrieval problem (Vemula et al., 23 May 2025).

Other competitive systems hybridized retrieval with supervised or semi-supervised learning. NCL-UoR adapted RefChecker and SelfCheckGPT into Modified-RefChecker (MRC) and Modified-SelfCheckGPT-H (MSCGH), adding external knowledge from Google CSE and Wikipedia, multilingual keyword extraction, repeated prompting with $n$0 runs, overlap-based interval merging, and thresholding at 0.5. Their best submitted system reported average IoU = 0.5310 and average COR = 0.5669 (Hong et al., 2 Mar 2025). ATLANTIS explored both with-context and without-context approaches, using Gemini 1.5 Pro, an XLM-RoBERTa large token classifier, and a LoRA-tuned Llama-3.2-3B-Instruct, all trained or evaluated with synthetic data built from MKQA. This team achieved 1st in Spanish, 3rd in English, 5th in German, and 11th in French, while emphasizing that adding relevant external context improved performance for every method they tested (Kobus et al., 7 Aug 2025). TUM-MiKaNi proposed a three-component architecture—a Retrieval-based Fact Verification Model (RFVM), a BERT-based Model (BM), and an SVR combiner—targeting token-level multilingual hallucination detection with explicit fact verification against English Wikipedia. It reached top-10 results in eight languages, including English (Anschütz et al., 1 Jul 2025).

Taken together, these systems indicate that Mu-SHROOM rewarded at least four distinct capabilities: high-quality external evidence retrieval, robust multilingual prompting, calibration of soft hallucination probabilities, and precise projection of factual judgments back to character spans. This suggests that the benchmark is as much about evidence alignment and localization as about factual verification per se.

6. Results, language effects, and long-term significance

The official winners by IoU illustrate a geographically and methodologically heterogeneous leaderboard. MSA won Arabic and Basque. UCSC won Catalan, German, Finnish, Italian, and Swedish and appeared in the top 3 for 11 languages. AILS-NTUA won Czech and Farsi. iai_MSU won English. ATLANTIS won Spanish. Deloitte won French. CCNU won Hindi. YNU-HPCC won Chinese. The language-winning IoU values ranged from 0.5311 in Spanish to 0.7872 in Italian, with the strongest single leaderboard entry reported for Italian (Vázquez et al., 16 Apr 2025).

The organizers also analyzed difficulty by averaging non-baseline system scores per language. Ranked by mean IoU, the languages were led by Italian (0.51) and Hindi (0.50), followed by Catalan (0.49) and Finnish (0.48). The most difficult languages on average were Spanish (0.31), Chinese (0.37), and Czech (0.37). Spanish was especially revealing: ATLANTIS won by IoU in Spanish, but its $n$1 was reported as nearly zero, which the organizers treat as a concrete example of strong binary decisions with weak probabilistic ranking (Vázquez et al., 16 Apr 2025).

Several broader findings temper the raw leaderboard narrative. Bootstrap analysis showed that most rank differences are not statistically stable: in many cases, the probability that a system truly outranks the one below it under resampling is well under 0.95. At the item level, every language showed a low to moderate positive Spearman correlation between inter-annotator agreement and performance, with correlations for IoU roughly between 0.16 and 0.37 and for $n$2 between 0.04 and 0.27. The organizers therefore argue that low human agreement is a genuine contributor to task difficulty, but not the only one (Vázquez et al., 16 Apr 2025).

The benchmark’s long-term significance lies in the combination of multilinguality, fine-grained span annotation, and explicit treatment of annotation ambiguity. Mu-SHROOM established that retrieval matters a great deal, that prompt-only systems are not sufficient for strong calibration, and that character-boundary disagreement is itself a first-order research problem. The organizers explicitly state that even top-scoring systems can perform at random on the most challenging items, and that even the best systems still fail to detect 20% or more of hallucination spans. This suggests that future progress will likely depend not only on stronger verifiers, but also on better reference retrieval, better multilingual evidence alignment, and better modeling of uncertain or disputed boundaries (Vázquez et al., 16 Apr 2025).