---
title: 'SemEval-2025 Task 11: Bridging Emotion Gaps'
url: https://www.emergentmind.com/topics/semeval-2025-shared-task-11
type: topic
---

# SemEval-2025 Task 11: Bridging Emotion Gaps

SemEval-2025 Shared Task 11, officially titled *Bridging the Gap in Text-Based Emotion Detection*, is a multilingual benchmark on perceived emotion recognition from text. It targets the emotion that most people believe the speaker might have felt, rather than the speaker’s “true” internal emotional state, and organizes this problem into multilabel emotion detection, emotion intensity prediction, and cross-lingual emotion detection. The task covers **32 languages** from **seven distinct language families**, emphasizes predominantly **low-resource and underrepresented languages**, and is built around **over 100,000 multi-labeled instances** with six emotional classes, while the annotation scheme also includes a **neutral** category [2503.07269].

## 1. Task definition and conceptual scope

The task is framed around **perceived emotion**: systems are asked to infer the emotion that general third-party observers would attribute to the speaker on the basis of a sentence or short text snippet. This formulation distinguishes the benchmark from settings that target either the speaker’s internal state or the reader’s own affective reaction. The organizers motivate this framing by arguing that emotion expression is culturally contextualized, subjectively perceived, and differently realized across languages and communities [2503.07269].

The benchmark is explicitly multilingual and cross-cultural in design. The central “gap” is the mismatch between high-resource-centric emotion NLP and a multilingual world in which emotional meaning is expressed across diverse scripts, domains, and sociolinguistic settings. The organizer paper states that prior work has focused mostly on high-resource languages and that multilingual datasets built primarily through translation may fail to preserve culture-specific emotional expression. This yields a resource-building agenda as much as a competition agenda: the task is intended to extend emotion detection beyond the usual concentration on English and a small set of other widely studied languages [2503.07269].

A recurring implication across system papers is that multilingual transfer alone is insufficient. Several participants explicitly treat emotional perception as language- and culture-specific, and their strongest systems therefore rely on language-specific adaptation, target-language-aware prompting, or language-conditioned retrieval rather than a single universal formulation [2508.01161].

## 2. Corpora, language coverage, and annotation design

The organizer paper states that the shared task spans **32 languages** across **Africa, Asia, Latin America, North America, and Europe**. Track coverage is not identical across subtasks. **Track A** covers languages such as Afrikaans, Algerian Arabic, Amharic, Chinese, Emakhuwa, English, German, Hausa, Hindi, Igbo, Indonesian, Javanese, Kinyarwanda, Marathi, Moroccan Arabic, Nigerian-Pidgin, Oromo, Portuguese-Brazilian, Portuguese-Mozambican, Romanian, Russian, Somali, Sundanese, Swahili, Swedish, Tatar, Tigrinya, Ukrainian, and Yoruba. **Track C** adds **isiXhosa** and **isiZulu**, which are used as cross-lingual-only targets. The paper is internally inconsistent about one Track B language listing: one passage mentions eleven datasets with intensity annotations, while another listing includes Sundanese and excludes Spanish; the results table reports **11 languages** and includes **Spanish** rather than **Sundanese** [2503.07269].

The data is multi-domain. The organizer description names **social media**, **Reddit**, **YouTube**, **Twitter**, **Weibo**, **personal narratives**, **diaries**, **talks**, **speeches**, **literary text**, **news data**, **reviews**, and both **human-written and machine-generated data**. System papers repeatedly identify **BRIGHTER** as the main multilingual resource and describe **EthioEmo** as an additional multilingual emotion dataset for Ethiopian languages and low-resource evaluation settings [2508.01161].

The annotation format supports overlapping emotions. The label inventory used for prediction is the six-way set **joy, sadness, anger, fear, surprise, and disgust**, while **neutral** is part of the annotation scheme but not part of the Track A prediction target. For intensity-enabled data, each emotion is annotated on a **4-point scale**: **0** for no emotion, **1** for low intensity, **2** for moderate or medium intensity, and **3** for high intensity [2503.07269].

Label aggregation follows explicit rules. An emotion is considered present if **at least two annotators** select intensity **1, 2, or 3** and the **average score exceeds** the threshold \(T = 0.5\). For Track B, final intensity scores are obtained by averaging selected intensity values and **rounding up to the nearest whole number**. Annotation reliability is reported using **Split-Half Class Match Percentage (SHCMP)**, with values ranging from **60%** to **more than 90%**, which the organizer paper presents as evidence of generally high annotation quality [2503.07269].

## 3. Tracks and evaluation protocol

The shared task is divided into three official tracks. Track A is monolingual multilabel emotion detection; Track B is monolingual emotion intensity prediction; Track C is cross-lingual multilabel emotion detection with **no target-language training data** allowed [2503.07269].

| Track | Prediction target | Official metric |
|---|---|---|
| A | Presence or absence of six emotions, \( \hat{y} \in \{0,1\}^6 \) | Average macro \(F\)-score |
| B | Emotion intensities, \( \hat{s} \in \{0,1,2,3\}^6 \) | Pearson correlation coefficient |
| C | Cross-lingual multilabel detection without target-language training data, \( \hat{y} \in \{0,1\}^6 \) | Average macro \(F\)-score |

In the organizer formulation, Track A can be written as
\[
f_A: x \mapsto \hat{y}, \quad \hat{y}\in\{0,1\}^6,
\]
and Track B as
\[
f_B: x \mapsto \hat{s}, \quad \hat{s}\in\{0,1,2,3\}^6.
\]
Track C is defined by the additional training constraint
\[
f_C: x^{(L_t)} \mapsto \hat{y}, \quad \text{trained on } \bigcup_{L\neq L_t} D_L,
\]
that is, transfer into a target language \(L_t\) without using \(D_{L_t}^{train}\) [2503.07269].

For Tracks A and C, the official score is **average macro F-score**, i.e. macro-averaging across the six emotion labels. For Track B, the official score is **Pearson correlation**, reported as one scalar per language/system pair. System papers follow the same official setup: Track A is routinely treated as a six-label multilabel problem, Track B as six independent ordinal or categorical intensity predictions, and Track C as target-language-zero-shot transfer with respect to labeled training data [2508.01161].

## 4. Competition organization, baselines, and participation

The task was hosted on **Codabench** and organized in **two phases**. The **development phase** used an open leaderboard and permitted up to **999 submissions per participant**. The **evaluation phase** used a closed leaderboard, allowed up to **3 submissions per participant**, and ranked teams by the **last submission**. The organizers also released **pilot datasets**, a **starter kit on GitHub**, an **evaluation script**, **resources for beginners**, a **Q&A session**, and a **writing tutorial** [2503.07269].

Participation was unusually large for a SemEval shared task. The organizer paper reports **700+ registered participants**, **220 final submissions**, and **93 system description papers**. It also reports **153**, **52**, and **25** development-phase submissions for Tracks A, B, and C, and **220**, **96**, and **46** test-phase submissions, respectively. Participant affiliations span **33 countries**, and the task is described as the **most popular competition on Codabench in 2024**. The average participation level was **11 languages per team** [2503.07269].

The organizer baselines were deliberately simple but informative. **Mbaseline** is a majority-class baseline. **Rbaseline** is a **RoBERTa** baseline fine-tuned separately for each language in Tracks A and B. For Track C, the RoBERTa baseline is trained in a **family-based leave-one-language-out** configuration. The organizer paper specifies special cases: for **tat**, training uses **rus + ukr**; for **pcm**, training uses **swa + yor**; and for **chn**, the cross-lingual baseline uses **rus** as the source language. Baseline performance is highly uneven across languages, which already indicates the strong resource sensitivity of the benchmark [2503.07269].

## 5. Methodological landscape and leaderboard structure

The organizer analysis identifies a clear high-level pattern: the strongest systems were typically based on **LLMs**, **fine-tuning or PEFT/LoRA**, **prompt engineering**, and often **ensembling**. In Tracks A and B, **PAI** is the dominant team. Its system combines **ChatGPT-4o**, **DeepSeek-V3**, **Gemma-9b**, **Qwen-2.5-32b**, and **Mistral-Small-24B**, with ensemble layers based on **neural networks**, **XGBoost**, **LightGBM**, **linear regression**, and **weighted voting**, plus **AdaLoRA** fine-tuning for Gemma and Qwen and iterative prompt optimization. In Track C, **Deepwave** is the strongest overall system, using **Gemma-2** fine-tuning, **Chain-of-Thought prompting**, **5-fold cross-validation**, model merging by averaging prediction probabilities, one **LoRA** module per target language, and augmented data generated through CoT prompting [2503.07269].

The broader system-paper literature shows that the task supported a wide range of competitive formulations rather than a single canonical architecture. **EmoRAG** treats Track A as a **training-free multilingual emotion detection system**: it builds a retrieval corpus from labeled training data, uses **LangChain n-gram** or **BGE-M3** retrieval, prompts multiple pretrained LLMs in **English**, and aggregates Boolean JSON outputs; its strongest aggregator is **label-wise dev-F1-weighted voting**, and its reported test average is **F1-micro = 0.638** and **F1-macro = 0.590** [2506.04409]. **CSIRO-LT** reports that the most effective strategy was usually to **fine-tune a pretrained multilingual LLM with LoRA separately for each language**, rather than relying on zero-shot prompting, few-shot prompting, English-bridged continual adaptation, or external affect instruction tuning [2508.01161]. **JNLP** compares a fine-tuned **XLM-RoBERTa-Large** classifier with instruction-tuned **Qwen** and **Llama2** variants, and contrasts a direct **base** formulation with a decomposed **pairwise** formulation; it reports **Top 4** Track A performance across **10 languages**, **1st in Hindi**, and **Top 5** Track B performance in **7 languages** [2505.13244].

Other submissions demonstrate that Track 11 was not exclusively an LLM contest. **UoB-NLP** uses **adapter-based fine-tuning** over **xlm-roberta-base** and **afro-xlmr-base**, and finds that **target-language-ready task adapters** are the strongest overall strategy, with reported competitive rankings for **Tigrinya**, **Kinyarwanda**, **Amharic**, **Oromo**, **Hausa**, and **Igbo** [2504.08543]. **PromotionGo** advances a **feature-centric framework** in which document representation, dimensionality reduction, and downstream learner are varied by language; the paper reports that **TF-IDF** remains highly effective in low-resource settings, while **SBERT + MLP** is strongest in **18 out of 28 languages**, and that **PCA** is selectively useful rather than uniformly beneficial [2507.08499]. **Team A** uses **multilingual-e5-large-instruct** embeddings with a **dropout + fully connected sigmoid** classifier and **BCE with label smoothing**, obtaining its strongest macro \(F_1\) values on **Hindi (0.8901)**, **Russian (0.8831)**, and **Marathi (0.8657)** [2502.19856]. **Chinchunmei** reformulates both tracks as structured generation with **LLaMA3-Instruct-8B**, and finds that **DPO** improves Track B more clearly than Track A, whereas **multilingual training degraded English performance** [2507.15714].

## 6. Empirical patterns, significance, and unresolved issues

The main empirical conclusion of the organizer paper is that performance varies sharply by language and track. High-resource settings remain easier: in Track A, the paper highlights strong outcomes for **English**, **Russian**, **Spanish**, and **Hindi**; in Track B, **Russian** reaches **0.925** for the top system; and in Track C, the highest gains over the family-based RoBERTa baseline are again concentrated in better-supported languages. Representative organizer comparisons include **English Track A: PAI 0.823 vs RoBERTa 0.708**, **Spanish Track A: PAI 0.849 vs RoBERTa 0.774**, **Hindi Track A: JNLP 0.926 vs RoBERTa 0.855**, **Oromo Track A: Tewodros 0.616 vs RoBERTa 0.126**, **Algerian Arabic Track B: PAI 0.650 vs RoBERTa 0.016**, **English Track C: Deepwave 0.797 vs RoBERTa 0.375**, and **Yoruba Track C: maomao 0.360 vs RoBERTa 0.053** [2503.07269].

Low-resource and cross-lingual settings remain substantially harder. The organizer paper explicitly states that Track C scores are generally lower than Track A, especially for low-resource languages, and system papers reinforce this conclusion. UoB-NLP emphasizes that adapter-based transfer can outperform larger LLMs in several low-resource languages, but also notes dev–test mismatch and resource limitations in language-adapter training [2504.08543]. PromotionGo shows that sparse lexical features may still outperform richer contextual representations in some languages, which suggests that multilingual semantic transfer is neither uniform nor architecture-agnostic [2507.08499]. Chinchunmei’s finding that multilingual training can reduce English performance further indicates that multilingual coverage is not equivalent to cross-cultural emotional alignment [2507.15714].

Another notable pattern is the relatively limited role of external labeled resources. The organizer paper reports that **92.1%** of combined systems used **no additional data**, implying that pretrained models, prompting, retrieval, and parameter-efficient tuning were often sufficient to build competitive entries [2503.07269]. This suggests a benchmark ecology in which adaptation strategy, label formulation, and aggregation often matter more than large task-specific auxiliary corpora.

The task’s broader significance lies in its combination of scale, language diversity, and explicit attention to perceived emotion rather than sentiment polarity alone. It institutionalizes the claim that multilingual language models are **not automatically multicultural**, and it provides a standardized setting in which the interaction among language resource level, cultural specificity, model family, and task formulation can be studied directly. At the same time, the documentation reveals some unresolved issues, including internal inconsistencies in one Track B language listing and uneven difficulty across languages. A plausible implication is that SemEval-2025 Task 11 functions simultaneously as a leaderboard benchmark and as a diagnostic resource for multilingual affective computing, especially in low-resource and cross-lingual regimes [2503.07269].

Source: https://www.emergentmind.com/topics/semeval-2025-shared-task-11