SemEval-2025 Task 11: Bridging Emotion Gaps

• SemEval-2025 Shared Task 11 is a multilingual benchmark targeting perceived emotion detection by evaluating multilabel classification and intensity prediction across 32 languages.
• The task employs innovative methodologies such as language-specific adaptation, prompt engineering, and ensemble strategies to tackle cultural and resource diversity in emotion recognition.
• Key evaluation metrics like macro F-score and Pearson correlation demonstrate significant performance improvements over baseline models in both high- and low-resource language settings.

SemEval-2025 Shared Task 11, officially titled Bridging the Gap in Text-Based Emotion Detection, is a multilingual benchmark on perceived emotion recognition from text. It targets the emotion that most people believe the speaker might have felt, rather than the speaker’s “true” internal emotional state, and organizes this problem into multilabel emotion detection, emotion intensity prediction, and cross-lingual emotion detection. The task covers 32 languages from seven distinct language families, emphasizes predominantly low-resource and underrepresented languages, and is built around over 100,000 multi-labeled instances with six emotional classes, while the annotation scheme also includes a neutral category (Muhammad et al., 10 Mar 2025).

1. Task definition and conceptual scope

The task is framed around perceived emotion: systems are asked to infer the emotion that general third-party observers would attribute to the speaker on the basis of a sentence or short text snippet. This formulation distinguishes the benchmark from settings that target either the speaker’s internal state or the reader’s own affective reaction. The organizers motivate this framing by arguing that emotion expression is culturally contextualized, subjectively perceived, and differently realized across languages and communities (Muhammad et al., 10 Mar 2025).

The benchmark is explicitly multilingual and cross-cultural in design. The central “gap” is the mismatch between high-resource-centric emotion NLP and a multilingual world in which emotional meaning is expressed across diverse scripts, domains, and sociolinguistic settings. The organizer paper states that prior work has focused mostly on high-resource languages and that multilingual datasets built primarily through translation may fail to preserve culture-specific emotional expression. This yields a resource-building agenda as much as a competition agenda: the task is intended to extend emotion detection beyond the usual concentration on English and a small set of other widely studied languages (Muhammad et al., 10 Mar 2025).

A recurring implication across system papers is that multilingual transfer alone is insufficient. Several participants explicitly treat emotional perception as language- and culture-specific, and their strongest systems therefore rely on language-specific adaptation, target-language-aware prompting, or language-conditioned retrieval rather than a single universal formulation (Chen et al., 2 Aug 2025).

2. Corpora, language coverage, and annotation design

The organizer paper states that the shared task spans 32 languages across Africa, Asia, Latin America, North America, and Europe. Track coverage is not identical across subtasks. Track A covers languages such as Afrikaans, Algerian Arabic, Amharic, Chinese, Emakhuwa, English, German, Hausa, Hindi, Igbo, Indonesian, Javanese, Kinyarwanda, Marathi, Moroccan Arabic, Nigerian-Pidgin, Oromo, Portuguese-Brazilian, Portuguese-Mozambican, Romanian, Russian, Somali, Sundanese, Swahili, Swedish, Tatar, Tigrinya, Ukrainian, and Yoruba. Track C adds isiXhosa and isiZulu, which are used as cross-lingual-only targets. The paper is internally inconsistent about one Track B language listing: one passage mentions eleven datasets with intensity annotations, while another listing includes Sundanese and excludes Spanish; the results table reports 11 languages and includes Spanish rather than Sundanese (Muhammad et al., 10 Mar 2025).

The data is multi-domain. The organizer description names social media, Reddit, YouTube, Twitter, Weibo, personal narratives, diaries, talks, speeches, literary text, news data, reviews, and both human-written and machine-generated data. System papers repeatedly identify BRIGHTER as the main multilingual resource and describe EthioEmo as an additional multilingual emotion dataset for Ethiopian languages and low-resource evaluation settings (Chen et al., 2 Aug 2025).

The annotation format supports overlapping emotions. The label inventory used for prediction is the six-way set joy, sadness, anger, fear, surprise, and disgust, while neutral is part of the annotation scheme but not part of the Track A prediction target. For intensity-enabled data, each emotion is annotated on a 4-point scale: 0 for no emotion, 1 for low intensity, 2 for moderate or medium intensity, and 3 for high intensity (Muhammad et al., 10 Mar 2025).

Label aggregation follows explicit rules. An emotion is considered present if at least two annotators select intensity 1, 2, or 3 and the average score exceeds the threshold $T = 0.5$. For Track B, final intensity scores are obtained by averaging selected intensity values and rounding up to the nearest whole number. Annotation reliability is reported using Split-Half Class Match Percentage (SHCMP), with values ranging from 60% to more than 90%, which the organizer paper presents as evidence of generally high annotation quality (Muhammad et al., 10 Mar 2025).

3. Tracks and evaluation protocol

The shared task is divided into three official tracks. Track A is monolingual multilabel emotion detection; Track B is monolingual emotion intensity prediction; Track C is cross-lingual multilabel emotion detection with no target-language training data allowed (Muhammad et al., 10 Mar 2025).

Track	Prediction target	Official metric
A	Presence or absence of six emotions, $\hat{y} \in \{0,1\}^6$	Average macro $F$-score
B	Emotion intensities, $\hat{s} \in \{0,1,2,3\}^6$	Pearson correlation coefficient
C	Cross-lingual multilabel detection without target-language training data, $\hat{y} \in \{0,1\}^6$	Average macro $F$-score

In the organizer formulation, Track A can be written as

$f_A: x \mapsto \hat{y}, \quad \hat{y}\in\{0,1\}^6,$

and Track B as

$$f_B: x \mapsto \hat{s}, \quad \hat{s}\in\{0,1,2,3\}^6.$$

Track C is defined by the additional training constraint

$$f_C: x^{(L_t)} \mapsto \hat{y}, \quad \text{trained on } \bigcup_{L\neq L_t} D_L,$$

that is, transfer into a target language $L_t$ without using $\hat{y} \in \{0,1\}^6$0 (Muhammad et al., 10 Mar 2025).

For Tracks A and C, the official score is average macro F-score, i.e. macro-averaging across the six emotion labels. For Track B, the official score is Pearson correlation, reported as one scalar per language/system pair. System papers follow the same official setup: Track A is routinely treated as a six-label multilabel problem, Track B as six independent ordinal or categorical intensity predictions, and Track C as target-language-zero-shot transfer with respect to labeled training data (Chen et al., 2 Aug 2025).

4. Competition organization, baselines, and participation

The task was hosted on Codabench and organized in two phases. The development phase used an open leaderboard and permitted up to 999 submissions per participant. The evaluation phase used a closed leaderboard, allowed up to 3 submissions per participant, and ranked teams by the last submission. The organizers also released pilot datasets, a starter kit on GitHub, an evaluation script, resources for beginners, a Q&A session, and a writing tutorial (Muhammad et al., 10 Mar 2025).

Participation was unusually large for a SemEval shared task. The organizer paper reports 700+ registered participants, 220 final submissions, and 93 system description papers. It also reports 153, 52, and 25 development-phase submissions for Tracks A, B, and C, and 220, 96, and 46 test-phase submissions, respectively. Participant affiliations span 33 countries, and the task is described as the most popular competition on Codabench in 2024. The average participation level was 11 languages per team (Muhammad et al., 10 Mar 2025).

The organizer baselines were deliberately simple but informative. Mbaseline is a majority-class baseline. Rbaseline is a RoBERTa baseline fine-tuned separately for each language in Tracks A and B. For Track C, the RoBERTa baseline is trained in a family-based leave-one-language-out configuration. The organizer paper specifies special cases: for tat, training uses rus + ukr; for pcm, training uses swa + yor; and for chn, the cross-lingual baseline uses rus as the source language. Baseline performance is highly uneven across languages, which already indicates the strong resource sensitivity of the benchmark (Muhammad et al., 10 Mar 2025).

5. Methodological landscape and leaderboard structure

The organizer analysis identifies a clear high-level pattern: the strongest systems were typically based on LLMs, fine-tuning or PEFT/LoRA, prompt engineering, and often ensembling. In Tracks A and B, PAI is the dominant team. Its system combines ChatGPT-4o, DeepSeek-V3, Gemma-9b, Qwen-2.5-32b, and Mistral-Small-24B, with ensemble layers based on neural networks, XGBoost, LightGBM, linear regression, and weighted voting, plus AdaLoRA fine-tuning for Gemma and Qwen and iterative prompt optimization. In Track C, Deepwave is the strongest overall system, using Gemma-2 fine-tuning, Chain-of-Thought prompting, 5-fold cross-validation, model merging by averaging prediction probabilities, one LoRA module per target language, and augmented data generated through CoT prompting (Muhammad et al., 10 Mar 2025).

The broader system-paper literature shows that the task supported a wide range of competitive formulations rather than a single canonical architecture. EmoRAG treats Track A as a training-free multilingual emotion detection system: it builds a retrieval corpus from labeled training data, uses LangChain n-gram or BGE-M3 retrieval, prompts multiple pretrained LLMs in English, and aggregates Boolean JSON outputs; its strongest aggregator is label-wise dev-F1-weighted voting, and its reported test average is F1-micro = 0.638 and F1-macro = 0.590 (Morozov et al., 4 Jun 2025). CSIRO-LT reports that the most effective strategy was usually to fine-tune a pretrained multilingual LLM with LoRA separately for each language, rather than relying on zero-shot prompting, few-shot prompting, English-bridged continual adaptation, or external affect instruction tuning (Chen et al., 2 Aug 2025). JNLP compares a fine-tuned XLM-RoBERTa-Large classifier with instruction-tuned Qwen and Llama2 variants, and contrasts a direct base formulation with a decomposed pairwise formulation; it reports Top 4 Track A performance across 10 languages, 1st in Hindi, and Top 5 Track B performance in 7 languages (Xue et al., 19 May 2025).

Other submissions demonstrate that Track 11 was not exclusively an LLM contest. UoB-NLP uses adapter-based fine-tuning over xlm-roberta-base and afro-xlmr-base, and finds that target-language-ready task adapters are the strongest overall strategy, with reported competitive rankings for Tigrinya, Kinyarwanda, Amharic, Oromo, Hausa, and Igbo (Leon et al., 11 Apr 2025). PromotionGo advances a feature-centric framework in which document representation, dimensionality reduction, and downstream learner are varied by language; the paper reports that TF-IDF remains highly effective in low-resource settings, while SBERT + MLP is strongest in 18 out of 28 languages, and that PCA is selectively useful rather than uniformly beneficial (Huang et al., 11 Jul 2025). Team A uses multilingual-e5-large-instruct embeddings with a dropout + fully connected sigmoid classifier and BCE with label smoothing, obtaining its strongest macro $\hat{y} \in \{0,1\}^6$1 values on Hindi (0.8901), Russian (0.8831), and Marathi (0.8657) (Sahil et al., 27 Feb 2025). Chinchunmei reformulates both tracks as structured generation with LLaMA3-Instruct-8B, and finds that DPO improves Track B more clearly than Track A, whereas multilingual training degraded English performance (Li et al., 21 Jul 2025).

6. Empirical patterns, significance, and unresolved issues

The main empirical conclusion of the organizer paper is that performance varies sharply by language and track. High-resource settings remain easier: in Track A, the paper highlights strong outcomes for English, Russian, Spanish, and Hindi; in Track B, Russian reaches 0.925 for the top system; and in Track C, the highest gains over the family-based RoBERTa baseline are again concentrated in better-supported languages. Representative organizer comparisons include English Track A: PAI 0.823 vs RoBERTa 0.708, Spanish Track A: PAI 0.849 vs RoBERTa 0.774, Hindi Track A: JNLP 0.926 vs RoBERTa 0.855, Oromo Track A: Tewodros 0.616 vs RoBERTa 0.126, Algerian Arabic Track B: PAI 0.650 vs RoBERTa 0.016, English Track C: Deepwave 0.797 vs RoBERTa 0.375, and Yoruba Track C: maomao 0.360 vs RoBERTa 0.053 (Muhammad et al., 10 Mar 2025).

Low-resource and cross-lingual settings remain substantially harder. The organizer paper explicitly states that Track C scores are generally lower than Track A, especially for low-resource languages, and system papers reinforce this conclusion. UoB-NLP emphasizes that adapter-based transfer can outperform larger LLMs in several low-resource languages, but also notes dev–test mismatch and resource limitations in language-adapter training (Leon et al., 11 Apr 2025). PromotionGo shows that sparse lexical features may still outperform richer contextual representations in some languages, which suggests that multilingual semantic transfer is neither uniform nor architecture-agnostic (Huang et al., 11 Jul 2025). Chinchunmei’s finding that multilingual training can reduce English performance further indicates that multilingual coverage is not equivalent to cross-cultural emotional alignment (Li et al., 21 Jul 2025).

Another notable pattern is the relatively limited role of external labeled resources. The organizer paper reports that 92.1% of combined systems used no additional data, implying that pretrained models, prompting, retrieval, and parameter-efficient tuning were often sufficient to build competitive entries (Muhammad et al., 10 Mar 2025). This suggests a benchmark ecology in which adaptation strategy, label formulation, and aggregation often matter more than large task-specific auxiliary corpora.

The task’s broader significance lies in its combination of scale, language diversity, and explicit attention to perceived emotion rather than sentiment polarity alone. It institutionalizes the claim that multilingual LLMs are not automatically multicultural, and it provides a standardized setting in which the interaction among language resource level, cultural specificity, model family, and task formulation can be studied directly. At the same time, the documentation reveals some unresolved issues, including internal inconsistencies in one Track B language listing and uneven difficulty across languages. A plausible implication is that SemEval-2025 Task 11 functions simultaneously as a leaderboard benchmark and as a diagnostic resource for multilingual affective computing, especially in low-resource and cross-lingual regimes (Muhammad et al., 10 Mar 2025).