SemEval-2025 Task 11: Emotion Detection

• SemEval-2025 Task 11 is a shared challenge that defines emotion detection as a multilingual, culturally mediated, and inherently multi-label problem across 32 languages.
• The task uniquely integrates intensity prediction on a four-point scale and cross-lingual transfer strategies to overcome gaps in traditional emotion analysis.
• Top submissions leveraged large language models and multilingual transformers, achieving significant improvements over baseline models, particularly in low-resource settings.

Searching arXiv for the official task paper and representative SemEval-2025 Task 11 system papers to ground the encyclopedia entry. arXiv search results for "SemEval-2025 Task 11 Bridging the Gap in Text-Based Emotion Detection":

1. (Muhammad et al., 10 Mar 2025) — "SemEval-2025 Task 11: Bridging the Gap in Text-Based Emotion Detection"
2. (Leon et al., 11 Apr 2025) — "UoB-NLP at SemEval-2025 Task 11: Leveraging Adapters for Multilingual and Cross-Lingual Emotion Detection"
3. (Sahil et al., 27 Feb 2025) — "Team A at SemEval-2025 Task 11: Breaking Language Barriers in Emotion Detection with Multilingual Models"
4. (Ranjbar et al., 27 Feb 2025) — "Lotus at SemEval-2025 Task 11: RoBERTa with Llama-3 Generated Explanations for Multi-Label Emotion Classification"
5. (Xue et al., 19 May 2025) — "JNLP at SemEval-2025 Task 11: Cross-Lingual Multi-Label Emotion Detection Using Generative Models"
6. (Morozov et al., 4 Jun 2025) — "Empaths at SemEval-2025 Task 11: Retrieval-Augmented Approach to Perceived Emotions Prediction"
7. (Sani et al., 19 Jun 2025) — "HausaNLP at SemEval-2025 Task 11: Hausa Text Emotion Detection"
8. (Hanif et al., 22 May 2025) — "University of Indonesia at SemEval-2025 Task 11: Evaluating State-of-the-Art Encoders for Multi-Label Emotion Detection"
9. (Li et al., 21 Jul 2025) — "Chinchunmei at SemEval-2025 Task 11: Boosting the LLM's Capability of Emotion Perception using Contrastive Learning" 10. (Chen et al., 2 Aug 2025) — "CSIRO-LT at SemEval-2025 Task 11: Adapting LLMs for Emotion Recognition for Multiple Languages" SemEval-2025 Task 11, “Bridging the Gap in Text-Based Emotion Detection,” was a shared task on multilingual text-based emotion recognition that covered more than 30 languages from seven distinct language families and was explicitly designed around predominantly low-resource languages (Muhammad et al., 10 Mar 2025). Its central problem formulation was perceived emotion recognition: given a sentence or short text snippet, systems were asked to predict the emotion that people believe the speaker might have felt, rather than the speaker’s objectively “true” internal emotional state (Muhammad et al., 10 Mar 2025). The task combined multilingual coverage, multi-label emotion prediction, emotion intensity prediction, and cross-lingual transfer, and it rapidly became one of the largest SemEval efforts in affective NLP, with over 700 participants, 220 final submissions, and 93 system description papers (Muhammad et al., 10 Mar 2025).

1. Problem definition and conceptual scope

The task was motivated by a set of long-standing gaps in emotion analysis. Existing multilingual emotion datasets often rely on translation, which can miss culture-specific emotional expression, and many prior resources are single-label even though real emotional expression is often overlapping and mixed (Muhammad et al., 10 Mar 2025). SemEval-2025 Task 11 therefore framed emotion detection as a multilingual, culturally mediated, and inherently multi-label problem, rather than a monolingual or single-label classification exercise (Muhammad et al., 10 Mar 2025).

The official task description states that the benchmark covered 32 languages in total, drawn from seven language families and distributed across Africa, Asia, Latin America, North America, and Europe (Muhammad et al., 10 Mar 2025). The task tracks did not use exactly the same language inventory: Track A and Track C operated on large multilingual multi-label datasets, while Track B used the subset with sufficiently robust intensity annotations (Muhammad et al., 10 Mar 2025). The benchmark’s emphasis on low-resource settings was not incidental; it was part of the task’s stated aim to move the field beyond its concentration on high-resource languages such as English, Spanish, and Arabic (Muhammad et al., 10 Mar 2025).

A recurrent source of confusion was the distinction between perceived emotion and internal emotion. The task did not ask whether the author “really” felt a given affective state, nor did it target the reader’s own emotional reaction. It targeted the emotional state that third-party observers would attribute to the speaker from the text alone (Muhammad et al., 10 Mar 2025). This framing made cultural and annotator interpretation central to the benchmark’s scientific meaning.

2. Dataset construction and annotation design

The data were assembled from a wide range of sources selected according to the availability of emotionally rich text and access to fluent annotators. The official description lists social media posts from Reddit, YouTube, Twitter, and Weibo; personal narratives, talks, and speeches; literary text; news data; and both human-written and machine-generated material for some languages (Muhammad et al., 10 Mar 2025). A notable example was the use of the French novel La Grande Maison, translated into Algerian Arabic and then manually post-edited before annotation (Muhammad et al., 10 Mar 2025). For Hindi and Marathi, annotators created emotive sentences from prompts, some Hindi data were translated into Marathi and manually corrected, and a few hundred ChatGPT-generated examples were added after quality control (Muhammad et al., 10 Mar 2025).

Overall, the task dataset collection contained over 100,000 multi-labeled instances (Muhammad et al., 10 Mar 2025). The annotation schema allowed annotators to select all emotions that applied. The annotation description lists anger, sadness, fear, disgust, joy, surprise, and neutral if no emotion is present (Muhammad et al., 10 Mar 2025). At the track-definition level, however, the official output space for Tracks A and B is described in terms of six emotions—joy, sadness, fear, anger, surprise, and disgust—so neutral functioned as an annotation-state notion rather than the standard evaluation target (Muhammad et al., 10 Mar 2025). This distinction later mattered for how participating systems verbalized labels and thresholds.

Intensity annotation was performed on a four-point ordinal scale: $0 =$ no emotion, $1 =$ low intensity, $2 =$ moderate intensity, and $3 =$ high intensity (Muhammad et al., 10 Mar 2025). The task paper gives an explicit presence rule: an emotion was considered present if at least two annotators selected that label with intensity $1$, $2$, or $3$, and the average score exceeded a predefined threshold $T$, where $T = 0.5$ (Muhammad et al., 10 Mar 2025). For Track B, final intensity labels were produced by averaging the selected intensity values and rounding up to the nearest whole number (Muhammad et al., 10 Mar 2025).

Reliability was reported using Split-Half Class Match Percentage (SHCMP), with overall reliability ranging from 60% to more than 90% (Muhammad et al., 10 Mar 2025). The official paper interprets these values as evidence of high-quality datasets, but they also underscore that the task concerns a subjective and culturally variable target rather than a purely objective label.

3. Track structure, outputs, and evaluation

The shared task consisted of three tracks, each operationalizing a different aspect of multilingual affect modeling (Muhammad et al., 10 Mar 2025).

Track	Prediction target	Official metric
A	Binary vector over joy, sadness, fear, anger, surprise, and disgust	Average macro F-score
B	Six-dimensional intensity vector with values in $\{0,1,2,3\}$	Pearson correlation coefficient
C	Binary vector over the same six emotions without target-language training data	Average macro F-score

Track A was the core monolingual multi-label detection setting. Track B extended the problem from presence/absence to graded intensity. Track C mirrored Track A’s output format but prohibited the use of target-language training data, making it a cross-lingual transfer benchmark rather than a standard supervised task (Muhammad et al., 10 Mar 2025).

The organizers supplied two main baseline families: a majority-class baseline and a RoBERTa fine-tuning baseline, with a family-based transfer variant for Track C (Muhammad et al., 10 Mar 2025). These baselines exposed how strongly difficulty varied by language. In Track A, the RoBERTa baseline ranged from 0.855 for Hindi to 0.126 for Oromo; in Track B it ranged from 0.877 Pearson for Russian to 0.016 for Algerian Arabic; and in Track C it dropped as low as 0.010 for Nigerian Pidgin and 0.052 for Emakhuwa (Muhammad et al., 10 Mar 2025). The Track C baseline design itself encoded assumptions about genealogical transfer: for example, the organizers trained on Slavic languages and tested on Tatar, trained on Niger-Congo languages and tested on Nigerian Pidgin, and trained on Russian and tested on Chinese (Muhammad et al., 10 Mar 2025).

These evaluation choices made the benchmark scientifically informative in two ways. First, they separated monolingual supervised performance from zero-shot or transfer-only performance. Second, they made it possible to compare systems not just by absolute score, but by the extent to which they overcame severe low-resource and cross-lingual degradation.

4. Participation scale and official outcomes

SemEval-2025 Task 11 was, according to the organizers, the most popular competition on Codabench in 2024 (Muhammad et al., 10 Mar 2025). The official report gives participation figures of over 700 participants, 220 final submissions, and 93 system description papers (Muhammad et al., 10 Mar 2025). This scale mattered because it produced a broad methodological ecosystem rather than a narrow comparison among a handful of architectures.

Track A was the largest and most competitive track. The strongest overall system was Team PAI, which ranked first in 20 out of 28 languages (Muhammad et al., 10 Mar 2025). Language-wise results illustrate both competitiveness and dispersion: PAI scored 0.823 in English and 0.849 in Spanish, JNLP reached 0.926 in Hindi, Heimerdinger reached 0.901 in Russian, and Team Tewodros scored 0.616 in Oromo, far above the RoBERTa baseline of 0.126 (Muhammad et al., 10 Mar 2025). The official paper also highlights unusually large gains in other low-resource settings, such as Yoruba, where PAI scored 0.461 versus a baseline of 0.092 (Muhammad et al., 10 Mar 2025).

Track B was dominated by PAI in almost all languages, with the main exception of Amharic, where CSECU-Learners took first place at 0.856 Pearson (Muhammad et al., 10 Mar 2025). PAI reached 0.925 in Russian and 0.840 in English, while Algerian Arabic remained comparatively difficult, with the best system scoring only 0.650 (Muhammad et al., 10 Mar 2025). The official analysis treats this as evidence that intensity prediction remained harder and more language-sensitive than its strongest scores alone might suggest.

Track C was the most distinctive and generally the hardest track. The strongest systems were Deepwave and maomao (Muhammad et al., 10 Mar 2025). Deepwave won a large number of languages, including 0.797 in English, 0.831 in Spanish, 0.919 in Hindi, and 0.906 in Russian, while maomao led in languages such as Afrikaans, Indonesian, Somali, Swahili, isiXhosa, Yoruba, and isiZulu (Muhammad et al., 10 Mar 2025). The drop from monolingual Track A to cross-lingual Track C was substantial for many languages, especially lower-resource targets, confirming that transfer without target-language supervision remained a major open challenge.

5. Dominant methodological patterns

The official task overview concludes that top systems overwhelmingly used LLMs and multilingual pretrained transformers, with strong uptake of XLM-RoBERTa, mBERT, DeBERTa, and IndicBERT on the encoder side and Gemma-2, Mistral, Phi-4, Qwen-2.5, DeepSeek, LLaMA-3, GPT, and Gemini on the LLM side (Muhammad et al., 10 Mar 2025). Fine-tuning was the most common training strategy, but it was accompanied by prompt engineering, LoRA and AdaLoRA, ensembling, synthetic data generation, retrieval-augmented generation, adapter tuning, threshold tuning, contrastive learning, and chain-of-thought prompting (Muhammad et al., 10 Mar 2025). Only a minority of teams used additional external data (Muhammad et al., 10 Mar 2025).

A representative encoder-based submission was Team A’s “TransferModel_FC_EmbeddingE5,” which combined multilingual-e5-large-instruct with a fully connected classifier, sigmoid outputs, Binary Cross Entropy with label smoothing $1 =$0, and evaluation across 13 languages (Sahil et al., 27 Feb 2025). Its reported macro F1 reached 0.8901 in Hindi and 0.8831 in Russian, while remaining notably weaker in Arabic, Chinese, Oromo, and Ukrainian (Sahil et al., 27 Feb 2025). This line of work exemplified the strong performance of multilingual representation learning with relatively simple task heads.

Parameter-efficient transfer was also prominent. UoB-NLP compared task-only adapters, target-language-ready task adapters, and language-family-based adapters on xlm-roberta-base and afro-xlmr-base, with adapter bottleneck size 16 and frozen base models (Leon et al., 11 Apr 2025). Their best-performing recipe used XLM-RoBERTa-base for non-African languages and Afro-XLMR-base for African languages, with target-language-ready task adapters performing best overall, especially for low-resource African languages (Leon et al., 11 Apr 2025). The system ranked 7th for Kinyarwanda and 8th for Tigrinya in Track A, and 3rd for Amharic plus 4th for Oromo, Tigrinya, Kinyarwanda, Hausa, and Igbo in Track C; the paper further states that the adapter approach outperformed LLMs in 11 languages and matched them in four others (Leon et al., 11 Apr 2025).

Several teams recast the task as structured generation. JNLP compared XLM-RoBERTa-Large with instruction-tuned Qwen and Llama models, and formalized both a base method and a pairwise method, including $1 =$1 for Track A and $1 =$2 for Track B (Xue et al., 19 May 2025). Their central empirical result was that mixed multilingual training generally outperformed separated per-language training, while pairwise prompting was especially effective for Track B and for languages with richer multi-emotion co-occurrence; the system ranked 1st in Hindi in Track A and achieved Top 5 performance in seven Track B languages (Xue et al., 19 May 2025). Chinchunmei likewise unified both tracks as instruction-following structured generation using LLaMA3-Instruct-8B and contrasted standard supervised fine-tuning with Contrastive Reasoning Calibration, DPO, and SimPO (Li et al., 21 Jul 2025). In English, the paper reports that plain supervised fine-tuning was strongest for Track A, while DPO was best for Track B; the official English ranks were 9th in Track A and 6th in Track B (Li et al., 21 Jul 2025). CSIRO-LT, using aya and Llama variants with LoRA, reached a closely related conclusion from a different experimental angle: the most effective method among those they tested was fine-tuning a pre-trained multilingual LLM with LoRA separately for each language (Chen et al., 2 Aug 2025).

Other strong submissions treated the benchmark as a testbed for auxiliary context or non-parametric memory. Lotus used Llama-3-generated explanations as additional context for English RoBERTa classification; by concatenating the original text and a generated explanation, the system improved macro F1 from $1 =$3 to $1 =$4 relative to text-only RoBERTa (Ranjbar et al., 27 Feb 2025). Empaths proposed “EmoRAG,” a training-free retrieval-augmented system that indexed the labeled training set, retrieved examples with either LangChain n-gram retrieval or BGE-M3, queried four instruction-tuned LLMs, and aggregated outputs with label-specific F1 weighting; its reported average test performance was 0.638 F1-micro and 0.590 F1-macro across languages (Morozov et al., 4 Jun 2025). Together, these systems show that the task did not converge on a single dominant paradigm: full supervised fine-tuning, PEFT, prompting, retrieval, and contrastive optimization all remained competitive under different language and track conditions.

6. Scientific significance, interpretive tensions, and limitations

The task’s main scientific significance lay in its treatment of emotion detection as a multilingual, low-resource, and culturally variable inference problem rather than a narrow English classification benchmark (Muhammad et al., 10 Mar 2025). It made multi-label prediction standard rather than exceptional, integrated intensity prediction into the shared-task structure, and institutionalized cross-lingual transfer without target-language supervision as a first-class evaluation setting (Muhammad et al., 10 Mar 2025). The official paper explicitly identifies future directions such as better modeling of label correlations, stronger low-resource transfer, more principled intensity regression or ranking methods, and methods that explicitly account for cultural variability in emotion expression (Muhammad et al., 10 Mar 2025).

At the same time, the benchmark exposed several interpretive and engineering tensions. The official annotation description lists neutral alongside the six non-neutral emotions, whereas the track definitions focus on six output emotions (Muhammad et al., 10 Mar 2025). Some system papers also reported language-specific label-space differences, such as English and Afrikaans having only five emotions in their implementations, or prompt templates in which English excluded disgust and Afrikaans excluded surprise (Leon et al., 11 Apr 2025, Chen et al., 2 Aug 2025). This suggests that, in practice, participants sometimes had to reconcile task-level definitions with language- or file-level specifics.

System descriptions also documented practical limits of current evaluation practice. Chinchunmei argued that the English development set was too small and imbalanced to support stable model selection, noting that the English dev set contained only 116 samples and that multilingual training and CRC looked stronger on dev than on test (Li et al., 21 Jul 2025). HausaNLP, by contrast, revealed a different kind of limitation: although the shared task was multi-label, their implemented Hausa system collapsed the multi-hot label vector to a single dominant emotion and then reconverted the prediction into one-hot form for submission (Sani et al., 19 Jun 2025). This does not invalidate the task formulation, but it shows how low-resource and engineering constraints could encourage simplifications that depart from strict multi-label modeling.

The benchmark also carried explicit usage caveats. The official paper states that the datasets are not suitable for critical decision-making without expert supervision, are susceptible to domain shift, and may produce unreliable instance-level predictions; it further notes restrictions against commercial use and against use by state actors in high-risk applications unless explicitly approved by dataset creators (Muhammad et al., 10 Mar 2025). These cautions are consistent with the task’s core premise: perceived emotion is meaningful and measurable, but it is not a transparent proxy for internal mental state, and it remains shaped by annotation design, language coverage, cultural interpretation, and model bias.