---
title: 'SemEval 2025 Task 11: Emotion Detection Benchmark'
url: https://www.emergentmind.com/topics/semeval-2025-task-11
type: topic
---

# SemEval 2025 Task 11: Emotion Detection Benchmark

Searching arXiv for the official task paper and closely related SemEval-2025 Task 11 system papers to ground the article in current sources.
arXiv search query: SemEval 2025 Task 11 Bridging the Gap in Text-Based Emotion Detection
SemEval-2025 Task 11, titled **“Bridging the Gap in Text-Based Emotion Detection,”** is a large-scale shared task on **perceived text-based emotion detection** in multilingual and low-resource settings. Its central objective is to predict the emotions that general third-party observers would attribute to a speaker from a sentence or short text snippet, rather than the speaker’s internal state [2503.07269]. The task introduced a new multilingual benchmark spanning **32 languages** from **seven language families**, with a strong focus on underserved languages and locally grounded annotation practices, and organized evaluation into **three tracks**: **Track A: Multi-label Emotion Detection**, **Track B: Emotion Intensity Detection**, and **Track C: Cross-lingual Emotion Detection** [2503.07269]. Across participating systems, the task also became a testbed for contrasting encoder fine-tuning, parameter-efficient adaptation, instruction-tuned generation, retrieval-augmented inference, explanation augmentation, and adapter-based transfer [2504.08543].

## 1. Task definition and formal structure

SemEval-2025 Task 11 defines emotion recognition as a **multi-label** prediction problem over a fixed annotation inventory of **seven emotion categories**: **joy, sadness, anger, fear, surprise, disgust, and neutral** [2503.07269]. In the official predictive setup, however, the main tasks target the **six non-neutral emotions**—**joy, sadness, fear, anger, surprise, and disgust**—while **neutral** is used in annotation for cases where no emotion is present [2503.07269]. The task is explicitly multi-label because annotators were instructed to **“select all emotions that apply,”** and a single text may express more than one emotion simultaneously [2503.07269].

The shared task was divided into three tracks. **Track A** is the monolingual multi-label classification task: the input is a text snippet in a given language, and the output is a binary decision for each of the six emotions [2503.07269]. **Track B** is emotion intensity detection: systems predict one of **0, 1, 2, 3** for each emotion, corresponding to **no emotion**, **low intensity**, **moderate / medium intensity**, and **high intensity** [2503.07269]. **Track C** uses the same prediction target as Track A but imposes a cross-lingual transfer condition: systems must predict labels for a target language **without using any training data in that target language** [2503.07269].

The official metrics reflect these task definitions. For **Tracks A and C**, systems are ranked by **average macro F-score** [2503.07269]. For **Track B**, the official metric is the **Pearson correlation coefficient** between predicted and gold labels [2503.07269]. This metric design led many participating systems to optimize directly for macro-F1 in Track A, especially because the label distributions are imbalanced, and to treat Track B as a structured ordinal or categorical prediction problem rather than free-form generation [2502.19856].

A recurring point across system papers is that the task’s “multi-label” framing matters operationally. Some submissions implemented genuine independent label prediction, for example with sigmoid-based outputs or per-label prompting [2502.19856; 2505.13244]. Others nominally addressed Track A but reduced the problem to single-label or dominant-emotion classification, which created a mismatch between the official formulation and the implemented model [2506.16388]. This divergence became one of the clearest methodological fault lines in the 2025 ecosystem.

## 2. Dataset design, annotation, and multilingual scope

The shared task dataset is one of its principal contributions. The organizers report that the benchmark covers **32 languages** across **Africa, Asia, Latin America, North America, and Europe**, with strong emphasis on **predominantly low-resource** languages [2503.07269]. The language inventory includes **Afrikaans, Algerian Arabic, Amharic, Chinese, Emakhuwa, English, German, Hausa, Hindi, Igbo, Indonesian, isiXhosa, isiZulu, Javanese, Kinyarwanda, Marathi, Moroccan Arabic, Nigerian Pidgin, Oromo, Brazilian Portuguese, Mozambican Portuguese, Romanian, Russian, Somali, Latin American Spanish, Sundanese, Swahili, Swedish, Tatar, Tigrinya, Ukrainian, and Yoruba** [2503.07269].

Not all languages appear in every track. **Track A** covered **28 languages** with training data, **Track B** covered **11 languages**, and **Track C** additionally included languages with no training split, used only as cross-lingual targets [2503.07269]. The collection contains **over 100,000 multi-labeled instances** overall [2503.07269]. One system paper referring to the underlying BRIGHTER resource describes it as a **manually curated multilingual benchmark with nearly 100,000 text instances from 28 languages and multiple domains including social media, personal narratives, speeches, literary texts, and news** [2502.19856]. This narrower count reflects the BRIGHTER core, whereas the task paper situates the shared task at the broader 32-language level [2503.07269].

The annotation protocol is unusually explicit about perceived emotion and intensity. Annotators first selected applicable emotions, then assigned intensity values. A given emotion was marked as present if **at least two annotators** selected that label with intensity **1, 2, or 3**, and the **average score** for that emotion exceeded the threshold **\(T = 0.5\)** [2503.07269]. For Track B, the final gold intensity score was obtained by **averaging the selected intensity values and rounding up to the nearest whole number** [2503.07269]. The organizers report annotator reliability using **Split-Half Class Match Percentage (SHCMP)**, with values ranging roughly from **60% to over 90%** [2503.07269].

Data collection was multilingual not only in coverage but also in source construction. The organizers used **social media posts**, **personal narratives, talks, and speeches**, **literary texts**, **news data and headlines**, and, for some languages, **human-written and machine-generated text** [2503.07269]. All instances were **manually annotated by fluent speakers**, with collaboration from local communities and compensation above minimum hourly wage [2503.07269]. This suggests that the benchmark was designed not merely as a translated multilingual benchmark, but as a culturally situated resource for perceived emotion detection.

A number of system papers supplement this dataset picture with language-specific details. Team A reports that BRIGHTER annotations include one or more labels from **anger, sadness, fear, disgust, joy, surprise, plus neutral**, along with intensity values on a **4-point scale from 0 to 3**, and notes complementary use of **EthioEmo** for **Amharic** and **Afan Oromo** [2502.19856]. UoB-NLP likewise describes its experiments as using **BRIGHTER** and **EthioEmo**, especially for low-resource African languages [2504.08543]. These descriptions are consistent with the official task framing, though exact per-language counts vary by paper because some focus on subsets or particular tracks.

## 3. Baselines, participation, and overall competitive landscape

SemEval-2025 Task 11 was one of the largest shared tasks of its cycle. The organizers report **over 700 participants**, **220 final submissions**, and **93 system description papers** [2503.07269]. They also state that the task was **the most popular competition hosted on Codabench in 2024** [2503.07269]. The reporting mixes **teams** and **submissions**, but the scale of participation is unambiguous [2503.07269].

The official baselines were intentionally simple. Across all tracks, the organizers used a **majority-class baseline** for each language [2503.07269]. For **Tracks A and B**, they additionally fine-tuned **RoBERTa** on the training data for each language [2503.07269]. For **Track C**, the RoBERTa baseline was cross-lingual and trained on all languages in the same language family, excluding the target language [2503.07269]. These baselines proved informative but left considerable headroom, especially in low-resource and zero-target-supervision settings.

Selected results from the task paper illustrate the spread. In **Track A**, **English** reached **0.823** for the top system versus **0.708** for the RoBERTa baseline and **0.367** for majority; **Hindi** reached **0.926** versus **0.855** and **0.246**; **Russian** reached **0.901** versus **0.838** and **0.262** [2503.07269]. In **Track B**, **Russian** reached **0.925** Pearson versus **0.877** for RoBERTa, **English** reached **0.840** versus **0.641**, and **Algerian Arabic** reached **0.650** versus **0.016** [2503.07269]. In **Track C**, performance dropped overall, with **English** at **0.797** versus **0.375**, **Hindi** at **0.919** versus **0.138**, and **Emakhuwa** only **0.210** versus **0.052** [2503.07269].

The organizers identify several dominant teams and patterns. In **Track A**, **Team PAI** ranked first in **20 out of 28 languages** using an **LLM ensemble** combining **ChatGPT-4o, DeepSeek-V3, Gemma-9B, Qwen-2.5-32B, and Mistral-Small-24B**, with ensembling methods including **neural networks, XGBoost, LightGBM, linear regression, and weighted voting** [2503.07269]. In **Track B**, **PAI** again led in nearly every language except **Amharic**, where **CSECU-Learners** ranked first with **0.856** Pearson using **language-specific transformers** and **multi-sample dropout** [2503.07269]. In **Track C**, **Deepwave** led overall with **Gemma-2**, **data augmentation**, **chain-of-thought prompting**, **5-fold cross-validation**, and **model merging** [2503.07269].

A broad trend emerges from the official analysis: the strongest systems largely relied on either **fine-tuned transformer encoders** such as **XLM-R, mBERT, DeBERTa**, or **instruction-tuned / prompted LLMs**, often combined with **LoRA / PEFT**, prompt engineering, and ensembling [2503.07269]. At the same time, the task landscape remained methodologically heterogeneous, with strong submissions also coming from adapter-based models, retrieval-augmented methods, and feature-centric pipelines [2504.08543; 2506.04409; 2507.08499].

## 4. Dominant modeling paradigms

The 2025 task ecosystem was shaped by several distinct modeling paradigms rather than a single consensus architecture.

A first paradigm was **multilingual encoder fine-tuning**. Team A’s **TransferModel\_FC\_EmbeddingE5** used **`multilingual-e5-large-instruct`** with **[CLS]** pooling, **dropout \(0.3\)**, a **1024-to-6 fully connected layer**, **sigmoid outputs**, **BCE with label smoothing \(a = 0.1\)**, **AdamW** with learning rate **\(1\times 10^{-5}\)**, **batch size 16**, **gradient clipping at 1.0**, **early stopping on dev macro F1 with patience 4**, and threshold **0.5** at inference [2502.19856]. Their conclusion was explicit: the best-performing approach was **multilingual E5 embeddings plus a fully connected layer** [2502.19856]. The University of Indonesia system, by contrast, reports that **training a classifier on top of prompt-based encoders such as mE5 and BGE yields significantly better results than fully fine-tuning XLMR and mBERT**, and that its best leaderboard model was an ensemble of **multiple BGE models** with **CatBoost** as classifier, achieving **56.58 average F1-macro across all languages** [2505.16460]. This juxtaposition suggests a broader 2025 pattern: sentence-embedding-style encoders plus lightweight classifiers were often competitive with, and in some cases superior to, full end-to-end multilingual transformer fine-tuning.

A second paradigm was **instruction-tuned and generative LLM modeling**. JNLP framed Track A and Track B either as direct generation of emotion labels or as **pairwise** text-label prediction, using **XLM-RoBERTa-Large**, **LLaMA 2**, and **Qwen2.5**, and reported **Top 4 performance across 10 languages** in Track A with **1st place in Hindi**, as well as **Top 5 performance in 7 languages** in Track B [2505.13244]. Chinchunmei unified Track A and Track B within a single **Llama3-Instruct-8B** framework using prompt-based structured generation, then compared **Standard Prediction**, **Contrastive Reasoning Calibration (CRC)**, **DPO**, and **SimPO** [2507.15714]. Their main conclusion was that a strong generative SFT baseline was hard to beat on Track A, whereas **DPO-style generation-level contrastive learning** was the most promising extension for Track B [2507.15714]. CSIRO-LT similarly concluded that the most effective practical strategy was to **fine-tune a pre-trained multilingual LLM with LoRA setting separately for each language**, outperforming prompting-heavy alternatives in their development comparisons [2508.01161].

A third paradigm was **adapter-based parameter-efficient transfer**. UoB-NLP compared **task-only adapters**, **target-language-ready task adapters (TLR)**, and **language-family-based adapters** on top of **XLM-RoBERTa-base** and **Afro-XLMR-base** [2504.08543]. Their central finding was that **target-language-ready task adapters achieve the best overall performance**, particularly for low-resource African languages, with **7th for Tigrinya** and **8th for Kinyarwanda** in Track A, and **3rd for Amharic** plus **4th** in several other Track C languages [2504.08543]. They also state that their adapter-based models **outperform large language models in 11 languages and match their performance in four others**, despite using far fewer parameters [2504.08543].

A fourth paradigm was **retrieval-augmented and training-free inference**. **EmoRAG** built a database of labeled training examples, retrieved the top-\(K\) most similar examples, and used them as few-shot context for **Llama-3.1-70B-Instruct**, **Qwen2.5-72B-Instruct**, **gpt-4o-mini-2024-07-18**, and **gemma-2-27b-it**, followed by several aggregation rules [2506.04409]. The strongest strategy was **`majority_vote_by_label_f1`**, a label-specific weighted aggregation based on development-set performance [2506.04409]. The system required **no additional model training** and achieved **0.638 average test F1-micro** and **0.590 average test F1-macro** [2506.04409].

A fifth paradigm was **feature-centric shallow or hybrid modeling**. PromotionGo compared **TF-IDF**, **FastText**, and **Sentence-BERT** representations, optionally with **PCA**, and downstream **DT**, **Voting**, and **MLP** classifiers across **28 languages** [2507.08499]. Their findings emphasize that **TF-IDF remains highly effective for low-resource languages**, while **SBERT generally outperforms TF-IDF and FastText in 18 out of 28 languages**, especially with **MLP**, and **PCA** can reduce training time selectively without universally improving accuracy [2507.08499].

These paradigms did not simply coexist; they frequently contradicted one another’s inductive biases. Some system papers argue that strong multilingual encoders already contain enough cross-lingual affective structure that a lightweight head suffices [2502.19856]. Others argue that language-specific or African-specialized pretraining remains essential [2504.08543; 2506.16388]. Still others show that explanation generation, retrieval, or pairwise prompting can compensate for sparse or ambiguous surface cues [2502.19935; 2506.04409; 2505.13244].

## 5. Recurrent empirical findings across systems

Several empirical regularities recur across the literature.

A first recurrent finding is that **multilingual pretrained representations are strong but unevenly reliable across languages**. Team A achieved highest macro F1 in **Hindi 0.8901**, **Russian 0.8831**, and **Marathi 0.8657**, but weaker results in **Arabic 0.5550** and **Oromo 0.5628** [2502.19856]. Their interpretation is that multilingual E5 captures shared semantic and affective structure across languages, but the weaker scores in **Arabic, Oromo, Chinese, and Ukrainian** indicate that multilingual alignment remains imperfect [2502.19856]. The task paper echoes this at scale, noting that higher-resource languages such as **English** and **Russian** generally achieved better results across tracks, whereas lower-resource languages and cross-lingual settings remained difficult [2503.07269].

A second recurrent finding is that **simple or parameter-efficient methods can remain competitive**. UoB-NLP shows that adapter-based models can outperform or match much larger LLMs in many languages [2504.08543]. PromotionGo shows that **TF-IDF** can be stronger than dense representations in languages such as **Marathi, Spanish, Russian, Oromo, Somali, Hausa, and Tatar**, especially under resource constraints [2507.08499]. EmoRAG demonstrates that a retrieval-augmented ensemble can be competitive **without additional model training** [2506.04409]. These results complicate any simple narrative that the task was won exclusively by larger end-to-end generative models.

A third recurrent finding is that **label imbalance remains central**. Team A preferred **macro F1** because it penalizes bias toward frequent labels [2502.19856]. Lotus explicitly notes that it did not use resampling, class weights, focal loss, or threshold tuning despite training on a class-imbalanced English subset where **fear** is by far the most frequent label and **anger** much rarer [2502.19935]. PromotionGo’s Hindi analysis reports that over **78%** of the **2,556** samples are labeled as **not anger**, producing high true-negative rates but lower recall on minority positives [2507.08499]. Chinchunmei’s Track B analysis similarly indicates that rare emotion-intensity combinations are hard to model [2507.15714]. This suggests that data imbalance was not merely a statistical inconvenience but a major determinant of architecture and metric choice.

A fourth recurrent finding is that **multi-label structure helps some methods but is not always handled faithfully**. Genuine multi-label systems include Team A’s sigmoid-based E5 model [2502.19856], Lotus’s five-label RoBERTa setup with explanation augmentation [2502.19935], JNLP’s base and pairwise formulations [2505.13244], and CSIRO-LT’s six independent binary or four-way per-emotion formulations [2508.01161]. In contrast, HausaNLP explicitly collapses the one-hot emotion vector to a single **dominant emotion**, converting Track A into a six-class classification task despite the shared task’s multi-label framing [2506.16388]. The paper itself acknowledges that this makes the system “effectively multi-class rather than multi-label” [2506.16388]. This is an important caution for interpreting SemEval-2025 Task 11 system descriptions: nominal participation in Track A does not always imply faithful multi-label modeling.

A fifth recurrent finding is that **language mixing is helpful in some settings but harmful in others**. JNLP reports that **mixed-language training generally outperforms separate per-language training** in both Track A and Track B [2505.13244]. Chinchunmei reports the opposite for English-focused experiments: **adding multilingual training data hurt English performance** in both tracks, which they attribute to **cross-cultural differences in emotion perception** [2507.15714]. This suggests that multilingual pooling is not uniformly beneficial; its effect depends on language composition, model family, and the extent to which emotion annotation conventions are aligned across languages.

## 6. Methodological debates, limitations, and significance

SemEval-2025 Task 11 exposed several methodological debates that are likely to remain relevant beyond the 2025 edition.

One debate concerns **whether multilingual generalization is best achieved through shared representation or language-specific specialization**. Encoder-centric systems such as Team A emphasize cross-lingual semantic alignment learned during pretraining [2502.19856]. Adapter-based systems argue for modular specialization, especially for low-resource African languages [2504.08543]. CSIRO-LT goes further, concluding that the best approach is **separate per-language LoRA fine-tuning**, not a single jointly trained multilingual model [2508.01161]. The coexistence of these results suggests that multilinguality in perceived emotion detection is not a monolithic transfer problem; it is shaped by pretraining coverage, typological distance, annotation design, and culture-specific expression.

A second debate concerns **whether generation improves classification**. Lotus argues that Llama-3-generated explanations improve multi-label emotion detection by making ambiguous or under-explained text more interpretable, yielding gains in both macro and micro F1 over text-only RoBERTa [2502.19935]. Chinchunmei finds that sample-based contrastive reasoning calibration offers limited benefit, whereas DPO helps Track B more than Track A [2507.15714]. JNLP shows that pairwise prompting can outperform direct generation when texts express multiple emotions, but the base method can be stronger when label cardinality is low [2505.13244]. These findings suggest that generative augmentation is most useful when it induces a more appropriate decision decomposition, not simply because natural-language output is intrinsically superior.

A third debate concerns **resource use versus reproducibility**. Retrieval-augmented systems avoid retraining but can depend on multiple large proprietary or closed models [2506.04409]. Many LLM-based system papers omit crucial hyperparameters such as decoding settings, calibration procedures, or preference-learning coefficients [2502.19935; 2507.15714; 2508.01161]. Feature-centric and adapter-based systems are often more transparent about training configurations, but may omit exact multi-label decision mechanics or hyperparameter grids [2507.08499; 2504.08543]. This suggests that the shared task advanced empirical performance more quickly than it resolved reproducibility standards.

A fourth debate concerns **the limits of benchmark fidelity**. Several papers identify mismatches between official task framing and actual system design. HausaNLP reduces Track A to dominant-emotion classification [2506.16388]. One paper in the supplied corpus, attributed to the University of Indonesia, cannot be summarized from its provided “paper” text because the available document is an ACL template rather than the actual system description, despite its metadata claiming a SemEval submission [2505.16460]. A plausible implication is that shared-task ecosystems remain partly dependent on inconsistent archival quality, especially when preprints, templates, and workshop proceedings circulate asynchronously.

Despite these tensions, the significance of SemEval-2025 Task 11 is clear. It established a multilingual benchmark for **perceived** emotion detection over a large and geographically broad language inventory, introduced formal evaluation for both **multi-label emotion detection** and **emotion intensity**, and catalyzed a wide comparative space of methods ranging from **XLM-R** and **mE5** encoders to **LoRA-adapted LLMs**, **retrieval-augmented ensembles**, **contrastive generation**, and **adapter-based transfer** [2503.07269; 2502.19856; 2505.13244; 2504.08543; 2506.04409]. The task also made clear that progress in multilingual emotion detection cannot be reduced to scaling model size alone. Low-resource coverage, annotation fidelity, label imbalance, culture-specific expression, and faithful handling of multi-label structure all remain central determinants of system quality.

Source: https://www.emergentmind.com/topics/semeval-2025-task-11