SemEval 2025 Task 11: Emotion Detection Benchmark

• SemEval 2025 Task 11 is a shared task on perceived text-based emotion detection that focuses on multilingual benchmarks and low-resource language inclusion.
• The task is divided into three tracks—multi-label classification, emotion intensity detection, and cross-lingual transfer—with rigorous evaluation metrics like macro F1 and Pearson correlation.
• Key innovations include diverse modeling paradigms, culturally grounded annotation practices, and empirical insights into handling label imbalance and cross-lingual challenges.

Searching arXiv for the official task paper and closely related SemEval-2025 Task 11 system papers to ground the article in current sources. arXiv search query: SemEval 2025 Task 11 Bridging the Gap in Text-Based Emotion Detection SemEval-2025 Task 11, titled “Bridging the Gap in Text-Based Emotion Detection,” is a large-scale shared task on perceived text-based emotion detection in multilingual and low-resource settings. Its central objective is to predict the emotions that general third-party observers would attribute to a speaker from a sentence or short text snippet, rather than the speaker’s internal state (Muhammad et al., 10 Mar 2025). The task introduced a new multilingual benchmark spanning 32 languages from seven language families, with a strong focus on underserved languages and locally grounded annotation practices, and organized evaluation into three tracks: Track A: Multi-label Emotion Detection, Track B: Emotion Intensity Detection, and Track C: Cross-lingual Emotion Detection (Muhammad et al., 10 Mar 2025). Across participating systems, the task also became a testbed for contrasting encoder fine-tuning, parameter-efficient adaptation, instruction-tuned generation, retrieval-augmented inference, explanation augmentation, and adapter-based transfer (Leon et al., 11 Apr 2025).

1. Task definition and formal structure

SemEval-2025 Task 11 defines emotion recognition as a multi-label prediction problem over a fixed annotation inventory of seven emotion categories: joy, sadness, anger, fear, surprise, disgust, and neutral (Muhammad et al., 10 Mar 2025). In the official predictive setup, however, the main tasks target the six non-neutral emotions—joy, sadness, fear, anger, surprise, and disgust—while neutral is used in annotation for cases where no emotion is present (Muhammad et al., 10 Mar 2025). The task is explicitly multi-label because annotators were instructed to “select all emotions that apply,” and a single text may express more than one emotion simultaneously (Muhammad et al., 10 Mar 2025).

The shared task was divided into three tracks. Track A is the monolingual multi-label classification task: the input is a text snippet in a given language, and the output is a binary decision for each of the six emotions (Muhammad et al., 10 Mar 2025). Track B is emotion intensity detection: systems predict one of 0, 1, 2, 3 for each emotion, corresponding to no emotion, low intensity, moderate / medium intensity, and high intensity (Muhammad et al., 10 Mar 2025). Track C uses the same prediction target as Track A but imposes a cross-lingual transfer condition: systems must predict labels for a target language without using any training data in that target language (Muhammad et al., 10 Mar 2025).

The official metrics reflect these task definitions. For Tracks A and C, systems are ranked by average macro F-score (Muhammad et al., 10 Mar 2025). For Track B, the official metric is the Pearson correlation coefficient between predicted and gold labels (Muhammad et al., 10 Mar 2025). This metric design led many participating systems to optimize directly for macro-F1 in Track A, especially because the label distributions are imbalanced, and to treat Track B as a structured ordinal or categorical prediction problem rather than free-form generation (Sahil et al., 27 Feb 2025).

A recurring point across system papers is that the task’s “multi-label” framing matters operationally. Some submissions implemented genuine independent label prediction, for example with sigmoid-based outputs or per-label prompting (Sahil et al., 27 Feb 2025, Xue et al., 19 May 2025). Others nominally addressed Track A but reduced the problem to single-label or dominant-emotion classification, which created a mismatch between the official formulation and the implemented model (Sani et al., 19 Jun 2025). This divergence became one of the clearest methodological fault lines in the 2025 ecosystem.

2. Dataset design, annotation, and multilingual scope

The shared task dataset is one of its principal contributions. The organizers report that the benchmark covers 32 languages across Africa, Asia, Latin America, North America, and Europe, with strong emphasis on predominantly low-resource languages (Muhammad et al., 10 Mar 2025). The language inventory includes Afrikaans, Algerian Arabic, Amharic, Chinese, Emakhuwa, English, German, Hausa, Hindi, Igbo, Indonesian, isiXhosa, isiZulu, Javanese, Kinyarwanda, Marathi, Moroccan Arabic, Nigerian Pidgin, Oromo, Brazilian Portuguese, Mozambican Portuguese, Romanian, Russian, Somali, Latin American Spanish, Sundanese, Swahili, Swedish, Tatar, Tigrinya, Ukrainian, and Yoruba (Muhammad et al., 10 Mar 2025).

Not all languages appear in every track. Track A covered 28 languages with training data, Track B covered 11 languages, and Track C additionally included languages with no training split, used only as cross-lingual targets (Muhammad et al., 10 Mar 2025). The collection contains over 100,000 multi-labeled instances overall (Muhammad et al., 10 Mar 2025). One system paper referring to the underlying BRIGHTER resource describes it as a manually curated multilingual benchmark with nearly 100,000 text instances from 28 languages and multiple domains including social media, personal narratives, speeches, literary texts, and news (Sahil et al., 27 Feb 2025). This narrower count reflects the BRIGHTER core, whereas the task paper situates the shared task at the broader 32-language level (Muhammad et al., 10 Mar 2025).

The annotation protocol is unusually explicit about perceived emotion and intensity. Annotators first selected applicable emotions, then assigned intensity values. A given emotion was marked as present if at least two annotators selected that label with intensity 1, 2, or 3, and the average score for that emotion exceeded the threshold $T = 0.5$ (Muhammad et al., 10 Mar 2025). For Track B, the final gold intensity score was obtained by averaging the selected intensity values and rounding up to the nearest whole number (Muhammad et al., 10 Mar 2025). The organizers report annotator reliability using Split-Half Class Match Percentage (SHCMP), with values ranging roughly from 60% to over 90% (Muhammad et al., 10 Mar 2025).

Data collection was multilingual not only in coverage but also in source construction. The organizers used social media posts, personal narratives, talks, and speeches, literary texts, news data and headlines, and, for some languages, human-written and machine-generated text (Muhammad et al., 10 Mar 2025). All instances were manually annotated by fluent speakers, with collaboration from local communities and compensation above minimum hourly wage (Muhammad et al., 10 Mar 2025). This suggests that the benchmark was designed not merely as a translated multilingual benchmark, but as a culturally situated resource for perceived emotion detection.

A number of system papers supplement this dataset picture with language-specific details. Team A reports that BRIGHTER annotations include one or more labels from anger, sadness, fear, disgust, joy, surprise, plus neutral, along with intensity values on a 4-point scale from 0 to 3, and notes complementary use of EthioEmo for Amharic and Afan Oromo (Sahil et al., 27 Feb 2025). UoB-NLP likewise describes its experiments as using BRIGHTER and EthioEmo, especially for low-resource African languages (Leon et al., 11 Apr 2025). These descriptions are consistent with the official task framing, though exact per-language counts vary by paper because some focus on subsets or particular tracks.

3. Baselines, participation, and overall competitive landscape

SemEval-2025 Task 11 was one of the largest shared tasks of its cycle. The organizers report over 700 participants, 220 final submissions, and 93 system description papers (Muhammad et al., 10 Mar 2025). They also state that the task was the most popular competition hosted on Codabench in 2024 (Muhammad et al., 10 Mar 2025). The reporting mixes teams and submissions, but the scale of participation is unambiguous (Muhammad et al., 10 Mar 2025).

The official baselines were intentionally simple. Across all tracks, the organizers used a majority-class baseline for each language (Muhammad et al., 10 Mar 2025). For Tracks A and B, they additionally fine-tuned RoBERTa on the training data for each language (Muhammad et al., 10 Mar 2025). For Track C, the RoBERTa baseline was cross-lingual and trained on all languages in the same language family, excluding the target language (Muhammad et al., 10 Mar 2025). These baselines proved informative but left considerable headroom, especially in low-resource and zero-target-supervision settings.

Selected results from the task paper illustrate the spread. In Track A, English reached 0.823 for the top system versus 0.708 for the RoBERTa baseline and 0.367 for majority; Hindi reached 0.926 versus 0.855 and 0.246; Russian reached 0.901 versus 0.838 and 0.262 (Muhammad et al., 10 Mar 2025). In Track B, Russian reached 0.925 Pearson versus 0.877 for RoBERTa, English reached 0.840 versus 0.641, and Algerian Arabic reached 0.650 versus 0.016 (Muhammad et al., 10 Mar 2025). In Track C, performance dropped overall, with English at 0.797 versus 0.375, Hindi at 0.919 versus 0.138, and Emakhuwa only 0.210 versus 0.052 (Muhammad et al., 10 Mar 2025).

The organizers identify several dominant teams and patterns. In Track A, Team PAI ranked first in 20 out of 28 languages using an LLM ensemble combining ChatGPT-4o, DeepSeek-V3, Gemma-9B, Qwen-2.5-32B, and Mistral-Small-24B, with ensembling methods including neural networks, XGBoost, LightGBM, linear regression, and weighted voting (Muhammad et al., 10 Mar 2025). In Track B, PAI again led in nearly every language except Amharic, where CSECU-Learners ranked first with 0.856 Pearson using language-specific transformers and multi-sample dropout (Muhammad et al., 10 Mar 2025). In Track C, Deepwave led overall with Gemma-2, data augmentation, chain-of-thought prompting, 5-fold cross-validation, and model merging (Muhammad et al., 10 Mar 2025).

A broad trend emerges from the official analysis: the strongest systems largely relied on either fine-tuned transformer encoders such as XLM-R, mBERT, DeBERTa, or instruction-tuned / prompted LLMs, often combined with LoRA / PEFT, prompt engineering, and ensembling (Muhammad et al., 10 Mar 2025). At the same time, the task landscape remained methodologically heterogeneous, with strong submissions also coming from adapter-based models, retrieval-augmented methods, and feature-centric pipelines (Leon et al., 11 Apr 2025, Morozov et al., 4 Jun 2025, Huang et al., 11 Jul 2025).

4. Dominant modeling paradigms

The 2025 task ecosystem was shaped by several distinct modeling paradigms rather than a single consensus architecture.

A first paradigm was multilingual encoder fine-tuning. Team A’s TransferModel_FC_EmbeddingE5 used multilingual-e5-large-instruct with [CLS] pooling, dropout $0.3$, a 1024-to-6 fully connected layer, sigmoid outputs, BCE with label smoothing $a = 0.1$, AdamW with learning rate $1\times 10^{-5}$, batch size 16, gradient clipping at 1.0, early stopping on dev macro F1 with patience 4, and threshold 0.5 at inference (Sahil et al., 27 Feb 2025). Their conclusion was explicit: the best-performing approach was multilingual E5 embeddings plus a fully connected layer (Sahil et al., 27 Feb 2025). The University of Indonesia system, by contrast, reports that training a classifier on top of prompt-based encoders such as mE5 and BGE yields significantly better results than fully fine-tuning XLMR and mBERT, and that its best leaderboard model was an ensemble of multiple BGE models with CatBoost as classifier, achieving 56.58 average F1-macro across all languages (Hanif et al., 22 May 2025). This juxtaposition suggests a broader 2025 pattern: sentence-embedding-style encoders plus lightweight classifiers were often competitive with, and in some cases superior to, full end-to-end multilingual transformer fine-tuning.

A second paradigm was instruction-tuned and generative LLM modeling. JNLP framed Track A and Track B either as direct generation of emotion labels or as pairwise text-label prediction, using XLM-RoBERTa-Large, LLaMA 2, and Qwen2.5, and reported Top 4 performance across 10 languages in Track A with 1st place in Hindi, as well as Top 5 performance in 7 languages in Track B (Xue et al., 19 May 2025). Chinchunmei unified Track A and Track B within a single Llama3-Instruct-8B framework using prompt-based structured generation, then compared Standard Prediction, Contrastive Reasoning Calibration (CRC), DPO, and SimPO (Li et al., 21 Jul 2025). Their main conclusion was that a strong generative SFT baseline was hard to beat on Track A, whereas DPO-style generation-level contrastive learning was the most promising extension for Track B (Li et al., 21 Jul 2025). CSIRO-LT similarly concluded that the most effective practical strategy was to fine-tune a pre-trained multilingual LLM with LoRA setting separately for each language, outperforming prompting-heavy alternatives in their development comparisons (Chen et al., 2 Aug 2025).

A third paradigm was adapter-based parameter-efficient transfer. UoB-NLP compared task-only adapters, target-language-ready task adapters (TLR), and language-family-based adapters on top of XLM-RoBERTa-base and Afro-XLMR-base (Leon et al., 11 Apr 2025). Their central finding was that target-language-ready task adapters achieve the best overall performance, particularly for low-resource African languages, with 7th for Tigrinya and 8th for Kinyarwanda in Track A, and 3rd for Amharic plus 4th in several other Track C languages (Leon et al., 11 Apr 2025). They also state that their adapter-based models outperform LLMs in 11 languages and match their performance in four others, despite using far fewer parameters (Leon et al., 11 Apr 2025).

A fourth paradigm was retrieval-augmented and training-free inference. EmoRAG built a database of labeled training examples, retrieved the top-$K$ most similar examples, and used them as few-shot context for Llama-3.1-70B-Instruct, Qwen2.5-72B-Instruct, gpt-4o-mini-2024-07-18, and gemma-2-27b-it, followed by several aggregation rules (Morozov et al., 4 Jun 2025). The strongest strategy was majority_vote_by_label_f1, a label-specific weighted aggregation based on development-set performance (Morozov et al., 4 Jun 2025). The system required no additional model training and achieved 0.638 average test F1-micro and 0.590 average test F1-macro (Morozov et al., 4 Jun 2025).

A fifth paradigm was feature-centric shallow or hybrid modeling. PromotionGo compared TF-IDF, FastText, and Sentence-BERT representations, optionally with PCA, and downstream DT, Voting, and MLP classifiers across 28 languages (Huang et al., 11 Jul 2025). Their findings emphasize that TF-IDF remains highly effective for low-resource languages, while SBERT generally outperforms TF-IDF and FastText in 18 out of 28 languages, especially with MLP, and PCA can reduce training time selectively without universally improving accuracy (Huang et al., 11 Jul 2025).

These paradigms did not simply coexist; they frequently contradicted one another’s inductive biases. Some system papers argue that strong multilingual encoders already contain enough cross-lingual affective structure that a lightweight head suffices (Sahil et al., 27 Feb 2025). Others argue that language-specific or African-specialized pretraining remains essential (Leon et al., 11 Apr 2025, Sani et al., 19 Jun 2025). Still others show that explanation generation, retrieval, or pairwise prompting can compensate for sparse or ambiguous surface cues (Ranjbar et al., 27 Feb 2025, Morozov et al., 4 Jun 2025, Xue et al., 19 May 2025).

5. Recurrent empirical findings across systems

Several empirical regularities recur across the literature.

A first recurrent finding is that multilingual pretrained representations are strong but unevenly reliable across languages. Team A achieved highest macro F1 in Hindi 0.8901, Russian 0.8831, and Marathi 0.8657, but weaker results in Arabic 0.5550 and Oromo 0.5628 (Sahil et al., 27 Feb 2025). Their interpretation is that multilingual E5 captures shared semantic and affective structure across languages, but the weaker scores in Arabic, Oromo, Chinese, and Ukrainian indicate that multilingual alignment remains imperfect (Sahil et al., 27 Feb 2025). The task paper echoes this at scale, noting that higher-resource languages such as English and Russian generally achieved better results across tracks, whereas lower-resource languages and cross-lingual settings remained difficult (Muhammad et al., 10 Mar 2025).

A second recurrent finding is that simple or parameter-efficient methods can remain competitive. UoB-NLP shows that adapter-based models can outperform or match much larger LLMs in many languages (Leon et al., 11 Apr 2025). PromotionGo shows that TF-IDF can be stronger than dense representations in languages such as Marathi, Spanish, Russian, Oromo, Somali, Hausa, and Tatar, especially under resource constraints (Huang et al., 11 Jul 2025). EmoRAG demonstrates that a retrieval-augmented ensemble can be competitive without additional model training (Morozov et al., 4 Jun 2025). These results complicate any simple narrative that the task was won exclusively by larger end-to-end generative models.

A third recurrent finding is that label imbalance remains central. Team A preferred macro F1 because it penalizes bias toward frequent labels (Sahil et al., 27 Feb 2025). Lotus explicitly notes that it did not use resampling, class weights, focal loss, or threshold tuning despite training on a class-imbalanced English subset where fear is by far the most frequent label and anger much rarer (Ranjbar et al., 27 Feb 2025). PromotionGo’s Hindi analysis reports that over 78% of the 2,556 samples are labeled as not anger, producing high true-negative rates but lower recall on minority positives (Huang et al., 11 Jul 2025). Chinchunmei’s Track B analysis similarly indicates that rare emotion-intensity combinations are hard to model (Li et al., 21 Jul 2025). This suggests that data imbalance was not merely a statistical inconvenience but a major determinant of architecture and metric choice.

A fourth recurrent finding is that multi-label structure helps some methods but is not always handled faithfully. Genuine multi-label systems include Team A’s sigmoid-based E5 model (Sahil et al., 27 Feb 2025), Lotus’s five-label RoBERTa setup with explanation augmentation (Ranjbar et al., 27 Feb 2025), JNLP’s base and pairwise formulations (Xue et al., 19 May 2025), and CSIRO-LT’s six independent binary or four-way per-emotion formulations (Chen et al., 2 Aug 2025). In contrast, HausaNLP explicitly collapses the one-hot emotion vector to a single dominant emotion, converting Track A into a six-class classification task despite the shared task’s multi-label framing (Sani et al., 19 Jun 2025). The paper itself acknowledges that this makes the system “effectively multi-class rather than multi-label” (Sani et al., 19 Jun 2025). This is an important caution for interpreting SemEval-2025 Task 11 system descriptions: nominal participation in Track A does not always imply faithful multi-label modeling.

A fifth recurrent finding is that language mixing is helpful in some settings but harmful in others. JNLP reports that mixed-language training generally outperforms separate per-language training in both Track A and Track B (Xue et al., 19 May 2025). Chinchunmei reports the opposite for English-focused experiments: adding multilingual training data hurt English performance in both tracks, which they attribute to cross-cultural differences in emotion perception (Li et al., 21 Jul 2025). This suggests that multilingual pooling is not uniformly beneficial; its effect depends on language composition, model family, and the extent to which emotion annotation conventions are aligned across languages.

6. Methodological debates, limitations, and significance

SemEval-2025 Task 11 exposed several methodological debates that are likely to remain relevant beyond the 2025 edition.

One debate concerns whether multilingual generalization is best achieved through shared representation or language-specific specialization. Encoder-centric systems such as Team A emphasize cross-lingual semantic alignment learned during pretraining (Sahil et al., 27 Feb 2025). Adapter-based systems argue for modular specialization, especially for low-resource African languages (Leon et al., 11 Apr 2025). CSIRO-LT goes further, concluding that the best approach is separate per-language LoRA fine-tuning, not a single jointly trained multilingual model (Chen et al., 2 Aug 2025). The coexistence of these results suggests that multilinguality in perceived emotion detection is not a monolithic transfer problem; it is shaped by pretraining coverage, typological distance, annotation design, and culture-specific expression.

A second debate concerns whether generation improves classification. Lotus argues that Llama-3-generated explanations improve multi-label emotion detection by making ambiguous or under-explained text more interpretable, yielding gains in both macro and micro F1 over text-only RoBERTa (Ranjbar et al., 27 Feb 2025). Chinchunmei finds that sample-based contrastive reasoning calibration offers limited benefit, whereas DPO helps Track B more than Track A (Li et al., 21 Jul 2025). JNLP shows that pairwise prompting can outperform direct generation when texts express multiple emotions, but the base method can be stronger when label cardinality is low (Xue et al., 19 May 2025). These findings suggest that generative augmentation is most useful when it induces a more appropriate decision decomposition, not simply because natural-language output is intrinsically superior.

A third debate concerns resource use versus reproducibility. Retrieval-augmented systems avoid retraining but can depend on multiple large proprietary or closed models (Morozov et al., 4 Jun 2025). Many LLM-based system papers omit crucial hyperparameters such as decoding settings, calibration procedures, or preference-learning coefficients (Ranjbar et al., 27 Feb 2025, Li et al., 21 Jul 2025, Chen et al., 2 Aug 2025). Feature-centric and adapter-based systems are often more transparent about training configurations, but may omit exact multi-label decision mechanics or hyperparameter grids (Huang et al., 11 Jul 2025, Leon et al., 11 Apr 2025). This suggests that the shared task advanced empirical performance more quickly than it resolved reproducibility standards.

A fourth debate concerns the limits of benchmark fidelity. Several papers identify mismatches between official task framing and actual system design. HausaNLP reduces Track A to dominant-emotion classification (Sani et al., 19 Jun 2025). One paper in the supplied corpus, attributed to the University of Indonesia, cannot be summarized from its provided “paper” text because the available document is an ACL template rather than the actual system description, despite its metadata claiming a SemEval submission (Hanif et al., 22 May 2025). A plausible implication is that shared-task ecosystems remain partly dependent on inconsistent archival quality, especially when preprints, templates, and workshop proceedings circulate asynchronously.

Despite these tensions, the significance of SemEval-2025 Task 11 is clear. It established a multilingual benchmark for perceived emotion detection over a large and geographically broad language inventory, introduced formal evaluation for both multi-label emotion detection and emotion intensity, and catalyzed a wide comparative space of methods ranging from XLM-R and mE5 encoders to LoRA-adapted LLMs, retrieval-augmented ensembles, contrastive generation, and adapter-based transfer (Muhammad et al., 10 Mar 2025, Sahil et al., 27 Feb 2025, Xue et al., 19 May 2025, Leon et al., 11 Apr 2025, Morozov et al., 4 Jun 2025). The task also made clear that progress in multilingual emotion detection cannot be reduced to scaling model size alone. Low-resource coverage, annotation fidelity, label imbalance, culture-specific expression, and faithful handling of multi-label structure all remain central determinants of system quality.