SemEval 2025 Task 11: Emotion Detection Benchmark

• SemEval 2025 Task 11 is a shared task on perceived text-based emotion detection that focuses on multilingual benchmarks and low-resource language inclusion.
• The task is divided into three tracks—multi-label classification, emotion intensity detection, and cross-lingual transfer—with rigorous evaluation metrics like macro F1 and Pearson correlation.
• Key innovations include diverse modeling paradigms, culturally grounded annotation practices, and empirical insights into handling label imbalance and cross-lingual challenges.

Searching arXiv for the official task paper and closely related SemEval-2025 Task 11 system papers to ground the article in current sources.
arXiv search query: SemEval 2025 Task 11 Bridging the Gap in Text-Based Emotion Detection
SemEval-2025 Task 11, titled “Bridging the Gap in Text-Based Emotion Detection,” is a large-scale shared task on perceived text-based emotion detection in multilingual and low-resource settings. Its central objective is to predict the emotions that general third-party observers would attribute to a speaker from a sentence or short text snippet, rather than the speaker’s internal state [2503.07269]. The task introduced a new multilingual benchmark spanning 32 languages from seven language families, with a strong focus on underserved languages and locally grounded annotation practices, and organized evaluation into three tracks: Track A: Multi-label Emotion Detection, Track B: Emotion Intensity Detection, and Track C: Cross-lingual Emotion Detection [2503.07269]. Across participating systems, the task also became a testbed for contrasting encoder fine-tuning, parameter-efficient adaptation, instruction-tuned generation, retrieval-augmented inference, explanation augmentation, and adapter-based transfer [2504.08543].

1. Task definition and formal structure

SemEval-2025 Task 11 defines emotion recognition as a multi-label prediction problem over a fixed annotation inventory of seven emotion categories: joy, sadness, anger, fear, surprise, disgust, and neutral [2503.07269]. In the official predictive setup, however, the main tasks target the six non-neutral emotions—joy, sadness, fear, anger, surprise, and disgust—while neutral is used in annotation for cases where no emotion is present [2503.07269]. The task is explicitly multi-label because annotators were instructed to “select all emotions that apply,” and a single text may express more than one emotion simultaneously [2503.07269].

The shared task was divided into three tracks. Track A is the monolingual multi-label classification task: the input is a text snippet in a given language, and the output is a binary decision for each of the six emotions [2503.07269]. Track B is emotion intensity detection: systems predict one of 0, 1, 2, 3 for each emotion, corresponding to no emotion, low intensity, moderate / medium intensity, and high intensity [2503.07269]. Track C uses the same prediction target as Track A but imposes a cross-lingual transfer condition: systems must predict labels for a target language without using any training data in that target language [2503.07269].

The official metrics reflect these task definitions. For Tracks A and C, systems are ranked by average macro F-score [2503.07269]. For Track B, the official metric is the Pearson correlation coefficient between predicted and gold labels [2503.07269]. This metric design led many participating systems to optimize directly for macro-F1 in Track A, especially because the label distributions are imbalanced, and to treat Track B as a structured ordinal or categorical prediction problem rather than free-form generation [2502.19856].

A recurring point across system papers is that the task’s “multi-label” framing matters operationally. Some submissions implemented genuine independent label prediction, for example with sigmoid-based outputs or per-label prompting [2502.19856; 2505.13244]. Others nominally addressed Track A but reduced the problem to single-label or dominant-emotion classification, which created a mismatch between the official formulation and the implemented model [2506.16388]. This divergence became one of the clearest methodological fault lines in the 2025 ecosystem.

2. Dataset design, annotation, and multilingual scope

The shared task dataset is one of its principal contributions. The organizers report that the benchmark covers 32 languages across Africa, Asia, Latin America, North America, and Europe, with strong emphasis on predominantly low-resource languages [2503.07269]. The language inventory includes Afrikaans, Algerian Arabic, Amharic, Chinese, Emakhuwa, English, German, Hausa, Hindi, Igbo, Indonesian, isiXhosa, isiZulu, Javanese, Kinyarwanda, Marathi, Moroccan Arabic, Nigerian Pidgin, Oromo, Brazilian Portuguese, Mozambican Portuguese, Romanian, Russian, Somali, Latin American Spanish, Sundanese, Swahili, Swedish, Tatar, Tigrinya, Ukrainian, and Yoruba [2503.07269].

Not all languages appear in every track. Track A covered 28 languages with training data, Track B covered 11 languages, and Track C additionally included languages with no training split, used only as cross-lingual targets [2503.07269]. The collection contains over 100,000 multi-labeled instances overall [2503.07269]. One system paper referring to the underlying BRIGHTER resource describes it as a manually curated multilingual benchmark with nearly 100,000 text instances from 28 languages and multiple domains including social media, personal narratives, speeches, literary texts, and news [2502.19856]. This narrower count reflects the BRIGHTER core, whereas the task paper situates the shared task at the broader 32-language level [2503.07269].

The annotation protocol is unusually explicit about perceived emotion and intensity. Annotators first selected applicable emotions, then assigned intensity values. A given emotion was marked as present if at least two annotators selected that label with intensity 1, 2, or 3, and the average score for that emotion exceeded the threshold (T = 0.5) [2503.07269]. For Track B, the final gold intensity score was obtained by averaging the selected intensity values and rounding up to the nearest whole number [2503.07269]. The organizers report annotator reliability using Split-Half Class Match Percentage (SHCMP), with values ranging roughly from 60% to over 90% [2503.07269].

Data collection was multilingual not only in coverage but also in source construction. The organizers used social media posts, personal narratives, talks, and speeches, literary texts, news data and headlines, and, for some languages, human-written and machine-generated text [2503.07269]. All instances were manually annotated by fluent speakers, with collaboration from local communities and compensation above minimum hourly wage [2503.07269]. This suggests that the benchmark was designed not merely as a translated multilingual benchmark, but as a culturally situated resource for perceived emotion detection.

A number of system papers supplement this dataset picture with language-specific details. Team A reports that BRIGHTER annotations include one or more labels from anger, sadness, fear, disgust, joy, surprise, plus neutral, along with intensity values on a 4-point scale from 0 to 3, and notes complementary use of EthioEmo for Amharic and Afan Oromo [2502.19856]. UoB-NLP likewise describes its experiments as using BRIGHTER and EthioEmo, especially for low-resource African languages [2504.08543]. These descriptions are consistent with the official task framing, though exact per-language counts vary by paper because some focus on subsets or particular tracks.

3. Baselines, participation, and overall competitive landscape

SemEval-2025 Task 11 was one of the largest shared tasks of its cycle. The organizers report over 700 participants, 220 final submissions, and 93 system description papers [2503.07269]. They also state that the task was the most popular competition hosted on Codabench in 2024 [2503.07269]. The reporting mixes teams and submissions, but the scale of participation is unambiguous [2503.07269].

The official baselines were intentionally simple. Across all tracks, the organizers used a majority-class baseline for each language [2503.07269]. For Tracks A and B, they additionally fine-tuned RoBERTa on the training data for each language [2503.07269]. For Track C, the RoBERTa baseline was cross-lingual and trained on all languages in the same language family, excluding the target language [2503.07269]. These baselines proved informative but left considerable headroom, especially in low-resource and zero-target-supervision settings.

Selected results from the task paper illustrate the spread. In Track A, English reached 0.823 for the top system versus 0.708 for the RoBERTa baseline and 0.367 for majority; Hindi reached 0.926 versus 0.855 and 0.246; Russian reached 0.901 versus 0.838 and 0.262 [2503.07269]. In Track B, Russian reached 0.925 Pearson versus 0.877 for RoBERTa, English reached 0.840 versus 0.641, and Algerian Arabic reached 0.650 versus 0.016 [2503.07269]. In Track C, performance dropped overall, with English at 0.797 versus 0.375, Hindi at 0.919 versus 0.138, and Emakhuwa only 0.210 versus 0.052 [2503.07269].

The organizers identify several dominant teams and patterns. In Track A, Team PAI ranked first in 20 out of 28 languages using an LLM ensemble combining ChatGPT-4o, DeepSeek-V3, Gemma-9B, Qwen-2.5-32B, and Mistral-Small-24B, with ensembling methods including neural networks, XGBoost, LightGBM, linear regression, and weighted voting [2503.07269]. In Track B, PAI again led in nearly every language except Amharic, where CSECU-Learners ranked first with 0.856 Pearson using language-specific transformers and multi-sample dropout [2503.07269]. In Track C, Deepwave led overall with Gemma-2, data augmentation, chain-of-thought prompting, 5-fold cross-validation, and model merging [2503.07269].

A broad trend emerges from the official analysis: the strongest systems largely relied on either fine-tuned transformer encoders such as XLM-R, mBERT, DeBERTa, or instruction-tuned / prompted LLMs, often combined with LoRA / PEFT, prompt engineering, and ensembling [2503.07269]. At the same time, the task landscape remained methodologically heterogeneous, with strong submissions also coming from adapter-based models, retrieval-augmented methods, and feature-centric pipelines [2504.08543; 2506.04409; 2507.08499].

4. Dominant modeling paradigms

The 2025 task ecosystem was shaped by several distinct modeling paradigms rather than a single consensus architecture.

A first paradigm was multilingual encoder fine-tuning. Team A’s TransferModel_FC_EmbeddingE5 used multilingual-e5-large-instruct with [CLS] pooling, dropout (0.3), a 1024-to-6 fully connected layer, sigmoid outputs, BCE with label smoothing (a = 0.1), AdamW with learning rate (1\times 10^{-5}), batch size 16, gradient clipping at 1.0, early stopping on dev macro F1 with patience 4, and threshold 0.5 at inference [2502.19856]. Their conclusion was explicit: the best-performing approach was multilingual E5 embeddings plus a fully connected layer [2502.19856]. The University of Indonesia system, by contrast, reports that training a classifier on top of prompt-based encoders such as mE5 and BGE yields significantly better results than fully fine-tuning XLMR and mBERT, and that its best leaderboard model was an ensemble of multiple BGE models with CatBoost as classifier, achieving 56.58 average F1-macro across all languages [2505.16460]. This juxtaposition suggests a broader 2025 pattern: sentence-embedding-style encoders plus lightweight classifiers were often competitive with, and in some cases superior to, full end-to-end multilingual transformer fine-tuning.

A second paradigm was instruction-tuned and generative LLM modeling. JNLP framed Track A and Track B either as direct generation of emotion labels or as pairwise text-label prediction, using XLM-RoBERTa-Large, LLaMA 2, and Qwen2.5, and reported Top 4 performance across 10 languages in Track A with 1st place in Hindi, as well as Top 5 performance in 7 languages in Track B [2505.13244]. Chinchunmei unified Track A and Track B within a single Llama3-Instruct-8B framework using prompt-based structured generation, then compared Standard Prediction, Contrastive Reasoning Calibration (CRC), DPO, and SimPO [2507.15714]. Their main conclusion was that a strong generative SFT baseline was hard to beat on Track A, whereas DPO-style generation-level contrastive learning was the most promising extension for Track B [2507.15714]. CSIRO-LT similarly concluded that the most effective practical strategy was to fine-tune a pre-trained multilingual LLM with LoRA setting separately for each language, outperforming prompting-heavy alternatives in their development comparisons [2508.01161].

A third paradigm was adapter-based parameter-efficient transfer. UoB-NLP compared task-only adapters, target-language-ready task adapters (TLR), and language-family-based adapters on top of XLM-RoBERTa-base and Afro-XLMR-base [2504.08543]. Their central finding was that target-language-ready task adapters achieve the best overall performance, particularly for low-resource African languages, with 7th for Tigrinya and 8th for Kinyarwanda in Track A, and 3rd for Amharic plus 4th in several other Track C languages [2504.08543]. They also state that their adapter-based models outperform large language models in 11 languages and match their performance in four others, despite using far fewer parameters [2504.08543].

A fourth paradigm was retrieval-augmented and training-free inference. EmoRAG built a database of labeled training examples, retrieved the top-(K) most similar examples, and used them as few-shot context for Llama-3.1-70B-Instruct, Qwen2.5-72B-Instruct, gpt-4o-mini-2024-07-18, and gemma-2-27b-it, followed by several aggregation rules [2506.04409]. The strongest strategy was majority_vote_by_label_f1, a label-specific weighted aggregation based on development-set performance [2506.04409]. The system required no additional model training and achieved 0.638 average test F1-micro and 0.590 average test F1-macro [2506.04409].

A fifth paradigm was feature-centric shallow or hybrid modeling. PromotionGo compared TF-IDF, FastText, and Sentence-BERT representations, optionally with PCA, and downstream DT, Voting, and MLP classifiers across 28 languages [2507.08499]. Their findings emphasize that TF-IDF remains highly effective for low-resource languages, while SBERT generally outperforms TF-IDF and FastText in 18 out of 28 languages, especially with MLP, and PCA can reduce training time selectively without universally improving accuracy [2507.08499].

These paradigms did not simply coexist; they frequently contradicted one another’s inductive biases. Some system papers argue that strong multilingual encoders already contain enough cross-lingual affective structure that a lightweight head suffices [2502.19856]. Others argue that language-specific or African-specialized pretraining remains essential [2504.08543; 2506.16388]. Still others show that explanation generation, retrieval, or pairwise prompting can compensate for sparse or ambiguous surface cues [2502.19935; 2506.04409; 2505.13244].

5. Recurrent empirical findings across systems

Several empirical regularities recur across the literature.

A first recurrent finding is that multilingual pretrained representations are strong but unevenly reliable across languages. Team A achieved highest macro F1 in Hindi 0.8901, Russian 0.8831, and Marathi 0.8657, but weaker results in Arabic 0.5550 and Oromo 0.5628 [2502.19856]. Their interpretation is that multilingual E5 captures shared semantic and affective structure across languages, but the weaker scores in Arabic, Oromo, Chinese, and Ukrainian indicate that multilingual alignment remains imperfect [2502.19856]. The task paper echoes this at scale, noting that higher-resource languages such as English and Russian generally achieved better results across tracks, whereas lower-resource languages and cross-lingual settings remained difficult [2503.07269].

A second recurrent finding is that simple or parameter-efficient methods can remain competitive. UoB-NLP shows that adapter-based models can outperform or match much larger LLMs in many languages [2504.08543]. PromotionGo shows that TF-IDF can be stronger than dense representations in languages such as Marathi, Spanish, Russian, Oromo, Somali, Hausa, and Tatar, especially under resource constraints [2507.08499]. EmoRAG demonstrates that a retrieval-augmented ensemble can be competitive without additional model training [2506.04409]. These results complicate any simple narrative that the task was won exclusively by larger end-to-end generative models.

A third recurrent finding is that label imbalance remains central. Team A preferred macro F1 because it penalizes bias toward frequent labels [2502.19856]. Lotus explicitly notes that it did not use resampling, class weights, focal loss, or threshold tuning despite training on a class-imbalanced English subset where fear is by far the most frequent label and anger much rarer [2502.19935]. PromotionGo’s Hindi analysis reports that over 78% of the 2,556 samples are labeled as not anger, producing high true-negative rates but lower recall on minority positives [2507.08499]. Chinchunmei’s Track B analysis similarly indicates that rare emotion-intensity combinations are hard to model [2507.15714]. This suggests that data imbalance was not merely a statistical inconvenience but a major determinant of architecture and metric choice.

A fourth recurrent finding is that multi-label structure helps some methods but is not always handled faithfully. Genuine multi-label systems include Team A’s sigmoid-based E5 model [2502.19856], Lotus’s five-label RoBERTa setup with explanation augmentation [2502.19935], JNLP’s base and pairwise formulations [2505.13244], and CSIRO-LT’s six independent binary or four-way per-emotion formulations [2508.01161]. In contrast, HausaNLP explicitly collapses the one-hot emotion vector to a single dominant emotion, converting Track A into a six-class classification task despite the shared task’s multi-label framing [2506.16388]. The paper itself acknowledges that this makes the system “effectively multi-class rather than multi-label” [2506.16388]. This is an important caution for interpreting SemEval-2025 Task 11 system descriptions: nominal participation in Track A does not always imply faithful multi-label modeling.

A fifth recurrent finding is that language mixing is helpful in some settings but harmful in others. JNLP reports that mixed-language training generally outperforms separate per-language training in both Track A and Track B [2505.13244]. Chinchunmei reports the opposite for English-focused experiments: adding multilingual training data hurt English performance in both tracks, which they attribute to cross-cultural differences in emotion perception [2507.15714]. This suggests that multilingual pooling is not uniformly beneficial; its effect depends on language composition, model family, and the extent to which emotion annotation conventions are aligned across languages.

6. Methodological debates, limitations, and significance

SemEval-2025 Task 11 exposed several methodological debates that are likely to remain relevant beyond the 2025 edition.

One debate concerns whether multilingual generalization is best achieved through shared representation or language-specific specialization. Encoder-centric systems such as Team A emphasize cross-lingual semantic alignment learned during pretraining [2502.19856]. Adapter-based systems argue for modular specialization, especially for low-resource African languages [2504.08543]. CSIRO-LT goes further, concluding that the best approach is separate per-language LoRA fine-tuning, not a single jointly trained multilingual model [2508.01161]. The coexistence of these results suggests that multilinguality in perceived emotion detection is not a monolithic transfer problem; it is shaped by pretraining coverage, typological distance, annotation design, and culture-specific expression.

A second debate concerns whether generation improves classification. Lotus argues that Llama-3-generated explanations improve multi-label emotion detection by making ambiguous or under-explained text more interpretable, yielding gains in both macro and micro F1 over text-only RoBERTa [2502.19935]. Chinchunmei finds that sample-based contrastive reasoning calibration offers limited benefit, whereas DPO helps Track B more than Track A [2507.15714]. JNLP shows that pairwise prompting can outperform direct generation when texts express multiple emotions, but the base method can be stronger when label cardinality is low [2505.13244]. These findings suggest that generative augmentation is most useful when it induces a more appropriate decision decomposition, not simply because natural-language output is intrinsically superior.

A third debate concerns resource use versus reproducibility. Retrieval-augmented systems avoid retraining but can depend on multiple large proprietary or closed models [2506.04409]. Many LLM-based system papers omit crucial hyperparameters such as decoding settings, calibration procedures, or preference-learning coefficients [2502.19935; 2507.15714; 2508.01161]. Feature-centric and adapter-based systems are often more transparent about training configurations, but may omit exact multi-label decision mechanics or hyperparameter grids [2507.08499; 2504.08543]. This suggests that the shared task advanced empirical performance more quickly than it resolved reproducibility standards.

A fourth debate concerns the limits of benchmark fidelity. Several papers identify mismatches between official task framing and actual system design. HausaNLP reduces Track A to dominant-emotion classification [2506.16388]. One paper in the supplied corpus, attributed to the University of Indonesia, cannot be summarized from its provided “paper” text because the available document is an ACL template rather than the actual system description, despite its metadata claiming a SemEval submission [2505.16460]. A plausible implication is that shared-task ecosystems remain partly dependent on inconsistent archival quality, especially when preprints, templates, and workshop proceedings circulate asynchronously.

Despite these tensions, the significance of SemEval-2025 Task 11 is clear. It established a multilingual benchmark for perceived emotion detection over a large and geographically broad language inventory, introduced formal evaluation for both multi-label emotion detection and emotion intensity, and catalyzed a wide comparative space of methods ranging from XLM-R and mE5 encoders to LoRA-adapted LLMs, retrieval-augmented ensembles, contrastive generation, and adapter-based transfer [2503.07269; 2502.19856; 2505.13244; 2504.08543; 2506.04409]. The task also made clear that progress in multilingual emotion detection cannot be reduced to scaling model size alone. Low-resource coverage, annotation fidelity, label imbalance, culture-specific expression, and faithful handling of multi-label structure all remain central determinants of system quality.