SemEval-2026 Task 9: Detecting Multilingual, Multicultural and Multievent Online Polarization

Abstract: We present SemEval-2026 Task 9, a shared task on online polarization detection, covering 22 languages and comprising over 110K annotated instances. Each data instance is multi-labeled with the presence of polarization, polarization type, and polarization manifestation. Participants were asked to predict labels in three sub-tasks: (1) detecting the presence of polarization, (2) identifying the type of polarization, and (3) recognizing the polarization manifestation. The three tasks attracted over 1,000 participants worldwide and more than 10k submission on Codabench. We received final submissions from 67 teams and 73 system description papers. We report the baseline results and analyze the performance of the best-performing systems, highlighting the most common approaches and the most effective methods across different subtasks and languages. The dataset of this task is publicly available.

• The paper introduces a novel multilingual benchmark dataset annotated for polarization across 22 languages and varied cultural contexts.
• It demonstrates that ensemble techniques, parameter-efficient fine-tuning, and data augmentation can enhance model performance despite data imbalances.
• Results highlight the need for improved cross-cultural adaptation and refined annotation protocols to boost generalization in polarization detection.

SemEval-2026 Task 9: Detecting Multilingual, Multicultural, and Multievent Online Polarization

Task Overview and Motivation

SemEval-2026 Task 9 targets automated detection of polarization in online text across a diverse set of languages, cultural contexts, and event types. The task is motivated by the need for scalable, cross-lingual computational models to recognize polarization—defined as antagonistic division between social, political, or identity groups—given its risks to democratic discourse, social cohesion, and digital governance. Prior computational efforts have been limited by region-, event-, or language-specific datasets, constraining generalizability. This task addresses that gap with an extensive benchmark dataset supporting fine-grained annotation in 22 languages, facilitating cross-lingual and context-aware modeling for polarization detection (2604.06817).

Dataset Construction and Annotation

The dataset comprises over 110,000 texts annotated for three aspects: polarization presence (binary), polarization type (political, racial/ethnic, religious, gender/sexual, other), and manifestation (stereotyping, vilification, dehumanization, extreme language, lack of empathy, invalidation). Data were sourced from major social platforms and local forums, capturing diverse real-world events including elections, public health crises, armed conflicts, and climate debates. Existing hate speech datasets were leveraged for several languages.

Annotation utilized a hybrid strategy: crowd-sourced platforms (Mechanical Turk, Prolific) for high-resource languages, and trained community annotators for low-resource contexts. Guidelines were translated and culturally adapted per language to address contextual nuances. Multi-label annotation was allowed to capture compositional and overlapping manifestations. Inter-annotator reliability, as measured by $\kappa$ and $\alpha$, varied across languages, reflecting both annotation quality and intrinsic task subjectivity.

Figure 1: The world map illustrating languages and regions represented in the polarization detection benchmark.

Task Structure and Competition Organization

The shared task consisted of three subtasks:

• Subtask 1: Binary polarization detection.
• Subtask 2: Classification of polarization type.
• Subtask 3: Identification of manifestation.

Data splits provided development, pilot, and evaluation phases. Participants could submit solutions for any combination of subtasks and languages. Official scoring was by average macro F1, reflecting both balanced label coverage and sensitivity to imbalanced ground truth distributions.

Participation was global, with over 1,000 individuals and 67 teams submitting 73 system description papers. Teams were supported with starter kits, pilot datasets, and communication channels, maximizing inclusivity and research engagement.

Figure 2: Participant diversity across 28 countries underscores the task's global and multicultural scope.

Methods and Model Families

The most popular approaches included model ensembling, parameter-efficient fine-tuning (LoRA, adapters), threshold calibration, and targeted data augmentation. The Qwen, LLaMA, Gemma, Mistral, GPT, and BERT model families predominated. Teams often combined monolingual and multilingual transformer-based encoders with LLMs via soft or weighted ensembling, leveraging out-of-fold logits for ensemble weighting and label-specific threshold optimization.

Loss functions were tailored to address label imbalance (ASL, weighted BCE, Focal Loss) and multi-label subtasks. Data augmentation techniques ranged from back-translation to paraphrasing and explanation generation. These methods demonstrated moderate gains, especially for low-resource languages and underrepresented labels.

System Results and Numerical Analysis

Performance varied significantly by subtask and language:

• Subtask 1 (Binary detection): Peak macro F1 exceeded 0.8 for most languages but declined for low-resource (Khmer, Burmese) and several high-resource (Italian, German) languages.
• Subtask 2 (Type classification): Maximum macro F1 above 0.8 was achieved in only three languages; performance was sharply lower for under-resourced contexts and for the "Other" category, reflecting persistent generalization challenges.
• Subtask 3 (Manifestation identification): Only one language (Urdu) exceeded a macro F1 of 0.8, with most languages below 0.6, indicating substantial difficulty in capturing fine-grained rhetorical tactics.

Best performing teams adopted distinct strategies: UTokyo Tsuruoka Lab leveraged instruction-tuned Gemma models with single-forward-pass token inference, NYCU-NLP employed stacking-based ensembles and multi-label auxiliaries, and SMASH combined monolingual/multilingual encoder transformer ensembles with threshold tuning and out-of-fold calibration. No single approach dominated; competitive scores were achieved via heterogeneous architectures and optimization.

Practical and Theoretical Implications

The results highlight several key implications:

• Cross-lingual and cross-cultural modeling: Generalization remains markedly limited due to cultural-specific manifestations, label distribution imbalances, and local event context dependencies. Even high-performance models showed significant variance across languages and tasks. This underscores the need for more region-specific modeling, targeted data collection, and cultural adaptation in future research.
• Annotation challenges: Despite rigorous protocols, inter-label and inter-annotator agreement were variable, indicating fundamental subjectivity in polarization detection. Improvements in annotation guidelines, cross-cultural training, and psychological support are necessary for sensitive content.
• Algorithmic innovation: PEFT, ensemble weighting, and data augmentation are effective but insufficient for robust generalization. Future work must advance multilingual model architectures, cross-lingual transfer, and hierarchical task conditioning.
• Dataset impact: The public release of the dataset establishes a benchmark for both academic and practical applications—content moderation, digital policy, and peace-building interventions.

Future Directions

Advances in AI for polarization detection necessitate:

• Expansion of dataset size and diversity, especially for monolingual and low-resource languages.
• Robust transfer learning, leveraging cross-lingual embeddings and region-specific fine-tuning.
• Hierarchical modeling strategies to separate event, type, and rhetorical signals.
• Improved loss formulations for handling imbalanced and multiclass labels.
• Ethical protocols to protect annotators in hostile content exposure situations.

Conclusion

SemEval-2026 Task 9 constituted the most popular shared task in its cycle, catalyzing methods and datasets for multilingual, multicultural, and multievent polarization detection (2604.06817). While ensemble techniques, tuned thresholds, and PEFT provided moderate advances, the overall findings expose fundamental challenges in generalization, cultural adaptation, and label imbalances. The resultant benchmark dataset and collaborative participation scaffold future research in computational social science, online moderation, and multilingual NLP, with broad implications for AI-driven discourse analysis.