BRIGHTER Multilingual Emotion Dataset

• BRIGHTER is a multilingual dataset for textual emotion recognition that includes nearly 100,000 annotated instances and covers 28 languages, emphasizing low-resource regions.
• It unifies diverse corpora under a shared annotation schema for multi-label classification, allowing both intensity prediction and crosslingual performance evaluation.
• The benchmark offers varied evaluation settings, highlighting challenges in multilingual representation and cultural variations in emotion perception.

BRIGHTER is a multilingual collection of human-annotated textual emotion datasets covering 28 languages and nearly 100,000 instances, designed for multi-label perceived emotion recognition and emotion intensity prediction, with a particular emphasis on predominantly low-resource languages from Africa, Asia, Eastern Europe, and Latin America (Muhammad et al., 17 Feb 2025). The resource unifies heterogeneous local corpora under a shared annotation schema, combines social media and non-social-media domains, and provides benchmark settings for monolingual supervised transfer, family-based crosslingual transfer, and few-shot LLM prompting. Its central technical role is to make multilingual emotion analysis comparable across languages that differ sharply in corpus size, label sparsity, domain composition, and representation in multilingual pretraining.

1. Nomenclature and conceptual scope

The term BRIGHTER, as given in the arXiv title and abstract, refers to a dataset family for human-annotated textual emotion recognition across 28 languages (Muhammad et al., 17 Feb 2025). The available descriptions contain a naming inconsistency: one detailed summary refers to the same resource as “black.” This suggests that the cited record and that summary refer to the same multilingual dataset collection, rather than to distinct resources. In either case, the object described is a multilingual benchmark for textual emotion analysis.

The name should be distinguished from unrelated “bright” or “brighter” resources in other domains, including the Gaia DR2 bright-star StarHorse catalog (Anders et al., 2019), the BRIGHT multimodal building-damage dataset (Chen et al., 10 Jan 2025), and the BRI3L brightness-illusion image dataset (Roy et al., 2024). Within NLP, BRIGHTER is not a photometric, remote-sensing, or vision benchmark; it is a text dataset for perceived-emotion labeling.

Conceptually, BRIGHTER addresses a specific asymmetry in multilingual NLP: emotion recognition work has concentrated on high-resource languages, whereas many low-resource languages lack human-annotated datasets with consistent label semantics. The collection therefore emphasizes local data acquisition and annotation by fluent speakers, rather than a translation-only paradigm. This design choice is important because the paper argues that cultural and linguistic differences affect how emotions are expressed and perceived, limiting direct transfer from translated resources (Muhammad et al., 17 Feb 2025).

2. Linguistic coverage and corpus composition

BRIGHTER includes Afrikaans, Algerian Arabic, Moroccan Arabic, Chinese, German, English, Latin American Spanish, Hausa, Hindi, Igbo, Indonesian, Javanese, Kinyarwanda, Marathi, Nigerian Pidgin, Brazilian Portuguese, Mozambican Portuguese, Romanian, Russian, Sundanese, Swahili, Swedish, Tatar, Ukrainian, Emakhuwa, isiXhosa, Yoruba, and isiZulu. The paper states that these languages span 7 language families and repeatedly characterizes the geographic coverage as centered on Africa, Asia, Eastern Europe, and Latin America (Muhammad et al., 17 Feb 2025).

The collection is explicitly multi-domain rather than domain-uniform. Sources include social media posts, personal narratives, talks and speeches, literary text, news, and human-written plus machine-generated text. Social-media sources include Reddit, YouTube, Twitter, and Weibo. Several corpora were re-annotated from preexisting sentiment datasets, including AfriSenti data for Moroccan Arabic, Hausa, and Kinyarwanda, as well as a Ukrainian Twitter dataset. Hindi and Marathi were partly created from scratch and supplemented with manually corrected translated material and a few hundred quality-approved ChatGPT-generated examples.

The domain structure varies by language. Single-domain languages include Afrikaans, Algerian Arabic, Chinese, German, English, Latin American Spanish, Indonesian, Javanese, Marathi, Brazilian Portuguese, Romanian, Russian, Sundanese, Swedish, Tatar, Ukrainian, and Yoruba. Multi-domain languages include Moroccan Arabic, Hausa, Igbo, Kinyarwanda, Nigerian Pidgin, Mozambican Portuguese, Swahili, Emakhuwa, isiXhosa, and isiZulu. Hindi is marked simply as created. This heterogeneity is methodologically important because performance differences across languages are not reducible to language alone; they are partly confounded by domain composition and label distribution (Muhammad et al., 17 Feb 2025).

Per-language sizes vary substantially. The largest reported dataset is Nigerian Pidgin with 6,218 instances, followed by English with 5,651, Swahili with 5,514, Chinese with 5,484, and German with 5,407. The smallest are Javanese with 988 and Indonesian with 1,007. Four languages—Indonesian, Javanese, isiXhosa, and isiZulu—have no training split and are used only for testing. This makes the benchmark structurally asymmetric: some languages support monolingual supervised learning, while others are available only for zero-shot or few-shot evaluation (Muhammad et al., 17 Feb 2025).

3. Annotation schema, label semantics, and aggregation

The shared label schema annotates each text for the presence of anger, sadness, fear, disgust, joy, surprise, and neutral/none. The task is defined in terms of perceived emotion: annotators were asked to identify what emotion most people would think the speaker may have felt, rather than the inaccessible “true” internal state. Because emotions may co-occur, the collection is fundamentally multi-label. Two practical exceptions are noted: English does not include disgust, and Afrikaans does not include surprise, because those classes were too underrepresented. The released data are nevertheless described as multi-label across the collection, with the important caveat that Tatar is single-labeled and Marathi and Hindi contain large proportions of single-label instances (Muhammad et al., 17 Feb 2025).

Emotion intensity is represented on a four-point ordinal scale:

• $0 =$ no emotion
• $1 =$ low/slight
• $2 =$ moderate
• $3 =$ high

The paper gives the final-label aggregation rule explicitly. An emotion is present in the final annotation if at least two annotators selected intensities $1$, $2$, or $3$, and the average score exceeds a threshold $T=0.5$:

$$L_{\text{final}} = \begin{cases} 1, & \text{if } \mathrm{Count}(1,2,3)\ge 2 \text{ and } \mathrm{AvgScore}>T,\ 0, & \text{otherwise.} \end{cases}$$

with

$$\mathrm{Count}(1,2,3)=\sum_{i=1}^{N}\mathbb{1}(A_i\in\{1,2,3\}), \qquad \mathrm{AvgScore}=\frac{1}{N}\sum_{i=1}^{N}A_i.$$

For intensity, once an emotion is deemed present, the final intensity is obtained by averaging annotator scores and discretizing:

$1 =$0

The body text also describes this procedure as averaging and taking the ceiling. Intensity labels are released only for datasets where most instances had at least 5 annotators, yielding 10 intensity datasets (Muhammad et al., 17 Feb 2025).

A distinctive design decision is that the release includes not only aggregated labels but also the individual raw annotations. This is consequential for downstream work on disagreement modeling, annotator variance, distributional supervision, and culture-sensitive uncertainty.

4. Annotators, preprocessing, and reliability measurement

Annotation was carried out by fluent speakers and, in many cases, by native speakers recruited directly by the language teams. Amazon Mechanical Turk was used for English, while Toloka was used for Russian, Ukrainian, and Tatar. For low-resource languages, the teams recruited speakers directly and used Label Studio and Potato as annotation platforms. Annotator counts vary from 3 annotators total for many African datasets to 122 for English and 106 for Ukrainian. The number of annotations per sample also varies, often 3 per sample, but reaching 5 for Chinese, Spanish, Hausa, and the English minimum, 4 to 9 for Algerian Arabic, and up to 30 for some English instances (Muhammad et al., 17 Feb 2025).

Before annotation, the creators removed duplicates, invisible characters, garbled encoding, and incorrectly rendered emoticons; anonymized all texts; and excluded content with excessive expletives or dehumanizing language. The paper does not describe tokenization, lemmatization, or language-specific normalization beyond these steps, so such preprocessing should not be assumed.

Reliability is reported with SHCMP, or Split-Half Class Match Percentage, rather than with conventional kappa statistics. The appendix defines

$1 =$1

and

$1 =$2

with scores averaged over 1,000 random repetitions. The paper states that SHCMP values are generally high, above 60% for $1 =$3, and interprets this as evidence of reliable annotation (Muhammad et al., 17 Feb 2025).

The quality-control regime includes pilot annotation rounds to refine guidelines, data cleaning before annotation, reliability analysis after annotation, and conservative aggregation rules for final labels. Annotators were paid more than minimum wage per hour. These choices indicate that BRIGHTER was designed not only as a benchmark but also as a resource for studying annotation subjectivity under multilingual, low-resource conditions.

5. Benchmark tasks, models, and empirical results

BRIGHTER supports two main benchmark tasks: multi-label emotion classification and emotion intensity prediction. Multi-label classification is evaluated in few-shot LLM settings and in crosslingual MLM transfer settings; the appendix also reports monolingual supervised MLM baselines for the 24 languages with training data. Intensity prediction is evaluated on the 10 languages with robust intensity annotations (Muhammad et al., 17 Feb 2025).

For few-shot multi-label classification, the evaluated LLMs are Qwen2.5-72B, Dolly-v2-12B, Llama-3.3-70B, Mixtral-8x7B, and DeepSeek-R1-70B. Prompts use chain-of-thought with 8 few-shot examples, and results are based on the top-1 generated answer. For crosslingual transfer, the multilingual encoder baselines are LaBSE, RemBERT, XLM-R, mBERT, and mDeBERTa. The appendix states that MLMs were trained for 2 epochs with learning rate $1 =$4, while LLMs used HuggingFace defaults except temperature $1 =$5 and top-$1 =$6 in the standard setting. Classification is evaluated with macro F1, and intensity prediction with Pearson correlation; the paper does not print formulas for either metric.

The headline benchmark outcome is that multilingual emotion recognition remains difficult, especially for low-resource languages. In few-shot LLM classification, average macro-F1 is highest for Qwen2.5-72B at $1 =$7, followed by DeepSeek-R1-70B at $1 =$8 and Llama-3.3-70B at $1 =$9; Dolly-v2-12B trails at $2 =$0. High scores tend to occur in languages with many single-label examples: Hindi reaches $2 =$1 with Qwen, Marathi $2 =$2 with DeepSeek, Russian $2 =$3 with DeepSeek, Spanish $2 =$4 with DeepSeek, and Romanian $2 =$5 with Llama. By contrast, Yoruba peaks at only $2 =$6, Emakhuwa around $2 =$7, and isiZulu around $2 =$8. The paper explicitly links some easier cases to label structure: Tatar performs well partly because it is single-labeled, while roughly 70% of Marathi and 80% of Hindi test data are single-labeled (Muhammad et al., 17 Feb 2025).

In monolingual supervised MLM classification, RemBERT is strongest on average in many languages, including English $2 =$9, Spanish $3 =$0, Hindi $3 =$1, Marathi $3 =$2, Romanian $3 =$3, and Russian $3 =$4. Low-resource languages remain difficult even here: Swahili reaches $3 =$5, Emakhuwa $3 =$6, and Yoruba $3 =$7 with RemBERT. In crosslingual transfer, LaBSE achieves the best average macro-F1 at $3 =$8, ahead of RemBERT at $3 =$9 and mDeBERTa at $1$0. Some targets benefit substantially from family-based transfer, including Marathi $1$1 with RemBERT, Romanian $1$2, Russian $1$3, Hindi $1$4, Chinese $1$5 with XLM-R, and Swedish $1$6 with RemBERT.

For emotion intensity prediction, evaluated on Algerian Arabic, Chinese, German, English, Spanish, Hausa, Brazilian Portuguese, Romanian, Russian, and Ukrainian, RemBERT is the strongest encoder on average at $1$7, while DeepSeek-R1-70B is the best LLM on average at $1$8. One of the more striking reported cases is Algerian Arabic, where DeepSeek reaches $1$9 while the best MLM score is only $2$0. This suggests that, in some vernacular low-resource settings, LLMs may recover ordinal emotion strength more effectively than MLM encoders, although performance remains highly variable across languages and domains (Muhammad et al., 17 Feb 2025).

The paper also reports prompt and decoding sensitivity. On English, performance generally improves with more shots and tends to plateau around 4 to 8 shots. In a top-$2$1 generation ablation, DeepSeek-R1-70B exceeds 90 F-score on English at top-$2$2, indicating that the correct label set is often among likely generations even when top-1 decoding fails. Another notable finding is that, for many low-resource languages, prompting in English performs better than prompting in the target language; the cited exceptions include Algerian Arabic, where Qwen performs better when prompted in MSA.

6. Limitations, ethics, and research significance

BRIGHTER is explicit about several limitations. The annotations capture perceived emotions, not speakers’ true emotions. The collection is not claimed to be fully representative for any language. Domain coverage is uneven, class distributions vary sharply, and some datasets are dominated by single-label or near-single-label instances. English lacks disgust and Afrikaans lacks surprise because of class sparsity. Several languages have modest dataset sizes, and four languages are test-only. These properties make the collection simultaneously valuable and methodologically non-uniform (Muhammad et al., 17 Feb 2025).

The benchmark results reinforce a second limitation: current multilingual pretraining transfers unevenly. Languages better represented in pretraining corpora consistently perform better, while several Niger-Congo languages remain difficult for both MLMs and LLMs. The paper notes, for example, that mDeBERTa performs poorly on Igbo, Emakhuwa, and Yoruba because they are not in the CC-100 corpus used for its training. This suggests that BRIGHTER is at least as much a stress test for multilingual representation learning as it is a pure dataset release.

The ethics statement imposes explicit usage restrictions. Commercial use is forbidden, as is use by state actors for high-risk applications unless specifically approved. The paper further warns against using systems trained on the data for critical individual-level decisions such as healthcare without expert supervision. These restrictions are stricter than a conventional permissive benchmark release. The collection is publicly released via the project GitHub linked in the paper, but a conventional open-source license is not explicitly stated in the provided text (Muhammad et al., 17 Feb 2025).

Within multilingual emotion recognition, BRIGHTER’s significance lies in the combination of shared annotation design, local-language human annotation, multi-label emotion structure, and emotion intensity labels across 28 languages. The paper contrasts this design with prior multilingual resources that rely heavily on translation and often assume single-label classification. A plausible implication is that BRIGHTER is best understood not merely as a leaderboard benchmark, but as a dataset for investigating how annotation subjectivity, domain variation, label multiplicity, and pretraining inequality jointly shape multilingual emotion modeling.