BanglaMultiHate: Multi-Task Hate Speech Benchmark
- BanglaMultiHate is a multi-task benchmark that annotates hate speech in Bangla along three dimensions: type, severity, and target.
- It is built from over 50,000 filtered YouTube news comments using hierarchical annotation by native speakers.
- Empirical findings demonstrate that culturally grounded models like BanglaBERT outperform generic LLMs in structured hate detection.
Searching arXiv for BanglaMultiHate and closely related Bangla hate-speech resources to ground the article. {"3query3 "3\3 OR 3\3 Multi-Task Bangla Hate Speech Detection: Type, Severity, and Target3\3 BanglaMultiHate is a manually annotated Bangla hate-speech benchmark designed as a multi-task resource rather than a simple hate/non-hate corpus. Each comment is labeled along three coordinated dimensions—type of hate, severity of hate, and target of hate—so that a system can model what kind of hate is expressed, how intense it is, and who or what it is directed at (&&&3query3&&&). The dataset was introduced as the first multi-task Bangla hate-speech dataset and as one of the largest manually annotated Bangla hate-speech corpora, with 53query3,746 filtered comments drawn from Bangla YouTube news discourse (&&&3query3&&&).
3\3. Definition and conceptual scope
BanglaMultiHate was created to address a limitation in earlier Bangla hate-speech research: most prior resources emphasized binary hate/offense detection, or at most one additional label dimension, which constrained both moderation research and fine-grained linguistic analysis (&&&3query3&&&). Its organizing idea is that Bangla hate speech is not adequately captured by a single yes/no label, because moderation decisions often depend on three separate but related judgments: whether the text contains hate, what subtype it instantiates, and what entity is targeted.
The dataset is therefore best understood as a multi-output classification benchmark. It is not a conventional within-task multi-label resource in which one comment receives an arbitrary subset of hate labels. Instead, each comment receives one label per task dimension: one hate-type label, one severity label, and one target label (&&&3query3&&&). This distinction is important because BanglaMultiHate differs structurally from Bangla multi-label corpora such as BOISHOMMO, where comments can receive several simultaneous hate categories, and from BanTH, where hateful transliterated Bangla comments receive one or more target-group labels (Kafi et al., 11 Apr 2025, Haider et al., 2024).
A further defining property is its hierarchical annotation logic. If the type label is None, then severity is automatically Little to None and target is automatically None (&&&3query3&&&). This creates an explicit dependency between tasks and makes the benchmark closer to operational moderation logic than a collection of independent flat labels.
3 OR \3. Corpus construction and topical coverage
The corpus was collected from YouTube via the YouTube API, primarily from Somoy TV, a major Bangla news channel (&&&3query3&&&). The raw crawl yielded approximately 55,3query3query3query3^ comments associated with Bangla news videos spanning 3\39 categories and 3\3 OR \3query3^ sub-topics. The 3\39 top-level categories are Business, Celebrities, Disaster, Entertainment, Fashion, Geopolitics, Health, History, International, Lifestyle, Literature, Miscellaneous, National, Opinion, Politics, Religion, Science, Sports, and Technology (&&&3query3&&&).
The dataset was filtered before annotation. The reported filtering operations removed comments containing only emojis, removed URLs, removed duplicate entries, and excluded Banglish comments, meaning Bangla written in the English alphabet (&&&3query3&&&). After filtering, the final corpus size became 53query3,746 comments (&&&3query3&&&).
This construction yields a specific domain profile. BanglaMultiHate is broad in topic coverage, but narrow in platform and genre, because it centers on Bangla YouTube news comments rather than general social media, private messaging, or cross-platform discourse. The exclusion of Banglish also makes it complementary to transliterated resources such as BanTH rather than a substitute for them (Haider et al., 2024). Most comments are short: the paper notes that the most common length bucket is PRESERVED_PLACEHOLDER_3query3^ words, which is consistent with YouTube-comment discourse (&&&3query3&&&).
3. Annotation framework and reliability
Annotation was performed by 35 native Bangla-speaking undergraduate students, including both male and female annotators, under expert supervision (&&&3query3&&&). Each comment was independently annotated by three annotators. The workflow combined explicit guidelines, periodic quality checks on randomly selected samples, feedback sessions, majority agreement, and consensus meetings for unresolved cases (&&&3query3&&&).
The three task dimensions are as follows. Type of Hate has six classes: Abusive, Sexism, Religious Hate, Political Hate, Profane, and None. Severity of Hate has three classes: Little to None, Mild, and Severe. Target of Hate has five classes: Individual, Organization, Community, Society, and None (&&&3query3&&&).
The paper reports Fleiss’ Kappa for each task: 3query3.73\3 for type, 3query3.84 for severity, and 3query3.79 for target (&&&3query3&&&). Using the interpretation cited in the paper, type and target show substantial agreement, while severity reaches almost perfect agreement. The relative ordering is informative. Type is the hardest annotation dimension, plausibly because six-way hate-type distinctions require finer semantic judgments than deciding whether a comment is mild or severe, while target requires inference about the social object of hostility.
The annotation design also clarifies a common misconception. BanglaMultiHate is “multi-task,” but not “multi-label” in the same sense as BOISHOMMO or BanTH. A comment does not receive several hate-type labels simultaneously; instead it receives one type label, one severity label, and one target label (&&&3query3&&&, Kafi et al., 11 Apr 2025, Haider et al., 2024).
4. Label space and statistical profile
BanglaMultiHate is strongly imbalanced across all three tasks. The imbalance is especially visible in the dominance of None and in the rarity of Sexism and Religious Hate within the type task (&&&3query3&&&).
| Dimension | Labels | Total counts |
|---|---|---|
| Type | None, Abusive, Political Hate, Profane, Religious Hate, Sexism | 3 OR \38,63query33; 3\3\3,637; 6,3query3 OR \3\3; 3,383 OR \3; 933; 3\373query3 |
| Severity | Little to None, Mild, Severe | 33,643; 9,763; 7,343query3^ |
| Target | None, Individual, Organization, Community, Society | 33query3,347; 7,973 OR \3; 5,583 OR \3; 3,733 OR \3; 3,3\3\33^ |
The dataset split is 73query3% train, 3\3query3% development, and 3 OR \3query3% test, with exact sizes 35,53 OR \3 OR \3^ / 5,3query3 OR \34 / 3\3query3,3 OR \3query3query3^ produced by stratified sampling (&&&3query3&&&). Within the type task, the train split contains 8,3 OR \3\3 OR \3^ Abusive, 4,3 OR \3 OR \37 Political Hate, 3 OR \3,333\3^ Profane, 676 Religious Hate, 3\3 OR \3 OR \3^ Sexism, and 3\39,954 None labels (&&&3query3&&&). For severity, the train split contains 3 OR \33,489 Little to None, 6,853 Mild, and 5,3\383query3 Severe labels. For target, it contains 3 OR \3\3,3\39 None, 5,646 Individual, 3,846 Organization, 3 OR \3,635 Community, and 3 OR \3,3 OR \3query35 Society labels (&&&3query3&&&).
The corpus also includes cross-task interaction analyses. The paper reports that Abusive is the most common hateful type and peaks at mild severity, Profane is concentrated in severe cases, and Individuals and organizations are the primary targets overall, with Abusive especially concentrated on individuals (&&&3query3&&&). These interactions make BanglaMultiHate more than a flat label inventory; they enable study of how hate subtype, intensity, and target co-vary in Bangla news-comment discourse.
5. Benchmarking, shared-task use, and empirical findings
The benchmark study compares majority and random baselines, a classical SVM with TF-IDF 3\3–5-grams and PRESERVED_PLACEHOLDER_3\3^, a monolingual BanglaBERT, and instruction-tuned LLMs in both zero-shot and LoRA-fine-tuned form (&&&3query3&&&). The LLMs are Llama-3.3 OR \3-3B-Instruct and Qwen3-4B-Instruct-3 OR \353query37. The LoRA configuration uses Adam, FP3\36, learning rate PRESERVED_PLACEHOLDER_3 OR \3^, , rank , maximum sequence length 53\3 OR \3^, batch size 8, and 3 epochs (&&&3query3&&&). BanglaBERT is fine-tuned for 3 epochs, with 3\3query3^ runs using different random seeds, and the best development model is selected (&&&3query3&&&).
Across all three tasks, BanglaBERT is the strongest model in the paper. It achieves 3query3.73\3 OR \3^ micro-F3\3^ for hate type, 3query3.73 OR \3 OR \3^ micro-F3\3^ for severity, and 3query3.73\3 micro-F3\3^ for target (&&&3query3&&&). The SVM remains competitive, reaching 3query3.63query3, 3query3.673 OR \3^, and 3query3.63 OR \39 micro-F3\3^ on type, severity, and target respectively (&&&3query3&&&). Zero-shot LLM performance is much weaker: for example, zero-shot Llama-3.3 OR \3-3B-Instruct obtains 3query3.3 OR \375 on type and 3query3.343query3 on target, while zero-shot Qwen3 obtains 3query3.53 OR \3query3^ on type and 3query3.434 on target (&&&3query3&&&). LoRA substantially improves both models, with Llama-3.3 OR \3-3B-Instruct rising to 3query3.63 OR \3query3^, 3query3.685, and 3query3.63\3query3 micro-F3\3^ on type, severity, and target, but still trailing BanglaBERT (&&&3query3&&&).
The paper’s principal empirical conclusion is that culturally and linguistically grounded pretraining remains critical for Bangla hate-speech analysis (&&&3query3&&&). In other words, adaptation helps general LLMs, but does not erase the advantage of a Bangla-specific encoder.
BanglaMultiHate also became the basis of the BLP-3 OR \3query3 OR \35 Task 3\3: Bangla Multi-task Hate Speech Identification shared-task ecosystem. A system paper by the Retriv team explicitly cites BanglaMultiHate as “the first multi-task Bangla hate speech dataset jointly modeling type, severity, and target” and reports official blind-test shared-task scores of 73 OR \3.75% micro-F3\3^ for hate-type classification, 73 OR \3.69% micro-F3\3^ for target-group identification, and 73 OR \3.63 OR \3% weighted micro-F3\3^ for joint detection (Saha et al., 10 Nov 2025). Another shared-task paper, by Gradient Masters, reports 73.3 OR \33% micro-F3\3^ on type classification and 73.3 OR \38% on target classification, with leaderboard positions of 6th and 3rd respectively (Hoque et al., 23 Nov 2025). Shared-task system papers further indicate that the original 5,3query3 OR \34-instance development portion was operationalized as a 3 OR \3,53\3 OR \3^ public Dev set plus a 3 OR \3,53\3 OR \3^ blind Dev Test split in competition settings (Saha et al., 10 Nov 2025, Hoque et al., 23 Nov 2025).
6. Relation to earlier Bangla hate-speech resources
BanglaMultiHate occupies a distinct position within Bangla hate-speech research because earlier datasets generally emphasized different task formulations. The 33query3,3query3query3query3 dataset of “Hate Speech detection in the Bengali language” is a broad binary benchmark with hate vs not-hate labels and seven source-topic categories that function only as metadata, not as hate subclasses (Romim et al., 2020). HS-BAN is also binary, although it is comparatively strong in annotation criteria and agreement reporting, with 53query3,33\3 comments and Fleiss’ kappa = 3query3.658 (Romim et al., 2021). BD-SHS introduces a hierarchical design with hate/non-hate, target, and type, but its type inventory is narrower and its structure differs from BanglaMultiHate’s explicit type–severity–target triad (Romim et al., 2022).
At the other end of the design spectrum are corpora that are multi-label within a single task. BOISHOMMO contains 3 OR \3,499 Facebook comments with overlapping labels such as Race, Behaviour, Physical, Class, Religion, Disability, Nationality/Ethnicity, Gender, Sexual Orientation, and Political Statement (Kafi et al., 11 Apr 2025). BanTH contains 37,353query3^ transliterated Bangla YouTube comments, with binary hate labeling followed by one-or-more target-group labels such as Political, Religious, Gender, Personal Offense, Abusive/Violence, Origin, and Body Shaming (Haider et al., 2024). BanglaMultiHate differs from both by remaining in native Bangla script, excluding Banglish, and using one label per dimension rather than within-dimension multi-label assignment (&&&3query3&&&).
This positioning makes BanglaMultiHate especially useful when the research objective is structured moderation reasoning rather than only hate detection or target attribution. It is broader than a binary detector, but more constrained and operationally cleaner than a free-form multi-label ontology.
7. Limitations, interpretation, and significance
Several limitations are explicit in the dataset paper. BanglaMultiHate is drawn from a single platform genre—YouTube news comments, primarily from Somoy TV—so cross-domain generalization is not guaranteed (&&&3query3&&&). It excludes Banglish/transliterated Bangla, which strengthens native-script consistency but limits direct use for mixed-script moderation (&&&3query3&&&). The class imbalance is severe, most notably Sexism = 3\373query3 and Religious Hate = 933 within the full type inventory, while None dominates all three tasks (&&&3query3&&&). The paper also notes that the dataset does not yet include reasoning annotations, and that hateful material poses obvious annotator-exposure and subjectivity risks (&&&3query3&&&).
These constraints shape how the benchmark should be interpreted. High micro-F3\3^ can coexist with weak minority-class behavior; this is one reason the paper itself notes the need for richer reporting such as macro-F3\3^ and per-class analysis in future evaluation (&&&3query3&&&). A plausible implication is that the rarest labels—especially Sexism, Religious Hate, and Society—will remain bottlenecks unless future versions expand those classes or use targeted collection.
Despite these limitations, BanglaMultiHate marks a substantial shift in Bangla hate-speech benchmarking. It formalizes hate analysis as a coordinated prediction problem over type, severity, and target; it provides a large manually annotated native-script corpus; and it demonstrates that, in Bangla, monolingual culturally grounded pretraining still outperforms generic zero-shot or lightly adapted LLMs (&&&3query3&&&). In the Bangla hate-speech literature, its importance lies less in novelty of model architecture than in establishing a benchmark where the structure of hateful expression is itself the object of study.