BOISHOMMO: Bangla Multi-Label Hate Speech Dataset

• BOISHOMMO is a multi-label Bangla hate-speech dataset capturing complex, overlapping abusive content across 10 hate categories.
• It comprises 2,499 annotated Facebook comments from leading Bangla newspapers during 2020–21, ensuring diverse linguistic challenges.
• Baseline experiments show Random Forest achieving 86.0% Macro-F1, highlighting effective performance in low-resource, multi-label settings.

BOISHOMMO is a multi-label Bangla hate-speech dataset and experimental benchmark built for low-resource-language analysis of abusive online discourse. It comprises 2,499 Facebook comments collected from the pages of Prothom Alo, Jugantor, and Kaler Kantho during 2020–21, and it is explicitly designed around the premise that hate speech has no single dimensionality: a single comment may simultaneously target multiple protected attributes. The accompanying study evaluates classical multi-label classifiers over Bangla-specific preprocessing and reports Random Forest as the strongest baseline, with 86.0% Macro-F1 (Kafi et al., 11 Apr 2025).

1. Corpus construction and annotation protocol

BOISHOMMO was assembled from user comments on Facebook posts published by three leading Bangla newspapers—Prothom Alo, Jugantor, and Kaler Kantho—during 2020–21. To capture a wide variety of hate expressions, the collection process focused on the most-commented news posts in that period and extracted comments containing potentially abusive or discriminatory language (Kafi et al., 11 Apr 2025).

The annotation design is multi-label. Inspired by Fortuna & Nunes (2018), the study defines ten coarse-grained hate categories: Race, Behaviour, Physical, Class, Religion, Disability, Nationality, Gender, Sexual Orientation, and Political Statement. Each comment is labeled with a binary vector $y \in \{0,1\}^{10}$, with multiple positive entries permitted when a comment targets more than one attribute. Three native-Bangla annotators, each with at least two years of undergraduate study, independently labeled all 2,499 comments.

Inter-annotator agreement was quantified with pairwise Cohen’s $\kappa$: $\kappa \;=\;\frac{P_o - P_e}{1 - P_e}\,.$ The reported sample $\kappa$ values illustrate substantial variation across categories. For Race, the three annotator pairs obtained $(0.33, 0.16, 0.68)$; for Religion, $(0.71, 0.61, 0.61)$; and for Class, $(0.37, 0.07, 0.16)$. Categories such as Religion and Sexual Orientation reached “substantial” agreement, whereas Class and Race were more subjective. Final labels were obtained by majority voting: for a given category, the label is 1 if at least two annotators assigned 1, and 0 otherwise.

This annotation protocol formalizes a central claim of the BOISHOMMO study: hate speech should not be reduced to a one-label taxonomy when multiple abusive dimensions can co-occur in the same utterance.

2. Label space, scale, and distributional characteristics

The corpus contains 2,499 examples and is split into train, validation, and test partitions at 80% / 10% / 10%, corresponding to 1,999 / 250 / 250 comments (Kafi et al., 11 Apr 2025).

Split	Size
Train	1,999
Validation	250
Test	250

The reported label fractions are Behaviour 19.3%, Religion 16.3%, Nationality 13.3%, Political Statement 10.6%, Race 10.4%, Ethnicity 9.1%, Class 8.2%, Gender 7.7%, Sexual Orientation 6.8%, and Disability with too few examples for modeling. Most comments carry between one and three simultaneous labels.

A notable reporting issue appears in the summarized description. The annotation schema enumerates Physical as one of the ten coarse-grained categories, whereas the distributional summary reports Ethnicity at 9.1% and does not separately list Physical. This suggests a reporting discrepancy in the summarized statistics. The category-wise performance summary later reports results for Physical, which reinforces that the multi-label space is intended to cover more than a single dominant abusive axis.

In substantive terms, the distribution shows both label imbalance and overlap. Behaviour and Religion are comparatively frequent, while Disability is sparse enough to be excluded from modeling. This matters methodologically because low-frequency labels can distort both training dynamics and aggregate evaluation in multi-label settings.

3. Bangla-specific preprocessing and feature representation

The preprocessing pipeline is classical and explicitly adapted to Bangla text. Unicode normalization converts all text to NFC form in order to unify code points. Punctuation removal strips common punctuation and extraneous symbols. Stopword removal uses a predefined Bangla stopword list, with examples including “করি,” “হবে,” and “কি.” Tokenization is whitespace-based, followed by removal of tokens shorter than two characters. Stemming is performed with the rule-based “Bangla Stemmer” of Mahmud et al. (2014), and the final representation is a TF–IDF feature space produced with scikit-learn’s TfidfVectorizer (Kafi et al., 11 Apr 2025).

The paper situates these steps within a set of language-specific difficulties: Bangla uses a non-Latin script with complex Unicode representations and diacritics; it exhibits rich morphology and compounding; and hate-speech comments may include code-switching and slang. The study also notes the scarcity of large, high-quality lexica and pretrained embeddings. Within that context, the preprocessing design is best understood as an attempt to stabilize sparse lexical features before multi-label classification rather than as a contextual or semantic modeling framework.

4. Multi-label learning formulation and baseline models

BOISHOMMO evaluates three classical algorithms under scikit-learn’s MultiOutputClassifier, which converts the task into label-wise supervised learning: for each label $j \in \{1,\dots,10\}$, a separate estimator $f_j$ is trained on $(X, y_j)$, and prediction concatenates the outputs across labels (Kafi et al., 11 Apr 2025).

The three base estimators are as follows.

• Random Forest: RandomForestClassifier(n_estimators=100, criterion='gini'), with max_depth=None. Each label-specific classifier is an ensemble of 100 decision trees. Node splitting is governed by Gini impurity,

$\kappa$0

• Support Vector Machine: SVC(kernel='sigmoid', decision_function_shape='ovr', break_ties=True), with $\kappa$1, $\kappa$2, and $C=1.0`. The kernel is</li> </ul> <p>$\kappa$3</p> <p>and the multi-label decomposition uses a One-vs-Rest strategy.</p> <ul> <li><strong>Logistic Regression</strong>: <code>LogisticRegression(penalty=&#39;l2&#39;, C=1.0)</code>, with <code>solver=&#39;lbfgs&#39;</code> and <code>max_iter=1000</code>. The conditional model is</li> </ul> <p>$\kappa$4

  These baselines are deliberately conventional. They test how far a multi-label Bangla hate-speech benchmark can be pushed with TF–IDF features and standard discriminative learners before introducing contextual encoders or transformer-based architectures.

  5. Evaluation framework and empirical findings

  The methodological summary defines the standard multi-label metrics Precision, Recall, F1, Hamming Loss, and Subset Accuracy, while the reported headline comparison uses Macro-averaged F1 (Kafi et al., 11 Apr 2025). For label $\kappa$5,

  $\kappa$6

  $\kappa$7

  Macro-averaging across $\kappa$8 labels is defined by

  $\kappa$9

  $\kappa \;=\;\frac{P_o - P_e}{1 - P_e}\,.$0

  The aggregate results are:

  Model	Macro-F1
  Random Forest	86.0%
  SVM	81.4%
  Logistic Regression	60.0%

  Category-wise F1 scores are also reported for several labels. For Religion, the scores are RF 95%, SVM 92%, LR 75%; for Race, RF 93%, SVM 90%, LR 78%; for Physical, RF 90%, SVM 88%, LR 72%; for Gender, RF 85%, SVM 82%, LR 65%; and for Political, RF 78%, SVM 75%, LR 58%.

  The study’s interpretation is explicit. Random Forest consistently leads, especially for categories with more training examples such as Religion, Race, and Physical. Its ensemble structure is described as capturing non-linear feature interactions in TF–IDF space. SVM is competitive but slightly weaker in low-data categories, while Logistic Regression underfits the nuanced, high-dimensional feature space because of its linear decision boundaries.

  6. Limitations, interpretive issues, and subsequent research directions

  The paper identifies several limitations that are integral to understanding BOISHOMMO as a benchmark rather than a solved task (Kafi et al., 11 Apr 2025). First, the dataset is affected by label imbalance; Disability is so sparse that it is dropped from modeling. Second, annotation remains partly subjective, as reflected in the low $\kappa \;=\;\frac{P_o - P_e}{1 - P_e}\,.$1 values for categories such as Race and Class. Third, Bangla hate-speech detection is complicated by non-Latin orthography, rich morphology, compounding, code-switching, and slang. These are not incidental nuisances; they directly affect tokenization, normalization, and lexical sparsity.

  A common misconception in hate-speech modeling is that detection can be reduced to a single binary or single-label decision. BOISHOMMO is structured against that assumption. Its multi-label design encodes the possibility that a comment may simultaneously attack race, religion, gender, political identity, or other attributes. The practical implication is that both annotation and evaluation must accommodate overlapping targets rather than force mutually exclusive categories.

  The study outlines several future directions. It proposes dataset expansion with more comments, including both benign and hateful material, especially to improve coverage of under-represented labels such as Disability. It recommends deep-learning experiments with Bangla-specific transformer models such as BanglaBERT and multilingual BERT, as well as CNN/BiLSTM + attention architectures. It also identifies multi-task and transfer learning, cross-lingual methods using aligned embeddings or adapters, and context modeling with surrounding post text and user metadata as promising extensions.

  Taken together, BOISHOMMO occupies a specific place in the Bangla NLP landscape: it is a multi-label hate-speech resource that couples corpus construction, explicit annotation formalism, and baseline evaluation, while also exposing the unresolved problems of low-resource moderation, label overlap, and language-specific preprocessing.