ETHOS Hate Speech Classification

• ETHOS hate speech classification is a benchmark that annotates social media comments for hate against protected groups with precise, multi-dimensional labels.
• The methodology leverages advanced text representations, including static BERT embeddings and transfer learning, to enhance accuracy and specificity.
• Experiments using CNNs, BiLSTMs, and zero-shot NLI approaches demonstrate significant gains in recall and F1-score, reducing false positives in moderation.

ETHOS hate speech classification refers to the application of statistical and neural models to the ETHOS dataset, a curated benchmark for automated hate speech detection encompassing both binary and multi-label variants. ETHOS annotates social media (YouTube, Reddit) comments for the presence of hate directed against protected group characteristics across categories such as gender, race, nationality, disability, sexual orientation, and religion. The classification task is central to understanding, modeling, and mitigating the proliferation of online hate, and recent work has focused on embedding strategies, model architectures, transfer learning, and the operationalization of nuanced discursive dimensions such as Ethos in annotation and downstream systems.

1. ETHOS Dataset: Structure, Annotation, and Statistics

The ETHOS dataset comprises two principal variants: a binary labeled set of 998 short-form social media comments (433 hate, 565 non-hate), and a multi-label subset in which each of the 433 hate-marked samples is further annotated for violence, directed-vs-generalized target, and six group-based hate categories (gender, race, national origin, disability, sexual orientation, religion) (Mollas et al., 2020).

The dataset construction used a hybrid active sampling protocol—uncertainty and extremity-based selection from a large initial pool, with iterative model-in-the-loop curation and crowdsourced validation. Each comment received five independent annotations on the Figure-Eight platform, and inter-annotator agreement is exceptionally high, with Fleiss’ κ > 0.90 for all protected characteristics. This high consistency is notable given persistent subjectivity in hate-speech annotation. The class distribution is approximately balanced (hate: 43.4%, non-hate: 56.6%) and this is substantiated by normalized Shannon entropy, $\mathrm{Balance} \approx 0.986$ for $k=2$ classes (Rajput et al., 2021).

2. Pre-processing and Text Representation Strategies

ETHOS classification pipelines apply both fundamental text normalization (lowercasing, punctuation removal) and advanced tokenization strategies. Early pipelines used BERT’s WordPiece tokenizer for subword decomposition, resolving out-of-vocabulary issues by splitting unknown words and mean-pooling subword vectors. Downstream, static word embeddings—fastText (FT), GloVe (GV), or their concatenation—serve as input to neural architectures, but recent work demonstrates a marked gain from static BERT embeddings (BE), where each word type’s vector is the mean of all its contextual BERT encodings over the corpus (Rajput et al., 2021).

The embedding matrix $E \in \mathbb{R}^{V \times d}$ is instantiated for each method ($d=768$ for BE, $d\in[300,600]$ for FT/GV), then frozen prior to model training. This shift from standard static embeddings to corpus-averaged BERT vectors directly increases the specificity and overall classification robustness, even relative to fine-tuned BERT (Rajput et al., 2021).

3. Neural and Transfer-Learning Architectures

The prevailing neural models for ETHOS binary and multi-label classification include CNNs with attention, LSTMs, bidirectional LSTMs (BiLSTM), and GRUs. Input sequences are mapped to embedding matrices and processed via either convolutional or recurrent blocks, with attention layers applied in some CNNs to capture contextual dependencies. Model outputs are fed to fully connected layers, generally with two neurons and softmax for binary predictions (Rajput et al., 2021, Mollas et al., 2020).

Transfer learning further enhances detection of implicit forms of hate. Cabrera et al. (Cabrera et al., 22 Aug 2025) show that BERT+BiLSTM models, when sequentially pre-trained on sarcasm-annotated corpora and then fine-tuned on hate speech (including ETHOS), significantly increase recall (+9.7%), area under the ROC curve (+7.8%), and F1-score (+6%). This result demonstrates that rhetorical misdirection and lexical overlap between sarcasm and implicit hate facilitate transferability of learned representations, especially for subtle and indirect forms.

4. Evaluation Protocols and Results

ETHOS classifiers are evaluated via stratified 10-fold cross-validation; loss is binary cross-entropy for the binary variant, with classical metrics—accuracy, precision, recall, F1-score, specificity—reported on test folds (Rajput et al., 2021, Mollas et al., 2020). Baseline models using only fastText or GloVe yield F1 scores in the 75–78% range, while fine-tuned BERT achieves F1 = 78.83, accuracy = 76.64, recall = 78.43, specificity = 74.31. Static BERT embeddings plugged into BiLSTM push these metrics to F1 = 79.71, accuracy = 80.15, recall = 79.76, specificity = 83.03 (Δ specificity ≈ +8.7 pp over fine-tuned BERT).

Multi-label ETHOS models are evaluated on example-based, macro/micro F1, precision, recall, subset accuracy, and Hamming loss. A BiLSTM+Attention+FF network achieves macro F1 = 76.23 and micro F1 = 74.87, substantially exceeding classical problem-transformation methods (Mollas et al., 2020).

Model	Architecture	F1	Recall	Specificity
Fine-tuned BERT	Transformer	78.83	78.43	74.31
BiLSTM + static BERT	RNN + static BE	79.71	79.76	83.03
BiLSTM + FT+GV	RNN + FT+GV	75.40	78.04	78.94
BERT+BiLSTM + sarcasm pretrain	Frozen BERT+BiLSTM	81.6	86.6	N/A

Quantitative gains reflect improvements principally in reducing false positives, a critical property for real-world moderation where overblocking is a concern (Rajput et al., 2021, Cabrera et al., 22 Aug 2025).

5. Advances in Embedding Learning and Contextual Features

Context-aware hate speech detection increasingly focuses on capturing indirect language and domain-evasive phenomena. Alorainy et al. (Alorainy et al., 2018) propose augmenting Doc2Vec-style embeddings with explicit “othering” features—syntactic and part-of-speech markers of intergroup threat (e.g., co-occurrence of in-group and out-group pronouns, action verbs)—constructed via dependency parsing. The pipeline yields learned embeddings sensitive to both local context (“send them back”) and global narrative threat structure. When adapted for ETHOS, such feature engineering can be integrated into the input layer as an augmentation to standard embeddings, improving detection of subtle, symbolic hate.

Extremist community mining and code-word induction add further nuance, as contextually learned word representations capture evolving hate-driven semantics (e.g., lexical “code words” created or appropriated to evade classifiers). Graph-based propagation over empirical contextual similarity surfaces candidate terms that can be fed into ETHOS models for improved recall on emerging slurs (Taylor et al., 2017).

6. Hypothesis-based and Zero-Shot Detection Paradigms

Recent paradigm shifts use zero-shot frameworks recasting hate detection as natural language inference (NLI): input texts are paired with hate-centric hypotheses and scored via NLI models such as BART-large-MNLI. Error analysis on ETHOS informs composite hypothesis engineering—filtering by protected target detection, counterspeech cues, self-directed/reclaimed slur identification, and animal/dehumanization metaphors (Goldzycher et al., 2022). On ETHOS, this method achieves 69.6% accuracy in the vanilla NLI setup, with further improvements via supporting strategies.

Strategy	Accuracy on ETHOS (%)
Vanilla NLI	69.6
Multi-hypothesis (all)	79.6

Such approaches enable rapid adaptation to new domains or languages without retraining, but require precise linguistic decomposition of hate-speech phenomena and careful handling of dataset-specific label criteria.

7. Rhetorical, Thematic, and Ethos Dimensions in Annotation and Detection

Annotation schemes have evolved to multi-dimensional taxonomies, most recently factoring thematic (call-for-action, contempt, narrative undermining) vs. rhetorical (Logos, Pathos, Ethos) dimensions. “Dialogues of Dissent” (Levi et al., 28 Jul 2025) case-studies ETHOS-style annotation, operationalizing Ethos as positive self-construction versus negative representation of the target. Manual annotation on a 92-conversation pilot—123 hate and 175 counter-hate tweets—shows Ethos is rare in hate messages but more prevalent in counter-hate, predominantly as credibility attacks (“neg. other”). The lack of feature sets and formal trainable models for Ethos classification underlines a gap: automated systems should encode explicit credibility cues and stratify Ethos sub-categories to fully realize the spectrum of hate discourse representation.

The extremely low frequency of Ethos in hate messages presents challenges for reliable supervised learning; expansion to multi-annotator, larger corpora is anticipated as a necessary next step for this dimension.

---

ETHOS hate speech classification presently combines balanced, granular annotation protocols with embedding-rich, neural and transfer-based classification systems. Advances in transfer learning and context modeling have improved detection of both explicit and especially implicit hate, with static BERT embeddings and feature-augmented representation yielding robust gains in specificity and recall. However, operationalizing discursive and rhetorical subtleties such as Ethos remains an open problem, requiring larger datasets, refined annotation, and new approaches to lexicon and deep feature engineering. The ETHOS benchmark, with its high-quality annotations and evolving methodology, is poised to remain a central testbed for the next generation of hate speech detection systems (Mollas et al., 2020, Rajput et al., 2021, Cabrera et al., 22 Aug 2025, Alorainy et al., 2018, Taylor et al., 2017, Goldzycher et al., 2022, Levi et al., 28 Jul 2025).