HateClipSeg: Multimodal Hate Video Benchmark
- The paper’s main contribution is recasting hate video analysis from coarse whole-video labeling to temporally precise, segment-level moderation.
- It comprises over 11,714 segments with annotations across text, audio, and visuals, categorizing content into Normal and five offensive classes.
- Benchmark tasks include trimmed classification, temporal localization, and online classification to address fairness and enhance content moderation.
HateClipSeg is a large-scale multimodal video dataset and benchmark suite for fine-grained hate video detection, introduced to address the lack of temporally precise annotations in existing resources for hateful and offensive video understanding. It combines video-level and segment-level annotation, comprises over 11,714 segments, and labels each segment as Normal or as one of five Offensive categories—Hateful, Insulting, Sexual, Violence, and Self-Harm—together with explicit target victim labels. Its central contribution is to recast hate video analysis from coarse whole-video classification into temporally resolved moderation, enabling benchmarking for trimmed classification, temporal localization, and online classification under multimodal conditions (Wang et al., 3 Aug 2025).
1. Problem formulation and motivation
HateClipSeg is motivated by the observation that online hate speech increasingly appears in videos that mix text, visuals, and audio, and that harmful intent may be conveyed by interactions across modalities rather than by any single modality in isolation. The motivating examples given for this setting include seemingly benign text accompanied by violent imagery or a sarcastic tone. In this formulation, hate video detection is not merely a lexical or frame-wise recognition problem; it is a multimodal and temporal inference problem in which the semantics of offense may depend on context unfolding over time (Wang et al., 3 Aug 2025).
A core premise of the dataset is that prior datasets focus largely on coarse video-level labels such as hateful versus normal. That coarse labeling obscures type-specific harms and makes moderation blunt: a platform may remove an entire video even when only part of it is harmful, or may miss subtle but consequential segments. HateClipSeg therefore treats segment-level annotation as operationally meaningful for moderation, transparency, and fairness. The dataset description explicitly connects fine-grained labeling to distinguishing racial slurs from sexual insults and to identifying targeted victim groups, thereby moving beyond undifferentiated toxicity labels.
A recurrent misconception in video moderation is that harmful content can be adequately characterized from isolated frames or transcript snippets. HateClipSeg is organized around the opposite view: temporal context and cross-modal interaction are intrinsic to the detection problem. This is why the resource is defined not simply as a multimodal corpus, but as a multimodal corpus with predefined, semantically coherent segment boundaries.
2. Corpus construction and annotation protocol
The dataset is built from videos collected from YouTube and BitChute, the latter described in the source material as a minimally moderated site known for extremist and conspiratorial content. Retrieval begins from a seed lexicon of over 100 terms spanning race, gender, religion, and sexuality, compiled from Hatebase and HateXplain plus manual curation. Examples listed in the source include race-related slurs, gendered insults, religious epithets, and sexuality-related slurs. Keyword search over this lexicon was applied to videos between 3 and 10 minutes, producing an initial pool of 4,745 videos. A fine-tuned LLaMA-3.2-11B model, trained on MultiHateClip, was then used to retain videos predicted hateful, yielding 435 candidate videos for further processing (Wang et al., 3 Aug 2025).
The annotation pipeline is explicitly three-stage, and its stated purpose is to improve reliability. The reported inter-annotator agreement is high, with Krippendorff's alpha equal to 0.817. Although the source material does not enumerate the three stages in procedural detail, it does frame the protocol as a reliability-centered design choice, paired with semantically coherent segmentation and explicit harm-category and victim-target labeling.
| Aspect | Specification |
|---|---|
| Sources | YouTube and BitChute |
| Initial retrieval pool | 4,745 videos |
| Filtered candidate videos | 435 |
| Segment count | over 11,714 |
| Segment labels | Normal, Hateful, Insulting, Sexual, Violence, Self-Harm |
| Agreement | Krippendorff's alpha = 0.817 |
The label space is broader than binary hate detection. In addition to Normal, the dataset separates five Offensive categories: Hateful, Insulting, Sexual, Violence, and Self-Harm. This categorical granularity matters because it encodes distinctions among different kinds of harmful or offensive content rather than collapsing them into a single toxic class. The inclusion of explicit target victim labels further supports analyses of targeted hostility, group-specific abuse, and moderation specificity.
3. Segmentation design and multimodal representation
HateClipSeg emphasizes segment boundaries that are predefined and semantically coherent. The segmentation pipeline begins with Whisper, which generates word-level timestamps. These are merged into sentence-level segments using NLTK Punkt. Additional splitting is introduced when silent intervals longer than 20 seconds occur; in those cases, scene changes are determined by drops in cosine similarity between consecutive ViT-based frame embeddings. The result is a segmentation regime designed to preserve linguistic and visual coherence rather than imposing arbitrary fixed windows (Wang et al., 3 Aug 2025).
The dataset includes RGB vision, audio, and text. For vision, frames or features are extracted from a frozen ViT-Large encoder. For temporal tasks, frame or temporal features are used, whereas trimmed classification uses a single representative frame per segment. For audio, features come from a frozen Wav2Vec-Emotion encoder. For text, Whisper provides ASR transcripts, and BERT-Base provides features for temporal and online tasks; the text stream is sentence-segmented via NLTK Punkt with word-level timestamps. This modality design indicates that HateClipSeg is intended not only as an annotation resource but also as a standardized multimodal feature benchmark.
The segmentation procedure has methodological significance. It is neither purely transcript-driven nor purely scene-driven. Instead, it couples sentence boundaries with silence-triggered scene analysis, which reflects the fact that harmful intent may align with spoken utterances, with visual transitions, or with their conjunction. A plausible implication is that the dataset is especially suitable for research on cross-modal temporal grounding, where the unit of analysis must remain meaningful across text, audio, and video streams.
4. Benchmark tasks
HateClipSeg defines three benchmark tasks: Trimmed Hateful Video Classification, Temporal Hateful Video Localization, and Online Hateful Video Classification (Wang et al., 3 Aug 2025).
Trimmed Hateful Video Classification evaluates classification on pre-segmented units. Because the segments are semantically coherent and temporally bounded in advance, this task isolates recognition from boundary discovery. It is therefore useful for studying category discrimination, multimodal fusion, and target-victim identification under controlled temporal scope.
Temporal Hateful Video Localization addresses the problem of determining where in a video hateful or offensive content occurs. This task is central to fine-grained moderation, because it supports operations that act on offending parts rather than on entire videos. In the logic of the dataset, localization is not an auxiliary add-on to classification; it is a primary task required for temporally precise content governance.
Online Hateful Video Classification is designed to reflect real-world moderation needs including live-streaming scenarios. In contrast to offline analysis of a completed clip, the online setting foregrounds incremental decision-making under partial temporal context. This makes it relevant to streaming moderation, latency-constrained inference, and early-warning detection.
The benchmark results are summarized qualitatively in the source material rather than numerically in the provided excerpt. The reported conclusion is that there are substantial gaps in current models, which emphasizes the need for more sophisticated multimodal and temporally aware approaches. That framing positions HateClipSeg not as a solved benchmark, but as a diagnostic resource exposing the limitations of existing video hate detection systems.
5. Moderation relevance, interpretability, and fairness
A defining feature of HateClipSeg is its explicit connection to operational moderation. Segment-level annotations enable flagging, removing, or restricting only the offending parts of a video. In the dataset rationale, this granularity is tied to improved transparency and fairness, because moderation decisions can be localized, categorized, and attributed more precisely than in whole-video binary pipelines (Wang et al., 3 Aug 2025).
The dataset’s harm taxonomy also changes what counts as interpretability. Rather than only predicting that a video is problematic, a system trained or evaluated on HateClipSeg can in principle specify whether a segment is Hateful, Insulting, Sexual, Violent, or related to Self-Harm, while also attaching an explicit target victim label when relevant. This makes the annotation schema useful for downstream auditing and policy analysis, particularly where different moderation policies apply to different categories of harm.
The emphasis on fairness should not be interpreted as a claim that segment-level labeling automatically resolves moderation bias. The source material makes the narrower claim that fine-grained annotation improves transparency and fairness relative to blunt whole-video decisions. This suggests a more auditable moderation regime, but it does not imply that model bias, annotation bias, or policy disagreement disappears. What HateClipSeg concretely provides is a representation format in which those issues can be studied with substantially greater precision.
The dataset is publicly available at https://github.com/Social-AI-Studio/HateClipSeg.git, which makes it usable for comparative benchmarking, annotation studies, and deployment-oriented moderation research.
6. Relation to temporally resolved hate localization research
HateClipSeg belongs to a broader shift from coarse multimodal hate classification toward temporal localization. Closely related later work illustrates the methodological directions that such segment-level benchmarks make possible, even when those works evaluate on other datasets. LELA, for example, formulates training-free multimodal hate localization at frame level and decomposes video into five modalities—image, speech, OCR, music, and video context—using modality-specific captioning and multi-stage LLM prompting to produce per-frame hateful scores and contiguous segments (Sun et al., 10 Feb 2026).
MultiHateLoc studies a different but complementary setting: weakly supervised multimodal hate localisation under video-level labels only. Its architecture combines modality-aware temporal encoders, dynamic cross-modal fusion, cross-modal contrastive alignment, and a modality-aware multiple instance learning objective to generate fine-grained frame-level predictions on HateMM and MultiHateClip (Sun et al., 11 Dec 2025). This suggests that the research agenda around segment-level hate detection now spans fully supervised, weakly supervised, and training-free regimes.
A separate naming confusion arises from the overlap between “HateClipSeg” and CLIP-based image segmentation nomenclature. Work such as Trident concerns training-free open-vocabulary semantic segmentation with CLIP, DINO, and SAM, and discusses adaptation to CLIP-based pipelines such as CLIPSeg or “HateCLIPSeg” in the context of image segmentation (Shi et al., 2024). That line of work is methodologically distinct from HateClipSeg. The former addresses open-vocabulary semantic segmentation in images; the latter denotes a multimodal video dataset and benchmark suite for temporally precise hate and offensive content detection.
Taken together, these adjacent developments situate HateClipSeg within a rapidly emerging subfield focused on when, not only whether, multimodal hate appears in video. Its main contribution is to provide the annotation granularity needed to study that question directly.
