---
title: 'MultiHateGNN: Dual-Stream Hate Video Detection'
url: https://www.emergentmind.com/topics/multihategnn
type: topic
---

# MultiHateGNN: Dual-Stream Hate Video Detection

Searching arXiv for the named paper and closely related graph-based hate detection works.
Search query: "ti:\"Multimodal Hate Detection Using Dual-Stream Graph Neural Networks\" OR MultiHateGNN hateful video graph neural network"
MultiHateGNN is the model name proposed in “Multimodal Hate Detection Using Dual-Stream Graph Neural Networks” by Jiangbei Yue, Shuonan Yang, Tailin Chen, Jianbo Jiao, and Zeyu Fu. It addresses **hateful video detection** as **binary multimodal hate classification**: given a video, predict whether it is **hate** (\(y=1\)) or **non-hate** (\(y=0\)). The model uses **visual frames, audio, and text transcripts**, and combines an **instance graph** with a **complementary weight graph** so that localized hateful evidence can determine the video-level decision [2509.13515].

## 1. Scope, task definition, and motivating asymmetry

The paper identifies a two-part limitation in prior multimodal hate detection methods. First, they treat all content too uniformly, even though in hateful videos a **small amount of hateful content** is enough to make the whole video hateful. Second, they do not explicitly model **structured multimodal relations**, including **intra-modal temporal structure** and **inter-modal relational structure** across aligned timestamps [2509.13515].

The motivating asymmetry is central to the method. A video may contain many non-hateful parts and only a few hateful parts, yet the hateful segments determine the label. The paper therefore argues against naive averaging or global pooling over the entire video. It instead separates the video into **multiple instances** and reasons at the instance level, so that hateful evidence becomes more concentrated within smaller temporal units. In the paper’s formulation, the model should both compute a representation for each instance and estimate how important each instance is to the final decision [2509.13515].

A common source of confusion is the model’s scope. Despite the breadth suggested by the name, MultiHateGNN is not presented as a generic hate-speech graph framework. In the cited work it is specifically a **dual-stream multimodal graph neural network for hateful video classification**. The text modality is the **speech transcript** obtained from audio using ASR; **no OCR stream is described**, and the paper explicitly states that the text modality is not OCR of on-screen text [2509.13515].

## 2. Dual-stream formulation and instance-level reasoning

MultiHateGNN uses two graph streams with a clear division of labor. The **instance graph** extracts instance-level multimodal features, while the **complementary weight graph** estimates instance importance weights. The paper summarizes this division as: **instance graph = what each instance contains** and **weight graph = how much each instance should matter** [2509.13515].

The end-to-end pipeline is organized as follows. A video is split into \(N\) temporal segments. For each segment, the model extracts a key frame, the corresponding audio chunk, and the corresponding text transcript. These are encoded into segment-level features of shared dimensionality \(D\). The model then builds two graph structures: a **weight graph** over all segments and modalities, and an **instance graph** made of \(K\) instance subgraphs. One GNN processes the weight graph to produce instance weights \(\{\alpha_i\}_{i=1}^K\), and another GNN, shared across instances, processes instance subgraphs to produce instance features \(\{f_i\}_{i=1}^K\). The final video representation is the weighted sum
\[
f=\sum_{i=1}^{K}\alpha_i f_i,
\]
which is fed to an **MLP + softmax** classifier [2509.13515].

The definition of an instance is temporal and contiguous. After the video is divided into \(N\) segments, it is divided into \(K\) non-overlapping instances, each containing the same number of segments:
\[
M_i = N/K.
\]
Thus an instance is a contiguous temporal block containing \(N/K\) segments. The main text does not provide the exact values of \(N\) and \(K\); those details are deferred to supplementary material. This omission is important for reproduction, because the segmentation granularity is directly tied to how localized hateful evidence can be isolated [2509.13515].

## 3. Modal encoders and graph construction

At the segment level, the model extracts one feature sequence per modality and projects all three to a shared dimensionality \(D\). For **vision**, the first frame from each segment is taken, the \(N\) frames are fed sequentially into a pre-trained **Vision Transformer (ViT)**, producing \(N\) features of dimension 768, and these are passed through an **LSTM** projection layer to obtain \(N\) visual features in \(\mathbb{R}^D\). For **audio**, segment-level audio is converted to **40-dimensional MFCC** features and passed through an **LSTM** to obtain \(N\) audio features in \(\mathbb{R}^D\). For **text**, the entire video audio is transcribed sentence by sentence with **Whisper**; for each segment, transcript sentences whose time spans overlap with that segment are merged into a paragraph; the \(N\) paragraphs are fed sequentially into **BERT**, producing \(N\) 768-dimensional textual features, which are projected by an **MLP** to \(N\) text features in \(\mathbb{R}^D\). The final result is three feature matrices of shape \([N,D]\): visual, audio, and text [2509.13515].

The **weight graph** is denoted
\[
\mathcal{G}_W = \{\mathcal{V}_W, \mathcal{E}_W\}.
\]
It has one node per modality per segment: \(N\) visual nodes, \(N\) audio nodes, and \(N\) text nodes, for a total of
\[
|\mathcal{V}_W| = 3N.
\]
Its edges are undirected and encode both **intra-modal** and **inter-modal** structure. Within each modality, temporal adjacency edges connect adjacent segments, and similarity-based edges are added using the \(\epsilon\)-graph rule: two same-modality nodes are connected if the cosine distance between their features is below threshold \(\epsilon\), where
\[
dist(v_i, v_j) = 1 - \frac{v_i \cdot v_j}{\|v_i\|_2 \|v_j\|_2}.
\]
At the same timestamp, nodes from different modalities are connected pairwise as visual-audio, visual-text, and audio-text. The paper notes that this adjacency is partly predefined by time and partly similarity-based by feature distance; it is not learned end-to-end in the paper’s formulation [2509.13515].

The **instance graph** is built by dividing the video into \(K\) instances and constructing one subgraph per instance using the same rule as the weight graph. If each instance contains \(M_i=N/K\) segments, each instance subgraph has \(3M_i\) nodes. Within each subgraph, the model again uses temporal adjacency within modality, \(\epsilon\)-graph similarity edges within modality, and pairwise cross-modal edges at aligned timestamps. A **shared GNN** is then applied to each subgraph one by one. The main text does not specify the exact GNN layer type, number of layers, or update equations; it cites GAT/GNN literature, but the precise message-passing operator is not given in the main paper [2509.13515].

## 4. Mathematical formulation and optimization

The weight graph produces node-level importance scores for visual, audio, and text nodes:
\[
\{\alpha_j^v\}_{j=1}^N,\quad \{\alpha_j^a\}_{j=1}^N,\quad \{\alpha_j^t\}_{j=1}^N.
\]
These are normalized with a softmax **within each modality**, yielding
\[
\{\hat{\alpha}_j^v\}_{j=1}^N,\quad \{\hat{\alpha}_j^a\}_{j=1}^N,\quad \{\hat{\alpha}_j^t\}_{j=1}^N.
\]
The instance weight is then computed by averaging normalized modality scores over the segments covered by instance \(i\):
\[
\alpha_i = \frac{1}{3}\sum_{l \in \Omega_i} \hat{\alpha}_l^v + \hat{\alpha}_l^a + \hat{\alpha}_l^t,
\]
where \(\Omega_i\) is the set of segment indices in instance \(i\). This is the mechanism that highlights potentially hateful temporal regions [2509.13515].

After GNN processing of an instance subgraph, the model averages node representations within each modality:
\[
f_i^{*} = \frac{1}{M} \sum_{p=1}^{M_i} f_i^{*,p}, \quad * \in \{v,a,t\},
\]
and then concatenates the three modality summaries:
\[
f_i = [f_i^v \,\|\, f_i^a \,\|\, f_i^t].
\]
The paper explicitly chooses **simple average** within each modality for node aggregation, not weighted average, and **concatenation** across modalities, not averaging, to preserve modality-specific information. It also contains a notational inconsistency: the formula writes \(1/M\) while the text defines \(M_i=N/K\). The intended meaning is averaging over the \(M_i\) nodes of that modality inside instance \(i\) [2509.13515].

Video-level fusion is the weighted aggregation
\[
f = \sum_{i=1}^K \alpha_i f_i.
\]
The classifier is
\[
\hat{h} = \phi(\mathrm{MLP}(f)),
\]
where \(\hat{h}=[\hat{h}_0,\hat{h}_1]\), \(\hat{h}_0\) is the predicted probability of non-hate, \(\hat{h}_1\) is the predicted probability of hate, and \(\phi\) is softmax. Training uses only cross-entropy loss:
\[
\mathcal{L}_{CE} = -\sum_{c=1}^2 h_c \log(\hat{h}_c).
\]
No auxiliary losses are described in the provided text. The paper also does not specify, in the provided excerpt, the optimizer, learning rate, batch size, number of epochs, early stopping, scheduler, or explicit regularization such as dropout or weight decay [2509.13515].

## 5. Datasets, evaluation protocol, and empirical results

The paper evaluates MultiHateGNN on two public datasets. **HateMM** contains **1083 videos total**, with **652 non-hateful** and **431 hateful**, and uses **5-fold stratified cross-validation** with a standard **70% train / 10% val / 20% test** split. **MultiHateClip (MHC)** contains **1000 English-language videos** with original classes **662 non-hateful**, **256 offensive**, and **82 hateful**; for binary classification, offensive and hateful are merged into **338 hateful**, leaving **662 non-hateful**. On MHC, each model is run **five times**, average performance is reported, and the split is again **70% train / 10% val / 20% test**. Evaluation uses **Accuracy**, **F1-score**, **Precision**, and **Recall**, with the **hate** label (\(y=1\)) treated as the positive class for F1, precision, and recall [2509.13515].

### HateMM results

| Model | Accuracy | F1 | Precision | Recall |
|---|---:|---:|---:|---:|
| ViT | 0.693 | 0.601 | 0.623 | 0.589 |
| MFCC | 0.682 | 0.622 | 0.602 | 0.651 |
| BERT | 0.706 | 0.630 | 0.646 | 0.622 |
| GPT-4o | 0.777 | 0.755 | 0.671 | 0.863 |
| HateMM | 0.805 | 0.753 | 0.765 | 0.751 |
| CMFusion | 0.799 | 0.739 | 0.763 | 0.719 |
| **MultiHateGNN** | **0.821** | **0.771** | **0.798** | 0.754 |

On HateMM, MultiHateGNN is best on all metrics except recall, where GPT-4o is higher but with much worse precision [2509.13515].

### MHC results

| Model | Accuracy | F1 | Precision | Recall |
|---|---:|---:|---:|---:|
| ViViT | 0.73 | 0.68 | 0.86 | 0.57 |
| MFCC | 0.54 | 0.36 | 0.40 | 0.33 |
| mBERT | 0.57 | 0.52 | 0.68 | 0.42 |
| GPT-4o | 0.68 | 0.29 | 0.57 | 0.19 |
| MHC | 0.75 | 0.67 | 0.77 | 0.61 |
| CMFusion | 0.73 | 0.72 | 0.72 | 0.73 |
| **MultiHateGNN** | **0.78** | **0.77** | 0.80 | **0.77** |

On MHC, the paper states that MultiHateGNN improves the prior best **accuracy by 3%** and **F1 by 5%** [2509.13515].

The ablation study on HateMM isolates the contribution of each stream:

| Model | Accuracy | F1 | Precision | Recall |
|---|---:|---:|---:|---:|
| No Graph | 0.792 | 0.742 | 0.759 | 0.727 |
| Only Instance Graph | 0.798 | 0.746 | 0.767 | 0.726 |
| Only Weight Graph | 0.813 | 0.756 | 0.789 | 0.738 |
| **Full Model** | **0.821** | **0.771** | **0.798** | **0.754** |

The reported interpretation is that **No Graph** performs worst, **Only Weight Graph** is the strongest single-stream baseline, and the **full dual-stream model** performs best, indicating that content extraction and weighting are complementary [2509.13515].

## 6. Explainability, strengths, and limitations

A major claim of the paper is that MultiHateGNN offers **explainability** through the estimated **instance importance weights** \(\alpha_i\). Because prediction uses
\[
f = \sum_{i=1}^{K} \alpha_i f_i,
\]
the learned weights directly indicate which temporal instances contributed most to the final decision. The authors state that these instance weights are helpful for **localization of hateful content**, although the model is not yet a full temporal localization system [2509.13515].

The qualitative example provided in the paper illustrates the intended behavior. In a HateMM case, a video is divided into **10 instances**, but only **instance 3** contains hateful content. Existing methods struggle because the other 9 instances are non-hateful and dominate the video. MultiHateGNN assigns instance 3 a high weight,
\[
\alpha_3 = 0.52,
\]
which allows correct hateful classification. This makes the explainability claim concrete: the model’s weighting stream is not merely auxiliary, but directly tied to the final video representation [2509.13515].

The paper and its technical summary also delineate several limitations. The method depends on segmentation quality; poor segmentation could dilute or split critical cues. It may be sensitive to ASR quality because the text modality relies on **Whisper** transcripts. The \(\epsilon\)-graph similarity construction may increase computational cost as \(N\) grows. Exact reproduction from the main text alone is difficult because the precise GNN architecture, \(N\), \(K\), optimizer, and training schedule are deferred to supplementary material. Finally, while the model provides instance-level explanations, it is not yet a full localization framework; the conclusion suggests future work on multi-scale localization. Code availability is explicitly stated at `https://github.com/Multimodal-Intelligence-Lab-MIL/MultiHateGNN` [2509.13515].

## 7. Position within related graph-based hate research

MultiHateGNN belongs to a broader landscape of graph-based hate research, but it occupies a specific niche. It is a **multimodal video classifier** that models structured relations across visual, audio, and transcript streams inside a video. This differs from graph methods designed for **multi-relation social diffusion** or **user-level hate-monger detection** [2509.13515].

One adjacent line is “Convolutional Learning on Multigraphs,” which develops a formal framework for **multigraph signal processing** and introduces **Multigraph Neural Networks (MGNNs)** for data supported on multiple edge classes over a shared node set. That paper is validated on a **hate speech source localization task on Twitter**, using a **follow graph** and a **retweet graph**, and shows improved performance over parallel and merged graph baselines [2209.11354]. The relation to MultiHateGNN is methodological rather than task-identical: MGNN is a multi-relation propagation framework for social-network diffusion, whereas MultiHateGNN is a dual-stream multimodal graph architecture for localized hateful evidence in videos. A plausible implication is that both methods respond to the same general problem—single-structure modeling can obscure crucial heterogeneous relations—but they instantiate that principle in different substrates.

Another nearby line is “Social Hatred: Efficient Multimodal Detection of Hatemongers,” which targets **user/account-level** hate-monger detection by aggregating post-level textual signals with social-network context across **Twitter/X, Gab, and Parler**. That paper is explicitly **not itself a graph neural network**; its graph component is a hand-constructed one-hop relational aggregation rather than learned message passing [2506.19603]. In contrast, MultiHateGNN learns graph-based representations directly over multimodal video segments and instances. The comparison is useful because it clarifies that “multimodal hate detection” spans at least three distinct units of analysis: video-level classification, social-diffusion localization, and user-level hate-monger detection. MultiHateGNN is specifically the video-level, dual-stream GNN instantiation among these possibilities.

Source: https://www.emergentmind.com/topics/multihategnn