MultiHateGNN: Dual-Stream Hate Video Detection

• The paper's main contribution is a dual-stream GNN that separately extracts instance-level features and computes importance weights to localize hateful evidence in videos.
• MultiHateGNN integrates visual, audio, and text modalities by constructing instance and weight graphs that capture intra-modal temporal and inter-modal relational structures.
• Empirical results on HateMM and MHC datasets show improved accuracy and F1 scores, validating its effectiveness in handling videos with sparse hateful segments.

Searching arXiv for the named paper and closely related graph-based hate detection works. Search query: "ti:\"Multimodal Hate Detection Using Dual-Stream Graph Neural Networks\" OR MultiHateGNN hateful video graph neural network" MultiHateGNN is the model name proposed in “Multimodal Hate Detection Using Dual-Stream Graph Neural Networks” by Jiangbei Yue, Shuonan Yang, Tailin Chen, Jianbo Jiao, and Zeyu Fu. It addresses hateful video detection as binary multimodal hate classification: given a video, predict whether it is hate ($y=1$) or non-hate ($y=0$). The model uses visual frames, audio, and text transcripts, and combines an instance graph with a complementary weight graph so that localized hateful evidence can determine the video-level decision (Yue et al., 16 Sep 2025).

1. Scope, task definition, and motivating asymmetry

The paper identifies a two-part limitation in prior multimodal hate detection methods. First, they treat all content too uniformly, even though in hateful videos a small amount of hateful content is enough to make the whole video hateful. Second, they do not explicitly model structured multimodal relations, including intra-modal temporal structure and inter-modal relational structure across aligned timestamps (Yue et al., 16 Sep 2025).

The motivating asymmetry is central to the method. A video may contain many non-hateful parts and only a few hateful parts, yet the hateful segments determine the label. The paper therefore argues against naive averaging or global pooling over the entire video. It instead separates the video into multiple instances and reasons at the instance level, so that hateful evidence becomes more concentrated within smaller temporal units. In the paper’s formulation, the model should both compute a representation for each instance and estimate how important each instance is to the final decision (Yue et al., 16 Sep 2025).

A common source of confusion is the model’s scope. Despite the breadth suggested by the name, MultiHateGNN is not presented as a generic hate-speech graph framework. In the cited work it is specifically a dual-stream multimodal graph neural network for hateful video classification. The text modality is the speech transcript obtained from audio using ASR; no OCR stream is described, and the paper explicitly states that the text modality is not OCR of on-screen text (Yue et al., 16 Sep 2025).

2. Dual-stream formulation and instance-level reasoning

MultiHateGNN uses two graph streams with a clear division of labor. The instance graph extracts instance-level multimodal features, while the complementary weight graph estimates instance importance weights. The paper summarizes this division as: instance graph = what each instance contains and weight graph = how much each instance should matter (Yue et al., 16 Sep 2025).

The end-to-end pipeline is organized as follows. A video is split into $N$ temporal segments. For each segment, the model extracts a key frame, the corresponding audio chunk, and the corresponding text transcript. These are encoded into segment-level features of shared dimensionality $D$. The model then builds two graph structures: a weight graph over all segments and modalities, and an instance graph made of $K$ instance subgraphs. One GNN processes the weight graph to produce instance weights $\{\alpha_i\}_{i=1}^K$, and another GNN, shared across instances, processes instance subgraphs to produce instance features $\{f_i\}_{i=1}^K$. The final video representation is the weighted sum

$f=\sum_{i=1}^{K}\alpha_i f_i,$

which is fed to an MLP + softmax classifier (Yue et al., 16 Sep 2025).

The definition of an instance is temporal and contiguous. After the video is divided into $N$ segments, it is divided into $K$ non-overlapping instances, each containing the same number of segments: $y=0$0 Thus an instance is a contiguous temporal block containing $y=0$1 segments. The main text does not provide the exact values of $y=0$2 and $y=0$3; those details are deferred to supplementary material. This omission is important for reproduction, because the segmentation granularity is directly tied to how localized hateful evidence can be isolated (Yue et al., 16 Sep 2025).

3. Modal encoders and graph construction

At the segment level, the model extracts one feature sequence per modality and projects all three to a shared dimensionality $y=0$4. For vision, the first frame from each segment is taken, the $y=0$5 frames are fed sequentially into a pre-trained Vision Transformer (ViT), producing $y=0$6 features of dimension 768, and these are passed through an LSTM projection layer to obtain $y=0$7 visual features in $y=0$8. For audio, segment-level audio is converted to 40-dimensional MFCC features and passed through an LSTM to obtain $y=0$9 audio features in $N$0. For text, the entire video audio is transcribed sentence by sentence with Whisper; for each segment, transcript sentences whose time spans overlap with that segment are merged into a paragraph; the $N$1 paragraphs are fed sequentially into BERT, producing $N$2 768-dimensional textual features, which are projected by an MLP to $N$3 text features in $N$4. The final result is three feature matrices of shape $N$5: visual, audio, and text (Yue et al., 16 Sep 2025).

The weight graph is denoted

$N$6

It has one node per modality per segment: $N$7 visual nodes, $N$8 audio nodes, and $N$9 text nodes, for a total of

$D$0

Its edges are undirected and encode both intra-modal and inter-modal structure. Within each modality, temporal adjacency edges connect adjacent segments, and similarity-based edges are added using the $D$1-graph rule: two same-modality nodes are connected if the cosine distance between their features is below threshold $D$2, where

$D$3

At the same timestamp, nodes from different modalities are connected pairwise as visual-audio, visual-text, and audio-text. The paper notes that this adjacency is partly predefined by time and partly similarity-based by feature distance; it is not learned end-to-end in the paper’s formulation (Yue et al., 16 Sep 2025).

The instance graph is built by dividing the video into $D$4 instances and constructing one subgraph per instance using the same rule as the weight graph. If each instance contains $D$5 segments, each instance subgraph has $D$6 nodes. Within each subgraph, the model again uses temporal adjacency within modality, $D$7-graph similarity edges within modality, and pairwise cross-modal edges at aligned timestamps. A shared GNN is then applied to each subgraph one by one. The main text does not specify the exact GNN layer type, number of layers, or update equations; it cites GAT/GNN literature, but the precise message-passing operator is not given in the main paper (Yue et al., 16 Sep 2025).

4. Mathematical formulation and optimization

The weight graph produces node-level importance scores for visual, audio, and text nodes: $D$8 These are normalized with a softmax within each modality, yielding

$D$9

The instance weight is then computed by averaging normalized modality scores over the segments covered by instance $K$0: $K$1 where $K$2 is the set of segment indices in instance $K$3. This is the mechanism that highlights potentially hateful temporal regions (Yue et al., 16 Sep 2025).

After GNN processing of an instance subgraph, the model averages node representations within each modality: $K$4 and then concatenates the three modality summaries: $K$5 The paper explicitly chooses simple average within each modality for node aggregation, not weighted average, and concatenation across modalities, not averaging, to preserve modality-specific information. It also contains a notational inconsistency: the formula writes $K$6 while the text defines $K$7. The intended meaning is averaging over the $K$8 nodes of that modality inside instance $K$9 (Yue et al., 16 Sep 2025).

Video-level fusion is the weighted aggregation

$\{\alpha_i\}_{i=1}^K$0

The classifier is

$\{\alpha_i\}_{i=1}^K$1

where $\{\alpha_i\}_{i=1}^K$2, $\{\alpha_i\}_{i=1}^K$3 is the predicted probability of non-hate, $\{\alpha_i\}_{i=1}^K$4 is the predicted probability of hate, and $\{\alpha_i\}_{i=1}^K$5 is softmax. Training uses only cross-entropy loss: $\{\alpha_i\}_{i=1}^K$6 No auxiliary losses are described in the provided text. The paper also does not specify, in the provided excerpt, the optimizer, learning rate, batch size, number of epochs, early stopping, scheduler, or explicit regularization such as dropout or weight decay (Yue et al., 16 Sep 2025).

5. Datasets, evaluation protocol, and empirical results

The paper evaluates MultiHateGNN on two public datasets. HateMM contains 1083 videos total, with 652 non-hateful and 431 hateful, and uses 5-fold stratified cross-validation with a standard 70% train / 10% val / 20% test split. MultiHateClip (MHC) contains 1000 English-language videos with original classes 662 non-hateful, 256 offensive, and 82 hateful; for binary classification, offensive and hateful are merged into 338 hateful, leaving 662 non-hateful. On MHC, each model is run five times, average performance is reported, and the split is again 70% train / 10% val / 20% test. Evaluation uses Accuracy, F1-score, Precision, and Recall, with the hate label ($\{\alpha_i\}_{i=1}^K$7) treated as the positive class for F1, precision, and recall (Yue et al., 16 Sep 2025).

HateMM results

Model	Accuracy	F1	Precision	Recall
ViT	0.693	0.601	0.623	0.589
MFCC	0.682	0.622	0.602	0.651
BERT	0.706	0.630	0.646	0.622
GPT-4o	0.777	0.755	0.671	0.863
HateMM	0.805	0.753	0.765	0.751
CMFusion	0.799	0.739	0.763	0.719
MultiHateGNN	0.821	0.771	0.798	0.754

On HateMM, MultiHateGNN is best on all metrics except recall, where GPT-4o is higher but with much worse precision (Yue et al., 16 Sep 2025).

MHC results

Model	Accuracy	F1	Precision	Recall
ViViT	0.73	0.68	0.86	0.57
MFCC	0.54	0.36	0.40	0.33
mBERT	0.57	0.52	0.68	0.42
GPT-4o	0.68	0.29	0.57	0.19
MHC	0.75	0.67	0.77	0.61
CMFusion	0.73	0.72	0.72	0.73
MultiHateGNN	0.78	0.77	0.80	0.77

On MHC, the paper states that MultiHateGNN improves the prior best accuracy by 3% and F1 by 5% (Yue et al., 16 Sep 2025).

The ablation study on HateMM isolates the contribution of each stream:

Model	Accuracy	F1	Precision	Recall
No Graph	0.792	0.742	0.759	0.727
Only Instance Graph	0.798	0.746	0.767	0.726
Only Weight Graph	0.813	0.756	0.789	0.738
Full Model	0.821	0.771	0.798	0.754

The reported interpretation is that No Graph performs worst, Only Weight Graph is the strongest single-stream baseline, and the full dual-stream model performs best, indicating that content extraction and weighting are complementary (Yue et al., 16 Sep 2025).

6. Explainability, strengths, and limitations

A major claim of the paper is that MultiHateGNN offers explainability through the estimated instance importance weights $\{\alpha_i\}_{i=1}^K$8. Because prediction uses

$\{\alpha_i\}_{i=1}^K$9

the learned weights directly indicate which temporal instances contributed most to the final decision. The authors state that these instance weights are helpful for localization of hateful content, although the model is not yet a full temporal localization system (Yue et al., 16 Sep 2025).

The qualitative example provided in the paper illustrates the intended behavior. In a HateMM case, a video is divided into 10 instances, but only instance 3 contains hateful content. Existing methods struggle because the other 9 instances are non-hateful and dominate the video. MultiHateGNN assigns instance 3 a high weight,

$\{f_i\}_{i=1}^K$0

which allows correct hateful classification. This makes the explainability claim concrete: the model’s weighting stream is not merely auxiliary, but directly tied to the final video representation (Yue et al., 16 Sep 2025).

The paper and its technical summary also delineate several limitations. The method depends on segmentation quality; poor segmentation could dilute or split critical cues. It may be sensitive to ASR quality because the text modality relies on Whisper transcripts. The $\{f_i\}_{i=1}^K$1-graph similarity construction may increase computational cost as $\{f_i\}_{i=1}^K$2 grows. Exact reproduction from the main text alone is difficult because the precise GNN architecture, $\{f_i\}_{i=1}^K$3, $\{f_i\}_{i=1}^K$4, optimizer, and training schedule are deferred to supplementary material. Finally, while the model provides instance-level explanations, it is not yet a full localization framework; the conclusion suggests future work on multi-scale localization. Code availability is explicitly stated at https://github.com/Multimodal-Intelligence-Lab-MIL/MultiHateGNN (Yue et al., 16 Sep 2025).

7. Position within related graph-based hate research

MultiHateGNN belongs to a broader landscape of graph-based hate research, but it occupies a specific niche. It is a multimodal video classifier that models structured relations across visual, audio, and transcript streams inside a video. This differs from graph methods designed for multi-relation social diffusion or user-level hate-monger detection (Yue et al., 16 Sep 2025).

One adjacent line is “Convolutional Learning on Multigraphs,” which develops a formal framework for multigraph signal processing and introduces Multigraph Neural Networks (MGNNs) for data supported on multiple edge classes over a shared node set. That paper is validated on a hate speech source localization task on Twitter, using a follow graph and a retweet graph, and shows improved performance over parallel and merged graph baselines (Butler et al., 2022). The relation to MultiHateGNN is methodological rather than task-identical: MGNN is a multi-relation propagation framework for social-network diffusion, whereas MultiHateGNN is a dual-stream multimodal graph architecture for localized hateful evidence in videos. A plausible implication is that both methods respond to the same general problem—single-structure modeling can obscure crucial heterogeneous relations—but they instantiate that principle in different substrates.

Another nearby line is “Social Hatred: Efficient Multimodal Detection of Hatemongers,” which targets user/account-level hate-monger detection by aggregating post-level textual signals with social-network context across Twitter/X, Gab, and Parler. That paper is explicitly not itself a graph neural network; its graph component is a hand-constructed one-hop relational aggregation rather than learned message passing (Marzea et al., 24 Jun 2025). In contrast, MultiHateGNN learns graph-based representations directly over multimodal video segments and instances. The comparison is useful because it clarifies that “multimodal hate detection” spans at least three distinct units of analysis: video-level classification, social-diffusion localization, and user-level hate-monger detection. MultiHateGNN is specifically the video-level, dual-stream GNN instantiation among these possibilities.