SV-FEND: Multimodal Short Video Fake News Model

• SV-FEND is a multimodal fake news detection model that combines text, audio, visuals, and social cues to address the heterogeneous nature of short videos.
• It employs co-attention and self-attention mechanisms to selectively integrate cross-modal correlations and enhance context understanding.
• Empirical results on the FakeSV dataset demonstrate significant accuracy improvements over prior single- and multi-modal approaches.

SV-FEND (Short Video Fake news dEtectioN moDel) is a multimodal neural architecture developed for the detection of fake news on short video platforms, specifically addressing the challenges posed by multimodal, heterogeneous, and context-rich content typical to services such as Douyin and Kuaishou. SV-FEND explicitly models cross-modal correlations and integrates social context, surpassing prior single- and multi-modal approaches by selecting informative features and leveraging cues from user interaction and publisher profiles. The model is introduced alongside FakeSV, the largest Chinese short video fake news dataset to date, in the context of practical detection tasks involving news videos, comments, and social metadata (Qi et al., 2022).

1. Motivation and Challenges

SV-FEND targets two significant challenges in short-video fake news detection:

1. Modal-Rich but Noisy Content: Short-video news is comprised of six inherently heterogeneous modalities—textual title and transcript, audio track, static frames, motion clips, user comments, and publisher profile. Prior approaches that simply concatenate modality features risk overlooking cross-modal signaling and may overfit to uninformative components. SV-FEND addresses this by employing co-attention modules for explicit pairwise correlation modeling and selective feature retention.
2. Weak Visual Discriminability: The prevalence of advanced video editing (text-boxes, filters, splicing) in both legitimate and fake posts diminishes the discriminative power of visual content alone. SV-FEND supplements content-based representations with social context, including weighted user comments and publisher metadata, integrating these in a unified self-attention mechanism. This enables skeptical commentary or signals of low publisher authority to compensate where content is ambiguous.

2. Architecture Overview

SV-FEND's architecture comprises four sequential components:

1. Multimodal Feature Extraction:
   • Textual encoding via BERT for combined title and transcript.
   • Audio encoding via VGGish for mel-spectrogram patches.
   • Static frame features via VGG19.
   • Motion clips encoded by C3D.
   • User comment embeddings via BERT, weighted by up-votes.
   • Publisher profile embedding via BERT on verified/self-introduction text.
2. Cross-Modal Correlation (Co-Attention):
   • Two-stream co-attention Transformer blocks, first between text and audio, then between the text-enhanced representation and static frames.
   • Encoded outputs are cross-filtered representations, enhancing informative synergy among pairs.
3. Social-Context Fusion (Self-Attention):
   • Single-vector collapse via average pooling for each content modality post co-attention.
   • Concatenation of all six modalities (three content, three social), with dimension equalization to $d_m = 768$, as input to a standard Transformer encoder layer (self-attention).
   • Production of a unified fused vector via sequence average pooling.
4. Classification:
   • Output fused vector is processed by a one-hidden-layer MLP and final softmax for binary real/fake probability prediction.

Stage	Input Modalities	Encoder(s)
Feature Extraction	Title+Transcript, Audio, Static Frames, Motion Clips	BERT, VGGish, VGG19, C3D
Social Context	User Comments, Publisher Profile	BERT
Correlation/Fusion	All modalities	Co-Attention, Self-Attention Transformers

3. Modality Encoding and Input Construction

Each modality is independently pre-encoded to form vector sequences of standard dimension:

• Text (Title + Transcript): Concatenation of short title (≤33 tokens) and transcript (≤211 tokens), BERT-encoded to $H_T \in \mathbb{R}^{l \times 768}$.
• Audio: Up to 50 mel-spectrogram patches, with each patch encoded via VGGish to $H_A \in \mathbb{R}^{n \times 128}$.
• Static Frames: ≤83 uniformly sampled video frames, with VGG19 (fc7) outputting $H_I \in \mathbb{R}^{m \times 4096}$.
• Motion Clips: For each frame, a corresponding 16-frame clip is encoded via C3D (fc7), yielding $H_V \in \mathbb{R}^{m \times 4096}$, with mean pooling to a single vector $x_V$.
• Comments: Top $k=23$ comments, BERT-encoded and weighted by up-votes as $x_C = \sum_{j=1}^k \alpha_j c_j$, where $\alpha_j = (\ell_j+1) / (\sum_{t=1}^k \ell_t + k)$.
• Publisher Profile: Self-introduction and verified status, BERT-encoded; the [CLS] output forms $x_U \in \mathbb{R}^{768}$.

All vectors are either zero-padded or linearly projected to standardized dimensions when aggregated for self-attention.

4. Cross-Modal Correlation and Feature Selection

SV-FEND’s co-attention is adapted from ViLBERT, instantiated as single- or multi-head attention:

For text and audio features projected to a common dimension $H_T \in \mathbb{R}^{l \times 768}$0, co-attention computes:

$H_T \in \mathbb{R}^{l \times 768}$1

$H_T \in \mathbb{R}^{l \times 768}$2

$H_T \in \mathbb{R}^{l \times 768}$3

with analogous formulation for audio attending to text, and subsequently, text to static-frame co-attention. The key effect is selective amplification of modality pairs exhibiting informative correlations.

After co-attention, feature selection is achieved by average pooling over the sequence dimension:

$H_T \in \mathbb{R}^{l \times 768}$4

yielding compact, cross-modal-filtered representations for downstream fusion.

5. Social Context Integration and Fusion

The three content vectors ($H_T \in \mathbb{R}^{l \times 768}$5, $H_T \in \mathbb{R}^{l \times 768}$6, $H_T \in \mathbb{R}^{l \times 768}$7) and three social vectors ($H_T \in \mathbb{R}^{l \times 768}$8, $H_T \in \mathbb{R}^{l \times 768}$9, $H_A \in \mathbb{R}^{n \times 128}$0) are concatenated as

$H_A \in \mathbb{R}^{n \times 128}$1

This sequence is input to a one-layer Transformer encoder with self-attention and feed-forward sublayers, producing $H_A \in \mathbb{R}^{n \times 128}$2. The final fused feature is obtained via average pooling across the six positions:

$H_A \in \mathbb{R}^{n \times 128}$3

This architecture allows global interactions among content and social modalities, enabling the model to aggregate both original and contextual cues.

6. Training Regimen and Objective

SV-FEND employs binary cross-entropy loss over the softmax outputs:

$H_A \in \mathbb{R}^{n \times 128}$4

with $H_A \in \mathbb{R}^{n \times 128}$5 as the binary label. The total objective contains only this classification loss; no auxiliary, alignment, or adversarial losses are introduced.

Key Hyperparameters:

• Optimizer: Adam ($H_A \in \mathbb{R}^{n \times 128}$6, $H_A \in \mathbb{R}^{n \times 128}$7)
• Learning rate: $H_A \in \mathbb{R}^{n \times 128}$8
• Batch size: 16 (balanced)
• Training epochs: up to 30, early stopping on dev set
• Co-attention heads: $H_A \in \mathbb{R}^{n \times 128}$9, dimension $H_I \in \mathbb{R}^{m \times 4096}$0
• Self-attention heads: $H_I \in \mathbb{R}^{m \times 4096}$1
• Max: $H_I \in \mathbb{R}^{m \times 4096}$2 frames, $H_I \in \mathbb{R}^{m \times 4096}$3 audio patches, $H_I \in \mathbb{R}^{m \times 4096}$4 comments

This setup is designed to balance effective learning of cross-modal and contextual signals with robust generalization.

7. Empirical Performance and Ablations

On the FakeSV benchmark (five-fold event-split cross-validation), SV-FEND achieves:

• Accuracy: $H_I \in \mathbb{R}^{m \times 4096}$5
• Macro-F1: $H_I \in \mathbb{R}^{m \times 4096}$6

SV-FEND surpasses all single-modality baselines (text-BERT: $H_I \in \mathbb{R}^{m \times 4096}$7) and previously leading multimodal approaches (best prior: $H_I \in \mathbb{R}^{m \times 4096}$875.07\%).

Ablation results demonstrate:

• Removal of all news content: accuracy drops to $H_I \in \mathbb{R}^{m \times 4096}$9 (−4.42).
• Removal of all social context: accuracy drops to $H_V \in \mathbb{R}^{m \times 4096}$0 (−0.69).
• Content ablation: text removal has the greatest effect ($H_V \in \mathbb{R}^{m \times 4096}$1), followed by visual (frames+clips, $H_V \in \mathbb{R}^{m \times 4096}$2), then audio ($H_V \in \mathbb{R}^{m \times 4096}$3).
• Social ablation: user profile removal yields $H_V \in \mathbb{R}^{m \times 4096}$4, comments $H_V \in \mathbb{R}^{m \times 4096}$5.

Temporal splits (train on earlier $H_V \in \mathbb{R}^{m \times 4096}$6 of events, test on future $H_V \in \mathbb{R}^{m \times 4096}$7) yield $H_V \in \mathbb{R}^{m \times 4096}$8 accuracy, indicating robustness to emerging fake news events.

Summary Table: SV-FEND vs. Baselines (FakeSV Benchmark)

Model	Accuracy (%)	Macro-F1 (%)
SV-FEND	79.31	79.24
Best prior (multi-modal)	~75.07	–
Best text (BERT)	76.82	–

The empirical results verify the utility of two-stage attention mechanisms for selecting and integrating informative cues from both multimodal content and social context. Removal of individual components quantifies their unique contribution, with textual content being most critical among content features and publisher profile among social signals.

---

In totality, SV-FEND advances multimodal fake news detection on short video platforms via hierarchical attention and context-aware social signal integration, establishing a new state of the art on the FakeSV dataset (Qi et al., 2022).