HBMNet: Hierarchical Audio-Visual Localization

• HBMNet is a hierarchical boundary modeling network that uses multiscale, bidirectional cues from audio and visual streams to detect deepfake boundaries in partially manipulated videos.
• It integrates an Audio-Visual Feature Encoder, Coarse Proposal Generator, and Fine-grained Probabilities Generator to fuse coarse proposals with fine-grained frame-level predictions for enhanced precision and recall.
• Empirical results on challenging datasets show that HBMNet outperforms previous methods, highlighting its robust scalability, data efficiency, and advanced boundary-content modeling.

HBMNet is a hierarchical boundary modeling network introduced for audio-visual temporal deepfake localization under the scenario of content-driven partial manipulation, where manipulated (“deepfake”) regions typically span only a few frames while the majority of the video remains unaltered. The HBMNet framework leverages three integrated modules—an Audio-Visual Feature Encoder, a Coarse Proposal Generator, and a Fine-grained Probabilities Generator—to capture multiscale, bidirectional, and cross-modal cues for precise and robust boundary localization of deepfake regions. Its architecture induces complementary strengths from both modal (audio and visual) and temporal (multiscale, boundary-aware) perspectives and achieves superior performance on challenging temporally localized manipulation detection tasks (Chen et al., 4 Aug 2025).

1. Architecture and Module Design

1.1 Audio–Visual Feature Encoder (AVFE)

HBMNet’s AVFE first constructs bidirectional representations from audio and visual inputs. Each raw stream—audio $X^a$ and video $X^v$—is duplicated as forward ($\overrightarrow{X}$) and reversed ($\overleftarrow{X}$) streams, enabling the model to acquire cues about both “real → fake” and “fake → real” transitions.

• Audio Encoder: Converts mel-spectrograms with a SENet.
• Video Encoder: Employs a 3D-CNN stem, ResNet-18, and Temporal Convolutional Networks (TCN).
• Cross-modal Fusion: Three transformer-style blocks perform:
  1. Audio-to-visual (AV) cross-attention,
  2. Visual-to-audio (VA) cross-attention,
  3. Self-attention fusion (AttFUS).

• Frame-wise Classification: A lightweight classifier provides per-frame “fake vs real” probabilities $P^f\in \mathbb{R}^T$.

• Feature Concatenation and Frame-level Supervision: Concatenation of features and probabilities forms cross-modal features ($\overrightarrow{F}^{cf}$, $\overleftarrow{F}^{cf}$). Frame-level contrastive supervision maximizes the audio–visual feature overlap on genuine frames and separation on manipulated frames by means of a margin-based contrastive loss $\mathcal L_{\rm FC}$.

1.2 Coarse Proposal Generator (CPG)

CPG implements a proposal-based boundary matching mechanism inspired by BMN. It receives forward stream features $\overrightarrow{F}^{cf}$ and generates a dense grid of candidate proposals, each characterized by a start frame $j$ and duration $X^v$0:

• Proposal Sampling and Confidence Estimation: For each proposal $X^v$1, a learnable sampling mask $X^v$2 extracts relevant feature snippets, and 3D-to-2D convolutions yield a proposal confidence map $X^v$3.
• Supervision: CPG is trained using Mean Squared Error loss $X^v$4 against a “ground-truth” IoU-based map $X^v$5 representing the maximum overlap with any true manipulated segment.

1.3 Fine-grained Probabilities Generator (FPG)

FPG refines region boundaries:

• Structure: A nested U-Net accepts both $X^v$6 and $X^v$7, extracting multiscale cues.
• Outputs: For each direction, FPG predicts six sequences—start ($X^v$8), end ($X^v$9), and content ($\overrightarrow{X}$0) probabilities.
• Bidirectional Fusion: Forward and backward predictions are combined via elementwise geometric mean at inference.
• Proposal Scoring: The final score for each proposal $\overrightarrow{X}$1 is the product of CPG’s confidence, fused start and end probabilities, and mean content probability over the proposal.
• Supervision: FPG loss $\overrightarrow{X}$2 uses a Focal variant on anchor-based start, end, and content predictions in both directions.

2. Hierarchical and Bidirectional Boundary Modeling

HBMNet unifies proposal-level and frame-level localization by explicitly modeling hierarchical boundaries and leveraging bidirectional context:

• Multi-scale Temporal Cues: The model integrates CPG’s global, coarse proposals and FPG’s fine, frame-level predictions. During both training and inference, aggregating these cues yields an effective trade-off between precision and recall.
• Bidirectional Boundary–Content Relationships: By processing time-reversed and forward streams, HBMNet can independently capture “entering” and “exiting” manipulated regions. This enables the system to model transition signals that unidirectional frameworks cannot, as FPG directly predicts start/end/content likelihoods in both directions and fuses them at inference.

3. Objective Functions and Optimization

HBMNet is trained to optimize a combination of frame-level, proposal-level, and content classification objectives:

• Total Loss:

$\overrightarrow{X}$3

with $\overrightarrow{X}$4.

• Audio-Visual Alignment: $\overrightarrow{X}$5 enforces discriminability by minimizing cross-modal feature distances on genuine frames and maximizing them on fakes.
• Proposal-level Regression: $\overrightarrow{X}$6 (MSE) drives the CPG to regress towards dense IoU targets.
• Frame-level Localization: $\overrightarrow{X}$7 uses Focal loss (with adjustments for sample imbalance and hardness) on start/end/content streams to sharpen boundary precision.

4. Evaluation Protocol and Empirical Results

Experiments are conducted on the AV-Deepfake-1M dataset (~2K subjects, 286K real videos, 860K manipulations, diverse in audio/video/both), using a computationally tractable subset (8,000 train, 1,000 val, 2,000 test). Evaluation metrics include Average Precision (AP) at IoU thresholds {0.5, 0.75, 0.95} and Average Recall (AR) at {50, 20, 10} proposals. Key results and ablation findings are summarized below.

Method	[email protected]	[email protected]	[email protected]	AR@50	AR@20	AR@10
BA-TFD	38.9	11.8	0.10	42.6	36.3	32.0
UMMAFormer	75.9	60.9	9.21	78.3	74.2	70.7
HBMNet	96.8	90.8	12.8	88.8	88.5	88.3

Ablation insights:

• Upgrading to cross-attentional AVFE with contrastive and fusion mechanisms increases [email protected] from 38.9 to 95.8 and AR@50 from 42.6 to 70.4 over BA-TFD.
• Individual audio-only or video-only models achieve ~64–66% [email protected]; audio-visual fusion reaches 96.5%.
• CPG dominates at [email protected] (95.8) but has negligible [email protected], while FPG improves [email protected] (9.2); combining both achieves best-in-class results for both metrics ([email protected] = 96.5, [email protected] = 12.7).
• Bidirectionality and explicit content-aware loss increase [email protected] from ~6.8 to 12.7.
• Scalability is demonstrated: [email protected] improves from 88.07 (on one-third of the data) to 97.88 (on threefold enlarged data), and [email protected] scales proportionally, confirming data efficiency and learning capacity.

5. Insights, Limitations, and Directions for Advancement

Insights

• Precision and Recall Boosting: Dedicated cross-attention fusion sharpens precision, while frame-level contrastive supervision improves recall.
• Complementarity of Hierarchical and Bidirectional Modeling: Integrating proposal- and frame-level, as well as forward- and backward-streams, delivers additive and complementary performance gains.
• Content-aware Boundaries: The explicit modeling of boundary-content relationships (start, end, and interior) mitigates misclassification on ambiguous or noisy transitions.

Limitations

• The nested U-Net and BMN sampling in CPG/FPG can incur significant computational overhead, particularly for long-duration inputs.
• Bidirectional encoding doubles computational requirements relative to unidirectional frameworks.
• Experiments and design focus on English speech; transfer to non-English or more diverse AV contexts remains untested.

Future Directions

• More efficient boundary-matching mechanisms or dynamical (e.g., adaptive) proposal sampling could enhance scalability.
• Lightweight bidirectional architectures (e.g., single-pass bidirectional Transformers) may reduce processing costs.
• Semi-supervised or self-supervised pretraining on large, unlabeled AV streams is identified as a route to further leverage unannotated data and improve generalizability.

6. Context and Significance

HBMNet’s unified architecture—melding robust audio-visual encoding, multi-scale hierarchical localization, and explicit bidirectional boundary-content modeling—demonstrates substantial advances in the localized detection of temporally sparse deepfake regions. Its design delivers complementary strengths in both precision and recall, and its scalability with larger datasets suggests practical utility in high-volume, diverse deployment scenarios. HBMNet outperforms prior methods, such as BA-TFD and UMMAFormer, by large margins across all principal evaluation metrics, advancing the state of the art in partial, content-driven temporal deepfake localization (Chen et al., 4 Aug 2025).