---
title: 'HBMNet: Hierarchical Audio-Visual Localization'
url: https://www.emergentmind.com/topics/hbmnet
type: topic
---

# HBMNet: Hierarchical Audio-Visual Localization

HBMNet is a hierarchical boundary modeling network introduced for audio-visual temporal deepfake localization under the scenario of content-driven partial manipulation, where manipulated (“deepfake”) regions typically span only a few frames while the majority of the video remains unaltered. The HBMNet framework leverages three integrated modules—an Audio-Visual Feature Encoder, a Coarse Proposal Generator, and a Fine-grained Probabilities Generator—to capture multiscale, bidirectional, and cross-modal cues for precise and robust boundary localization of deepfake regions. Its architecture induces complementary strengths from both modal (audio and visual) and temporal (multiscale, boundary-aware) perspectives and achieves superior performance on challenging temporally localized manipulation detection tasks [2508.02000].

## 1. Architecture and Module Design

### 1.1 Audio–Visual Feature Encoder (AVFE)

HBMNet’s AVFE first constructs bidirectional representations from audio and visual inputs. Each raw stream—audio $X^a$ and video $X^v$—is duplicated as forward ($\overrightarrow{X}$) and reversed ($\overleftarrow{X}$) streams, enabling the model to acquire cues about both “real → fake” and “fake → real” transitions.

- **Audio Encoder**: Converts mel-spectrograms with a SENet.
- **Video Encoder**: Employs a 3D-CNN stem, ResNet-18, and Temporal Convolutional Networks (TCN).
- **Cross-modal Fusion**: Three transformer-style blocks perform:
  1. Audio-to-visual (AV) cross-attention,
  2. Visual-to-audio (VA) cross-attention,
  3. Self-attention fusion (AttFUS).
- **Frame-wise Classification**: A lightweight classifier provides per-frame “fake vs real” probabilities $P^f\in \mathbb{R}^T$.
- **Feature Concatenation and Frame-level Supervision**: Concatenation of features and probabilities forms cross-modal features ($\overrightarrow{F}^{cf}$, $\overleftarrow{F}^{cf}$). Frame-level contrastive supervision maximizes the audio–visual feature overlap on genuine frames and separation on manipulated frames by means of a margin-based contrastive loss $\mathcal L_{\rm FC}$.

### 1.2 Coarse Proposal Generator (CPG)

CPG implements a proposal-based boundary matching mechanism inspired by BMN. It receives forward stream features $\overrightarrow{F}^{cf}$ and generates a dense grid of candidate proposals, each characterized by a start frame $j$ and duration $i$:

- **Proposal Sampling and Confidence Estimation**: For each proposal $(i,j)$, a learnable sampling mask $W$ extracts relevant feature snippets, and 3D-to-2D convolutions yield a proposal confidence map $M'$.
- **Supervision**: CPG is trained using Mean Squared Error loss $\mathcal L_{\rm CP}$ against a “ground-truth” IoU-based map $M$ representing the maximum overlap with any true manipulated segment.

### 1.3 Fine-grained Probabilities Generator (FPG)

FPG refines region boundaries:

- **Structure**: A nested U-Net accepts both $\overrightarrow{F}^{cf}$ and $\overleftarrow{F}^{cf}$, extracting multiscale cues.
- **Outputs**: For each direction, FPG predicts six sequences—start ($P^{s'}$), end ($P^{e'}$), and content ($P^{c'}$) probabilities.
- **Bidirectional Fusion**: Forward and backward predictions are combined via elementwise geometric mean at inference.
- **Proposal Scoring**: The final score for each proposal $(i,j)$ is the product of CPG’s confidence, fused start and end probabilities, and mean content probability over the proposal.
- **Supervision**: FPG loss $\mathcal L_{\rm FP}$ uses a Focal variant on anchor-based start, end, and content predictions in both directions.

## 2. Hierarchical and Bidirectional Boundary Modeling

HBMNet unifies proposal-level and frame-level localization by explicitly modeling hierarchical boundaries and leveraging bidirectional context:

- **Multi-scale Temporal Cues**: The model integrates CPG’s global, coarse proposals and FPG’s fine, frame-level predictions. During both training and inference, aggregating these cues yields an effective trade-off between precision and recall.
- **Bidirectional Boundary–Content Relationships**: By processing time-reversed and forward streams, HBMNet can independently capture “entering” and “exiting” manipulated regions. This enables the system to model transition signals that unidirectional frameworks cannot, as FPG directly predicts start/end/content likelihoods in both directions and fuses them at inference.

## 3. Objective Functions and Optimization

HBMNet is trained to optimize a combination of frame-level, proposal-level, and content classification objectives:

- **Total Loss**:
  $$
  \mathcal L_{\rm HBMNet} = \alpha\,\mathcal L_{\rm FC} + \mathcal L_{\rm CP} + \mathcal L_{\rm FP}
  $$
  with $\alpha=0.1$.
- **Audio-Visual Alignment**: $\mathcal L_{\rm FC}$ enforces discriminability by minimizing cross-modal feature distances on genuine frames and maximizing them on fakes.
- **Proposal-level Regression**: $\mathcal L_{\rm CP}$ (MSE) drives the CPG to regress towards dense IoU targets.
- **Frame-level Localization**: $\mathcal L_{\rm FP}$ uses Focal loss (with adjustments for sample imbalance and hardness) on start/end/content streams to sharpen boundary precision.

## 4. Evaluation Protocol and Empirical Results

Experiments are conducted on the AV-Deepfake-1M dataset (~2K subjects, 286K real videos, 860K manipulations, diverse in audio/video/both), using a computationally tractable subset (8,000 train, 1,000 val, 2,000 test). Evaluation metrics include Average Precision (AP) at IoU thresholds {0.5, 0.75, 0.95} and Average Recall (AR) at {50, 20, 10} proposals. Key results and ablation findings are summarized below.

| Method           | AP@0.5 | AP@0.75 | AP@0.95 | AR@50 | AR@20 | AR@10 |
|------------------|--------|---------|---------|-------|-------|-------|
| BA-TFD           | 38.9   | 11.8    | 0.10    | 42.6  | 36.3  | 32.0  |
| UMMAFormer       | 75.9   | 60.9    | 9.21    | 78.3  | 74.2  | 70.7  |
| HBMNet           | 96.8   | 90.8    | 12.8    | 88.8  | 88.5  | 88.3  |

**Ablation insights**:
- Upgrading to cross-attentional AVFE with contrastive and fusion mechanisms increases AP@0.5 from 38.9 to 95.8 and AR@50 from 42.6 to 70.4 over BA-TFD.
- Individual audio-only or video-only models achieve ~64–66% AP@0.5; audio-visual fusion reaches 96.5%.
- CPG dominates at AP@0.5 (95.8) but has negligible AP@0.95, while FPG improves AP@0.95 (9.2); combining both achieves best-in-class results for both metrics (AP@0.5 = 96.5, AP@0.95 = 12.7).
- Bidirectionality and explicit content-aware loss increase AP@0.95 from ~6.8 to 12.7.
- Scalability is demonstrated: AP@0.5 improves from 88.07 (on one-third of the data) to 97.88 (on threefold enlarged data), and AP@0.95 scales proportionally, confirming data efficiency and learning capacity.

## 5. Insights, Limitations, and Directions for Advancement

### Insights

- **Precision and Recall Boosting**: Dedicated cross-attention fusion sharpens precision, while frame-level contrastive supervision improves recall.
- **Complementarity of Hierarchical and Bidirectional Modeling**: Integrating proposal- and frame-level, as well as forward- and backward-streams, delivers additive and complementary performance gains.
- **Content-aware Boundaries**: The explicit modeling of boundary-content relationships (start, end, and interior) mitigates misclassification on ambiguous or noisy transitions.

### Limitations

- The nested U-Net and BMN sampling in CPG/FPG can incur significant computational overhead, particularly for long-duration inputs.
- Bidirectional encoding doubles computational requirements relative to unidirectional frameworks.
- Experiments and design focus on English speech; transfer to non-English or more diverse AV contexts remains untested.

### Future Directions

- More efficient boundary-matching mechanisms or dynamical (e.g., adaptive) proposal sampling could enhance scalability.
- Lightweight bidirectional architectures (e.g., single-pass bidirectional Transformers) may reduce processing costs.
- Semi-supervised or self-supervised pretraining on large, unlabeled AV streams is identified as a route to further leverage unannotated data and improve generalizability.

## 6. Context and Significance

HBMNet’s unified architecture—melding robust audio-visual encoding, multi-scale hierarchical localization, and explicit bidirectional boundary-content modeling—demonstrates substantial advances in the localized detection of temporally sparse deepfake regions. Its design delivers complementary strengths in both precision and recall, and its scalability with larger datasets suggests practical utility in high-volume, diverse deployment scenarios. HBMNet outperforms prior methods, such as BA-TFD and UMMAFormer, by large margins across all principal evaluation metrics, advancing the state of the art in partial, content-driven temporal deepfake localization [2508.02000].

Source: https://www.emergentmind.com/topics/hbmnet