---
title: 'HBMNet: Hierarchical Deepfake Localization'
url: https://www.emergentmind.com/topics/hierarchical-boundary-modeling-network-hbmnet
type: topic
---

# HBMNet: Hierarchical Deepfake Localization

Searching arXiv for the specified HBMNet paper and closely related hierarchical boundary modeling work.
arxiv_search.query({"search_query":"id:2508.02000 OR ti:\"Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modeling\"","start":0,"max_results":5})
Hierarchical Boundary Modeling Network (HBMNet) is an end-to-end framework for **audio-visual temporal deepfake localization** under **content-driven partial manipulation**, introduced in “Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modeling” [2508.02000]. Rather than deciding whether an entire clip is fake, HBMNet localizes **when** manipulated content occurs, with particular emphasis on the difficult regime in which fake content appears only in short, sparse temporal fragments while most frames remain authentic. Its central design combines stronger audio-visual representation learning with **hierarchical boundary modeling** across two temporal granularities: a **coarse proposal-level stage** that scans candidate segments globally and a **fine frame-level stage** that sharpens start and end boundaries and content probabilities using bidirectional temporal cues.

## 1. Task formulation and problem setting

HBMNet is defined for **Audio-Visual Temporal Deepfake Localization (AVTDL)** under **content-driven partial manipulation** [2508.02000]. The input consists of an audio sequence \(X^a\) and a visual sequence \(X^v\). The ground truth is a set of temporal segments
\[
\Psi = \{\varphi_n = (t_n^s, t_n^e)\}_{n=1}^N,
\]
where each \(\varphi_n\) denotes a temporal segment with start time \(t_n^s\) and end time \(t_n^e\). A segment is labeled **real** iff **both** audio and video are unmodified throughout \([t_n^s, t_n^e]\); otherwise it is labeled **fake**. The model predicts localized fake proposals
\[
\Psi' = \{\varphi_n' = (t_n^{\prime s}, t_n^{\prime e})\}_{n=1}^{N'},
\]
and during inference each proposal carries a confidence score, yielding
\[
\Psi' = \{(t_n^s, t_n^e, s_n')\}_{n=1}^{N'}.
\]

The setting differs from full-video forgery detection because the manipulations are tied to edits such as **replacement**, **deletion**, and **insertion** in audio, video, or both. Consequently, deepfake spans often cover **only a few frames**, and most frames remain identical to the original. The problem is therefore closer to **temporal action localization / proposal generation** than to ordinary video classification. A common misconception is to treat HBMNet as a binary deepfake detector; in the formal setup, the relevant question is the temporal location, start boundary, end boundary, and confidence of each manipulated segment.

HBMNet produces localization signals at multiple temporal levels. The **Coarse Proposal Generator** outputs a proposal-level confidence map
\[
M' \in \mathbb{R}^{L \times T},
\]
whose cells correspond to candidate segments indexed by start time and duration. The **Fine-grained Probabilities Generator** outputs frame-level **start probability**, **end probability**, and **content probability** in both forward and backward directions. The final scored temporal proposals combine proposal-level and frame-level evidence as
\[
s' = M'_{i,j}\cdot \overleftrightarrow{P}^{s'}_j \cdot \overleftrightarrow{P}^{e'}_{j+i}\cdot \mathrm{mean}\!\left(\overleftrightarrow{P}^{c'}_{j:j+i}\right).
\]

## 2. Architectural organization and the meaning of hierarchical boundary modeling

HBMNet comprises three principal components: an **Audio-Visual Feature Encoder (AVFE)**, a **Coarse Proposal Generator (CPG)**, and a **Fine-grained Probabilities Generator (FPG)** [2508.02000]. In pipeline form,
\[
(X^a, X^v) \rightarrow \text{bidirectional inputs} \rightarrow \text{single-modality encoders} \rightarrow \text{cross-modal attention fusion} \rightarrow F^{cf} \rightarrow
\begin{cases}
\text{CPG} \rightarrow M'\\
\text{FPG} \rightarrow (P^{s'}, P^{e'}, P^{c'})
\end{cases}
\rightarrow \text{proposal scoring + Soft-NMS} \rightarrow \Psi'.
\]
The original and time-reversed audio and video streams are both processed. Audio and video are encoded separately, fused through cross-attention and self-attention, passed through a frame-wise classifier producing \(P^f\), and then concatenated into enriched features \(F^{cf}\) that feed the proposal-level and frame-level branches.

The expression **hierarchical boundary modeling** has three distinct meanings in the system. First, it denotes a **temporal-scale hierarchy**: HBMNet models boundaries at the **proposal level** and at the **frame level**. Second, it denotes a **semantic hierarchy** among **start boundary**, **end boundary**, and **content** probabilities. Third, it includes a **directional hierarchy**, because forward and backward temporal transitions are modeled separately and then fused.

The coarse-to-fine organization is central. The CPG performs global scanning over candidate temporal regions and captures segment extent, while the FPG restores local temporal precision that is otherwise weakened by proposal sampling and aggregation. This suggests that HBMNet treats localization as a joint problem of segment proposal quality and framewise boundary-content consistency, rather than as either pure proposal scoring or pure per-frame classification.

## 3. Audio-visual feature encoding, fusion, and supervision

The AVFE is responsible for discriminative cross-modal frame-level representations [2508.02000]. HBMNet explicitly processes both original and reversed inputs:
\[
\overrightarrow{X^a},\ \overrightarrow{X^v},\ \overleftarrow{X^a},\ \overleftarrow{X^v}.
\]
The stated motivation is that a forward-only model may miss useful post-transition information; reversing the sequence turns fake-to-real transitions into real-to-fake transitions and vice versa.

The visual branch uses a **3D CNN stem**, followed by **ResNet-18**, followed by **TCN modules**. The audio branch converts the audio stream to **Mel-spectrograms** and encodes them with **SENet**:
\[
X^a \rightarrow \text{Mel-spectrogram} \rightarrow \text{SENet} \rightarrow F^a.
\]
For each direction, AVFE outputs per-frame audio features \(\{\overrightarrow{F^a}, \overleftarrow{F^a}\}\) and visual features \(\{\overrightarrow{F^v}, \overleftarrow{F^v}\}\).

Fusion is **mid-level cross-modal fusion**, not late score fusion. AVFE uses three attention blocks: **Audio-to-Visual (AV) cross-attention**, **Visual-to-Audio (VA) cross-attention**, and an **AttFUS** self-attention fusion block. The resulting directional features are fused into \(\{\overrightarrow{F^f}, \overleftarrow{F^f}\}\). A frame-wise classifier then produces
\[
P^f \in \mathbb{R}^T,
\]
and the model concatenates fused features with frame probabilities:
\[
\overrightarrow{F^{cf}} = \overrightarrow{F^f} \oplus \overrightarrow{P^f}, \qquad
\overleftarrow{F^{cf}} = \overleftarrow{F^f} \oplus \overleftarrow{P^f}.
\]

A notable contribution is **frame-level contrastive supervision**. The frame label sequence is
\[
Y = \{y_t\}_{t=1}^T,
\]
with a frame labeled **real** iff both audio and visual are real at that frame, and **fake** otherwise. The cross-modal feature distance at frame \(t\) is
\[
d_t = \left\|\overrightarrow{F^{av}_t} - \overrightarrow{F^{va}_t}\right\|_2 + \left\|\overleftarrow{F^{av}_t} - \overleftarrow{F^{va}_t}\right\|_2.
\]
The model applies a margin-based **contrastive loss** \(\mathcal{L}_{\mathrm{FC}}\), whose intended effect is to pull cross-modal representations closer on real frames and push them apart on fake frames. The reported ablation states that encoding and fusion mainly improve **precision**, whereas frame-level supervision mainly improves **recall**. After adding frame-level supervision following encoder replacement, **AR@50** increases from **69.03** to **70.95**, **AR@20** from **66.56** to **68.91**, **AR@10** from **65.80** to **68.10**, and **AP@0.75** from **58.05** to **62.30**. This supports the paper’s claim that frame-level supervision reduces misses on short fake fragments.

## 4. Coarse proposal generation, fine-grained probabilities, and inference

The **Coarse Proposal Generator** is the proposal-level branch. Given
\[
\overrightarrow{F^{cf}} \in \mathbb{R}^{C \times T},
\]
it predicts
\[
M' \in \mathbb{R}^{L \times T},
\]
where \(T\) is the number of possible start times and \(L\) is the maximum proposal duration [2508.02000]. Each entry \(M'_{i,j}\) corresponds to a proposal
\[
\varphi_{i,j} = (t^s=t_j,\ t^e=t_j+t_i).
\]
CPG is built on a **Boundary-Matching (BM) layer** from BMN. It constructs a sampling mask
\[
W \in \mathbb{R}^{N \times T \times L \times T},
\]
whose per-proposal masks softly sample interval features, producing a boundary-matching tensor
\[
M_F \in \mathbb{R}^{C \times N \times L \times T}.
\]
This tensor is processed by **3D and 2D convolutions** to obtain \(M'\). Proposal supervision is dense and soft:
\[
M \in [0,1]^{L \times T}, \qquad
M_{i,j} = \max_{\varphi \in \Psi}\mathrm{IoU}(\varphi^a_{i,j}, \varphi),
\]
and the loss is
\[
\mathcal{L}_{\mathrm{CP}} = \mathrm{MSE}(M', M).
\]

The **Fine-grained Probabilities Generator** is the frame-level refinement branch. It is based on **Nested U-Net (U-Net++)**, takes \(\{\overrightarrow{F^{cf}}, \overleftarrow{F^{cf}}\}\) as input, downsamples through **two pooling stages** with interleaved convolutional units, upsamples using **skip connections**, and predicts forward and backward triplets
\[
\overrightarrow{P'} = \{\overrightarrow{P^{s'}}, \overrightarrow{P^{e'}}, \overrightarrow{P^{c'}}\}, \qquad
\overleftarrow{P'} = \{\overleftarrow{P^{s'}}, \overleftarrow{P^{e'}}, \overleftarrow{P^{c'}}\},
\]
with each in \(\mathbb{R}^{3 \times T}\). The three channels correspond to **start**, **end**, and **fake-content** probability sequences.

Frame-level labels are built by defining a temporal region
\[
r(t)=\left[t-\frac{d^f}{2},\, t+\frac{d^f}{2}\right],
\]
then deriving start, end, and content regions \(r^s\), \(r^e\), and \(r^c\). For each temporal anchor
\[
a_t = [a_t^{\min}, a_t^{\max}] = [d^f(t-0.5),\, d^f(t+0.5)],
\]
the labels are
\[
p_t^s = \max_k \mathrm{IoA}(a_t, r_k^s), \qquad
p_t^e = \max_k \mathrm{IoA}(a_t, r_k^e), \qquad
p_t^c = \max_k \mathrm{IoA}(a_t, r_k^c),
\]
where
\[
\mathrm{IoA}(a_t, r_k)=\frac{\mathrm{dur}(a_t\cap r_k)}{\mathrm{dur}(a_t)}.
\]
The FPG objective uses **Focal loss**:
\[
\mathrm{Focal}(P',P) = -\beta (1-P)^{\beta_1}\,\mathrm{BCE}(P',P),
\]
with
\[
\beta = \beta_0 P + (1-\beta_0)(1-P),
\]
and
\[
\mathcal{L}_{\mathrm{FP}} = \sum_{b\in\{s,e,c\}} \mathrm{Focal}(\overrightarrow{P}^{b'}, \overrightarrow{P}^{b}) + \mathrm{Focal}(\overleftarrow{P}^{b'}, \overleftarrow{P}^{b}).
\]

During inference, forward and backward frame-level probabilities are fused by geometric mean:
\[
\overleftrightarrow{P'} = \sqrt{\overrightarrow{P'} \times \overleftarrow{P'}}.
\]
The final score for proposal \(\varphi_{i,j}\) multiplies coarse proposal confidence, start confidence, end confidence, and mean content confidence:
\[
s' = M'_{i,j} \cdot \overleftrightarrow{P}^{s'}_j \cdot \overleftrightarrow{P}^{e'}_{j+i} \cdot \mathrm{mean}\!\left(\overleftrightarrow{P}^{c'}_{j:j+i}\right).
\]
**Soft-NMS** then suppresses overlapping proposals. The full training objective is
\[
\mathcal{L}_{\mathrm{HBMNet}} = \alpha \mathcal{L}_{\mathrm{FC}} + \mathcal{L}_{\mathrm{CP}} + \mathcal{L}_{\mathrm{FP}},
\]
with \(\alpha = 0.1\). The hierarchy is empirically complementary: **CPG only** achieves **AP@0.95 = 0.66**, **FPG only** achieves **AP@0.95 = 9.17**, and using **both** yields **AP@0.95 = 12.71**.

## 5. Evaluation, ablations, and empirical behavior

Experiments are conducted on **AV-Deepfake-1M**, which contains **2,068 subjects**, **286,721 real videos**, and **860,039 deepfake videos**, with video-only, audio-only, and combined audio-video manipulation scenarios [2508.02000]. Because of GPU limits, the reported experiments use a subset of **8,000** training videos, **1,000** validation videos, and **2,000** test videos sampled with a fixed random seed. Evaluation uses **Average Precision (AP)** at **AP@0.5**, **AP@0.75**, and **AP@0.95**, and **Average Recall (AR)** at **AR@50**, **AR@20**, and **AR@10**. AR is computed across IoU thresholds from 0.5 to 0.95 with step 0.05.

Against the principal baselines, HBMNet reports the following values:

| Model | AP@0.5 | AP@0.75 | AP@0.95 |
|---|---:|---:|---:|
| BA-TFD | 38.89 | 11.84 | 0.10 |
| UMMAFormer | 75.93 | 60.88 | 9.21 |
| HBMNet | 96.82 | 90.84 | 12.79 |

For recall, **AR@50** is **42.56** for BA-TFD, **78.31** for UMMAFormer, and **88.83** for HBMNet. The paper attributes the gap to stronger audio and video encoders, explicit cross-attention fusion, frame-level supervision, the proposal-plus-frame hierarchy, and bidirectional boundary-content modeling.

Ablations isolate the role of multimodality. **Visual only** yields **AP@0.5 = 65.56**, **AP@0.75 = 57.64**, **AP@0.95 = 5.30**; **Audio only** yields **63.57**, **61.47**, **8.60**; **Audio + visual** yields **96.50**, **92.74**, **12.71**. The reported interpretation is that multimodal fusion is crucial. Ablations over temporal scale show that **CPG only** gives strong coarse localization, whereas **FPG only** improves strict boundary precision and recall, and the combination is best overall. A further ablation over frame-level modeling reports **AP@0.95 = 0.66** with **No S/E, no content, no flip**, **6.80** with **Start/end only**, **6.80** with **Start/end + content**, and **12.71** with **Start/end + content + flip**. The paper highlights that **Flip** raises **AP@0.95** from **6.80** to **12.71**, roughly a **50%** relative increase.

The scalability experiment reports **improved potential scalability with more training data**. With **0.33x data**, the model reaches **AP@0.5 = 88.07**, **AP@0.95 = 10.73**, **AR@50 = 88.29**; with **1x data**, **96.50**, **12.71**, **89.19**; and with **3x data**, **97.88**, **14.58**, **89.22**. This suggests that strict-boundary precision continues to benefit from additional data. The paper does not report cross-dataset generalization. Implementation details include **\(L=60\)**, **\(T=512\)**, **Adam**, initial learning rate **\(10^{-4}\)**, halving the learning rate if validation loss does not decrease for 3 epochs, **40** training epochs, batch size **8**, and training on an **NVIDIA Tesla V100 GPU**.

The paper does not present a dedicated limitations section, but several constraints are explicit or implied: very short and fragmented manipulations remain difficult, noise can induce false positives if boundary cues are overemphasized without content cues, **AP@0.95** remains far below **AP@0.5**, and only a subset of AV-Deepfake-1M is used because of resource constraints.

## 6. Related interpretations in hierarchical sequence modeling and classification

Outside AVTDL, the phrase **Hierarchical Boundary Modeling Network** also appears as an interpretive lens for architectures in which learned boundaries organize hierarchical computation, rather than as a single canonical model name. In “SOMBRERO: Measuring and Steering Boundary Placement in End-to-End Hierarchical Sequence Models” [2601.22805], the relevant abstraction is a hierarchical autoregressive model over **UTF-8 bytes** with an **encoder** \(\mathcal{E}\), **backbone** \(\mathcal{B}\), **decoder** \(\mathcal{D}\), and boundary variables \(p_i\), \(b_i\), and \(c_i\). That work introduces the router-agnostic **boundary enrichment** metric
\[
B = \frac{ \frac{1}{\sum_t b_t}\sum_t b_t\, h_t }{ \frac{1}{T}\sum_t h_t },
\]
where \(h_t\) is next-byte surprisal, together with the **confidence-alignment boundary loss**
\[
\mathcal L_{\text{CAB}} = \frac{1}{T}\sum_{t=1}^{T} \left( 1-\operatorname{sg}\!\left[p_\theta(x_{t+1}\mid x_{1:t})\right]-p_t \right)^2.
\]
At 1B scale, the reported design ladder moves from **H-Net** with \(\text{BPB}=0.6701\), \(B=1.19\), \(C_{\text{emp}}=4.975\) to final **SOMBRERO** with \(\text{BPB}=0.6568\), \(B=3.035\), \(C_{\text{emp}}=4.999\), indicating that in that literature boundary placement is treated as a measurable compute-allocation policy rather than as a localization problem.

“Dynamic Chunking for End-to-End Hierarchical Sequence Modeling” [2507.07955] provides a related but distinct formulation, centered on **H-Net**. There, learned boundaries
\[
p_t = \frac{1}{2} \left(1 - \frac{q_t^\top k_{t-1}}{\left\Vert q_t \right\Vert \left\Vert k_{t-1} \right\Vert}\right), \qquad
b_t = \mathds{1}_{\{p_t \geq 0.5\}},
\]
drive dynamic compression and dechunking, with smoothing
\[
\bar{z}_t = P_t \hat{z}_t + (1-P_t) \bar{z}_{t-1}.
\]
The model is explicitly recursive and supports **1-stage** and **2-stage** hierarchies. On FineWeb-Edu, the reported **2-stage H-Net** attains **0.743 BPB** at large scale and **0.715 BPB** at XL scale, surpassing the corresponding tokenized Transformer baselines (**0.756** and **0.730**). The paper repeatedly presents the system as a learned alternative to tokenization, in which boundaries are content- and context-dependent.

A different line appears in “HiNet: Hierarchical Classification with Neural Network” [1705.11105], where the hierarchy is not temporal but label-structural. HiNet organizes outputs into hierarchical layers, introduces a **stop neuron**, and performs structured **MAP trace** inference with the recurrence
\[
p(T_{a_k}^{(k)}) = \max_{a_{k-1}} p(a_k \mid a_{k-1})\, p(T_{a_{k-1}}^{(k-1)}).
\]
Its training objective is the multi-level loss
\[
E = \sum_{k=1}^n \left\| \tilde{\mathbf{y}^{(k)}} - f_{\theta_k}(\mathbf{X}) \right\|^2,
\]
and the reported parameter complexity is \(O(kn + hn^2)\), contrasted with \(O(kn^h)\) for a flat classifier. On **DMOZ** with **11,947 classes**, **HiNet** reports **41.4** accuracy versus **39.2** for a **Flatten Network**.

Taken together, these works indicate that **HBMNet** is not a universally standardized architecture across domains. In the strict sense established by [2508.02000], it denotes an audio-visual temporal deepfake localization network that combines proposal-level and frame-level boundary modeling. In a broader, interpretive sense, the term can denote hierarchical systems in which learned boundaries structure compression, decoding, or coarse-to-fine inference in sequence modeling and hierarchical classification. This suggests that the enduring idea is not a single blueprint but a family of methods that place boundaries at the center of hierarchical representation and decision making.

Source: https://www.emergentmind.com/topics/hierarchical-boundary-modeling-network-hbmnet