HBMNet: Hierarchical Deepfake Localization

• HBMNet is an end-to-end framework for audio-visual deepfake localization that precisely detects short, sparse manipulated segments.
• It integrates a coarse proposal generator with a fine frame-level refinement module to accurately model start, end, and content boundaries.
• The model employs bidirectional, multi-scale, and cross-modal attention mechanisms with contrastive supervision to enhance detection precision and recall.

Searching arXiv for the specified HBMNet paper and closely related hierarchical boundary modeling work. arxiv_search.query({"^^^^^^^^2search_query2 OR ti:\2^^^^^^^^"Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modeling\"","start":^^^^2search_query2^^^^,"max_results":5}) Hierarchical Boundary Modeling Network (HBMNet) is an end-to-end framework for audio-visual temporal deepfake localization under content-driven partial manipulation, introduced in “Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modeling” (&&&^^^^2search_query2^^^^&&&). Rather than deciding whether an entire clip is fake, HBMNet localizes when manipulated content occurs, with particular emphasis on the difficult regime in which fake content appears only in short, sparse temporal fragments while most frames remain authentic. Its central design combines stronger audio-visual representation learning with hierarchical boundary modeling across two temporal granularities: a coarse proposal-level stage that scans candidate segments globally and a fine frame-level stage that sharpens start and end boundaries and content probabilities using bidirectional temporal cues.

^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^. Task formulation and problem setting

HBMNet is defined for Audio-Visual Temporal Deepfake Localization (AVTDL) under content-driven partial manipulation (&&&^^^^2search_query2^^^^&&&). The input consists of an audio sequence PRESERVED_PLACEHOLDER_^^^^2search_query2^^^^ and a visual sequence PRESERVED_PLACEHOLDER_^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^. The ground truth is a set of temporal segments

$\Psi = \{\varphi_n = (t_n^s, t_n^e)\}_{n=1}^N,$

where each $\varphi_n$ denotes a temporal segment with start time $t_n^s$ and end time $t_n^e$. A segment is labeled real iff both audio and video are unmodified throughout $[t_n^s, t_n^e]$; otherwise it is labeled fake. The model predicts localized fake proposals

$$\Psi' = \{\varphi_n' = (t_n^{\prime s}, t_n^{\prime e})\}_{n=1}^{N'},$$

and during inference each proposal carries a confidence score, yielding

$\Psi' = \{(t_n^s, t_n^e, s_n')\}_{n=1}^{N'}.$

The setting differs from full-video forgery detection because the manipulations are tied to edits such as replacement, deletion, and insertion in audio, video, or both. Consequently, deepfake spans often cover only a few frames, and most frames remain identical to the original. The problem is therefore closer to temporal action localization / proposal generation than to ordinary video classification. A common misconception is to treat HBMNet as a binary deepfake detector; in the formal setup, the relevant question is the temporal location, start boundary, end boundary, and confidence of each manipulated segment.

HBMNet produces localization signals at multiple temporal levels. The Coarse Proposal Generator outputs a proposal-level confidence map

$M' \in \mathbb{R}^{L \times T},$

whose cells correspond to candidate segments indexed by start time and duration. The Fine-grained Probabilities Generator outputs frame-level start probability, end probability, and content probability in both forward and backward directions. The final scored temporal proposals combine proposal-level and frame-level evidence as

PRESERVED_PLACEHOLDER_^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2search_query2^^^^

2. Architectural organization and the meaning of hierarchical boundary modeling

HBMNet comprises three principal components: an Audio-Visual Feature Encoder (AVFE), a Coarse Proposal Generator (CPG), and a Fine-grained Probabilities Generator (FPG) (&&&^^^^2search_query2^^^^&&&). In pipeline form,

PRESERVED_PLACEHOLDER_^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^

The original and time-reversed audio and video streams are both processed. Audio and video are encoded separately, fused through cross-attention and self-attention, passed through a frame-wise classifier producing PRESERVED_PLACEHOLDER_^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\22^^^^, and then concatenated into enriched features PRESERVED_PLACEHOLDER_^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^3 that feed the proposal-level and frame-level branches.

The expression hierarchical boundary modeling has three distinct meanings in the system. First, it denotes a temporal-scale hierarchy: HBMNet models boundaries at the proposal level and at the frame level. Second, it denotes a semantic hierarchy among start boundary, end boundary, and content probabilities. Third, it includes a directional hierarchy, because forward and backward temporal transitions are modeled separately and then fused.

The coarse-to-fine organization is central. The CPG performs global scanning over candidate temporal regions and captures segment extent, while the FPG restores local temporal precision that is otherwise weakened by proposal sampling and aggregation. This suggests that HBMNet treats localization as a joint problem of segment proposal quality and framewise boundary-content consistency, rather than as either pure proposal scoring or pure per-frame classification.

3. Audio-visual feature encoding, fusion, and supervision

The AVFE is responsible for discriminative cross-modal frame-level representations (&&&^^^^2search_query2^^^^&&&). HBMNet explicitly processes both original and reversed inputs: PRESERVED_PLACEHOLDER_^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^4 The stated motivation is that a forward-only model may miss useful post-transition information; reversing the sequence turns fake-to-real transitions into real-to-fake transitions and vice versa.

The visual branch uses a 3D CNN stem, followed by ResNet-^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^8, followed by TCN modules. The audio branch converts the audio stream to Mel-spectrograms and encodes them with SENet: PRESERVED_PLACEHOLDER_^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^5 For each direction, AVFE outputs per-frame audio features PRESERVED_PLACEHOLDER_^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^6 and visual features PRESERVED_PLACEHOLDER_^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^7.

Fusion is mid-level cross-modal fusion, not late score fusion. AVFE uses three attention blocks: Audio-to-Visual (AV) cross-attention, Visual-to-Audio (VA) cross-attention, and an AttFUS self-attention fusion block. The resulting directional features are fused into PRESERVED_PLACEHOLDER_^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^8. A frame-wise classifier then produces

PRESERVED_PLACEHOLDER_^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^9

and the model concatenates fused features with frame probabilities: $\Psi = \{\varphi_n = (t_n^s, t_n^e)\}_{n=1}^N,$^^^^2search_query2^^^^

A notable contribution is frame-level contrastive supervision. The frame label sequence is

$\Psi = \{\varphi_n = (t_n^s, t_n^e)\}_{n=1}^N,$^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^

with a frame labeled real iff both audio and visual are real at that frame, and fake otherwise. The cross-modal feature distance at frame $\Psi = \{\varphi_n = (t_n^s, t_n^e)\}_{n=1}^N,$2 is

$\Psi = \{\varphi_n = (t_n^s, t_n^e)\}_{n=1}^N,$3

The model applies a margin-based contrastive loss $\Psi = \{\varphi_n = (t_n^s, t_n^e)\}_{n=1}^N,$4, whose intended effect is to pull cross-modal representations closer on real frames and push them apart on fake frames. The reported ablation states that encoding and fusion mainly improve precision, whereas frame-level supervision mainly improves recall. After adding frame-level supervision following encoder replacement, AR@5^^^^2search_query2^^^^ increases from 69.^^^^2search_query2^^^^3 to 7^^^^2search_query2^^^^.95, AR@2^^^^2search_query2^^^^ from 66.56 to 68.9^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^, AR@^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2search_query2^^^^ from 65.8^^^^2search_query2^^^^ to 68.^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2search_query2^^^^, and AP@^^^^2search_query2^^^^.75 from 58.^^^^2search_query2^^^^5 to 62.3^^^^2search_query2^^^^. This supports the paper’s claim that frame-level supervision reduces misses on short fake fragments.

4. Coarse proposal generation, fine-grained probabilities, and inference

The Coarse Proposal Generator is the proposal-level branch. Given

$\Psi = \{\varphi_n = (t_n^s, t_n^e)\}_{n=1}^N,$5

it predicts

$\Psi = \{\varphi_n = (t_n^s, t_n^e)\}_{n=1}^N,$6

where $\Psi = \{\varphi_n = (t_n^s, t_n^e)\}_{n=1}^N,$7 is the number of possible start times and $\Psi = \{\varphi_n = (t_n^s, t_n^e)\}_{n=1}^N,$8 is the maximum proposal duration (&&&^^^^2search_query2^^^^&&&). Each entry $\Psi = \{\varphi_n = (t_n^s, t_n^e)\}_{n=1}^N,$9 corresponds to a proposal

$\varphi_n$^^^^2search_query2^^^^

CPG is built on a Boundary-Matching (BM) layer from BMN. It constructs a sampling mask

$\varphi_n$^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^

whose per-proposal masks softly sample interval features, producing a boundary-matching tensor

$\varphi_n$2

This tensor is processed by 3D and 2D convolutions to obtain $\varphi_n$3. Proposal supervision is dense and soft: $\varphi_n$4 and the loss is

$\varphi_n$5

The Fine-grained Probabilities Generator is the frame-level refinement branch. It is based on Nested U-Net (U-Net++), takes $\varphi_n$6 as input, downsamples through two pooling stages with interleaved convolutional units, upsamples using skip connections, and predicts forward and backward triplets

$\varphi_n$7

with each in $\varphi_n$8. The three channels correspond to start, end, and fake-content probability sequences.

Frame-level labels are built by defining a temporal region

$\varphi_n$9

then deriving start, end, and content regions $t_n^s$^^^^2search_query2^^^^, $t_n^s$^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^, and $t_n^s$2. For each temporal anchor

$t_n^s$3

the labels are

$t_n^s$4

where

$t_n^s$5

The FPG objective uses Focal loss: $t_n^s$6 with

$t_n^s$7

and

$t_n^s$8

During inference, forward and backward frame-level probabilities are fused by geometric mean: $t_n^s$9 The final score for proposal $t_n^e$^^^^2search_query2^^^^ multiplies coarse proposal confidence, start confidence, end confidence, and mean content confidence: $t_n^e$^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^ Soft-NMS then suppresses overlapping proposals. The full training objective is

$t_n^e$2

with $t_n^e$3. The hierarchy is empirically complementary: CPG only achieves AP@^^^^2search_query2^^^^.95 = ^^^^2search_query2^^^^.66, FPG only achieves AP@^^^^2search_query2^^^^.95 = 9.^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^7, and using both yields AP@^^^^2search_query2^^^^.95 = ^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\22^^^^.7^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^.

5. Evaluation, ablations, and empirical behavior

Experiments are conducted on AV-Deepfake-^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^M, which contains 2,^^^^2search_query2^^^^68 subjects, 286,72^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^ real videos, and 86^^^^2search_query2^^^^,^^^^2search_query2 deepfake videos, with video-only, audio-only, and combined audio-video manipulation scenarios (&&&^^^^2search_query2^^^^&&&). Because of GPU limits, the reported experiments use a subset of 8,^^^^2search_query2search_query2search_query2^^^^ training videos, ^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^,^^^^2search_query2search_query2search_query2^^^^ validation videos, and 2,^^^^2search_query2search_query2search_query2^^^^ test videos sampled with a fixed random seed. Evaluation uses Average Precision (AP) at AP@^^^^2search_query2^^^^.5, AP@^^^^2search_query2^^^^.75, and AP@^^^^2search_query2^^^^.95, and Average Recall (AR) at AR@5^^^^2search_query2^^^^, AR@2^^^^2search_query2^^^^, and AR@^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2search_query2^^^^. AR is computed across IoU thresholds from ^^^^2search_query2^^^^.5 to ^^^^2search_query2^^^^.95 with step ^^^^2search_query2^^^^.^^^^2search_query2

Against the principal baselines, HBMNet reports the following values:

Model	AP@^^^^2search_query2^^^^.5	AP@^^^^2search_query2^^^^.75	AP@^^^^2search_query2^^^^.95
BA-TFD	38.89	^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^.84	^^^^2search_query2^^^^.^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2search_query2^^^^
UMMAFormer	75.93	6^^^^2search_query2^^^^.88	9.2^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^
HBMNet	96.82	9^^^^2search_query2^^^^.84	^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\22^^^^.79

For recall, AR@5^^^^2search_query2^^^^ is 42.56 for BA-TFD, 78.3^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^ for UMMAFormer, and 88.83 for HBMNet. The paper attributes the gap to stronger audio and video encoders, explicit cross-attention fusion, frame-level supervision, the proposal-plus-frame hierarchy, and bidirectional boundary-content modeling.

Ablations isolate the role of multimodality. Visual only yields AP@^^^^2search_query2^^^^.5 = 65.56, AP@^^^^2search_query2^^^^.75 = 57.64, AP@^^^^2search_query2^^^^.95 = 5.3^^^^2search_query2^^^^; Audio only yields 63.57, 6^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^.47, 8.6^^^^2search_query2^^^^; Audio + visual yields 96.5^^^^2search_query2^^^^, 92.74, ^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\22^^^^.7^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^. The reported interpretation is that multimodal fusion is crucial. Ablations over temporal scale show that CPG only gives strong coarse localization, whereas FPG only improves strict boundary precision and recall, and the combination is best overall. A further ablation over frame-level modeling reports AP@^^^^2search_query2^^^^.95 = ^^^^2search_query2^^^^.66 with No S/E, no content, no flip, 6.8^^^^2search_query2^^^^ with Start/end only, 6.8^^^^2search_query2^^^^ with Start/end + content, and ^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\22^^^^.7^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^ with Start/end + content + flip. The paper highlights that Flip raises AP@^^^^2search_query2^^^^.95 from 6.8^^^^2search_query2^^^^ to ^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\22^^^^.7^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^, roughly a 5^^^^2search_query2^^^^% relative increase.

The scalability experiment reports improved potential scalability with more training data. With ^^^^2search_query2^^^^.33x data, the model reaches AP@^^^^2search_query2^^^^.5 = 88.^^^^2search_query2^^^^7, AP@^^^^2search_query2^^^^.95 = ^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2search_query2^^^^.73, AR@5^^^^2search_query2^^^^ = 88.29; with ^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^x data, 96.5^^^^2search_query2^^^^, ^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\22^^^^.7^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^, 89.^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^9; and with 3x data, 97.88, ^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^4.58, 89.22. This suggests that strict-boundary precision continues to benefit from additional data. The paper does not report cross-dataset generalization. Implementation details include $t_n^e$4, $t_n^e$5, Adam, initial learning rate $t_n^e$6, halving the learning rate if validation loss does not decrease for 3 epochs, 4^^^^2search_query2^^^^ training epochs, batch size 8, and training on an NVIDIA Tesla V^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2search_query2search_query2^^^^ GPU.

The paper does not present a dedicated limitations section, but several constraints are explicit or implied: very short and fragmented manipulations remain difficult, noise can induce false positives if boundary cues are overemphasized without content cues, AP@^^^^2search_query2^^^^.95 remains far below AP@^^^^2search_query2^^^^.5, and only a subset of AV-Deepfake-^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^M is used because of resource constraints.

6. Related interpretations in hierarchical sequence modeling and classification

Outside AVTDL, the phrase Hierarchical Boundary Modeling Network also appears as an interpretive lens for architectures in which learned boundaries organize hierarchical computation, rather than as a single canonical model name. In “SOMBRERO: Measuring and Steering Boundary Placement in End-to-End Hierarchical Sequence Models” (Neitemeier et al., 30 Jan 2026), the relevant abstraction is a hierarchical autoregressive model over UTF-8 bytes with an encoder $t_n^e$7, backbone $t_n^e$8, decoder $t_n^e$9, and boundary variables $[t_n^s, t_n^e]$^^^^2search_query2^^^^, $[t_n^s, t_n^e]$^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^, and $[t_n^s, t_n^e]$2. That work introduces the router-agnostic boundary enrichment metric

$[t_n^s, t_n^e]$3

where $[t_n^s, t_n^e]$4 is next-byte surprisal, together with the confidence-alignment boundary loss

$[t_n^s, t_n^e]$5

At ^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^B scale, the reported design ladder moves from H-Net with $[t_n^s, t_n^e]$6, $[t_n^s, t_n^e]$7, $[t_n^s, t_n^e]$8 to final SOMBRERO with $[t_n^s, t_n^e]$9, $$\Psi' = \{\varphi_n' = (t_n^{\prime s}, t_n^{\prime e})\}_{n=1}^{N'},$$^^^^2search_query2^^^^, $$\Psi' = \{\varphi_n' = (t_n^{\prime s}, t_n^{\prime e})\}_{n=1}^{N'},$$^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^, indicating that in that literature boundary placement is treated as a measurable compute-allocation policy rather than as a localization problem.

“Dynamic Chunking for End-to-End Hierarchical Sequence Modeling” (Hwang et al., 10 Jul 2025) provides a related but distinct formulation, centered on H-Net. There, learned boundaries

$$\Psi' = \{\varphi_n' = (t_n^{\prime s}, t_n^{\prime e})\}_{n=1}^{N'},$$2

drive dynamic compression and dechunking, with smoothing

$$\Psi' = \{\varphi_n' = (t_n^{\prime s}, t_n^{\prime e})\}_{n=1}^{N'},$$3

The model is explicitly recursive and supports ^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^-stage and 2-stage hierarchies. On FineWeb-Edu, the reported 2-stage H-Net attains ^^^^2search_query2^^^^.743 BPB at large scale and ^^^^2search_query2^^^^.7^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^5 BPB at XL scale, surpassing the corresponding tokenized Transformer baselines (^^^^2search_query2^^^^.756 and ^^^^2search_query2^^^^.73^^^^2search_query2). The paper repeatedly presents the system as a learned alternative to tokenization, in which boundaries are content- and context-dependent.

A different line appears in “HiNet: Hierarchical Classification with Neural Network” (Wu et al., 2017), where the hierarchy is not temporal but label-structural. HiNet organizes outputs into hierarchical layers, introduces a stop neuron, and performs structured MAP trace inference with the recurrence

$$\Psi' = \{\varphi_n' = (t_n^{\prime s}, t_n^{\prime e})\}_{n=1}^{N'},$$4

Its training objective is the multi-level loss

$$\Psi' = \{\varphi_n' = (t_n^{\prime s}, t_n^{\prime e})\}_{n=1}^{N'},$$5

and the reported parameter complexity is $$\Psi' = \{\varphi_n' = (t_n^{\prime s}, t_n^{\prime e})\}_{n=1}^{N'},$$6, contrasted with $$\Psi' = \{\varphi_n' = (t_n^{\prime s}, t_n^{\prime e})\}_{n=1}^{N'},$$7 for a flat classifier. On DMOZ with ^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^,947 classes, HiNet reports 4^^^^2id:(Chen et al., 4 Aug 2025) OR ti:\2^^^^.4 accuracy versus 39.2 for a Flatten Network.

Taken together, these works indicate that HBMNet is not a universally standardized architecture across domains. In the strict sense established by (&&&^^^^2search_query2^^^^&&&), it denotes an audio-visual temporal deepfake localization network that combines proposal-level and frame-level boundary modeling. In a broader, interpretive sense, the term can denote hierarchical systems in which learned boundaries structure compression, decoding, or coarse-to-fine inference in sequence modeling and hierarchical classification. This suggests that the enduring idea is not a single blueprint but a family of methods that place boundaries at the center of hierarchical representation and decision making.