Hierarchical Deep Fusion Framework (HDFF)
- The paper introduces HDFF, an ensemble deep learning framework that fuses features from four diverse backbones to detect facial forgeries across 50+ manipulation methods.
- It employs a hierarchical multi-stage training strategy, evolving from head-only to full-network adaptation before fusing representations for binary classification.
- The framework balances local and global artifact detection while operating under a strict 200 MB model-size constraint, achieving competitive leaderboard results.
The Hierarchical Deep Fusion Framework (HDFF) is an ensemble-based deep learning architecture for multi-dimensional facial forgery detection that was introduced for the 2024 Global Deepfake Image Detection Challenge. In this formulation, HDFF is designed for binary classification of real versus fake facial images under a setting that includes over 50 different forgery methods, a 200 MB model-size limit, and the need to generalize beyond known artifacts. Its defining idea is feature-level fusion of four heterogeneous ImageNet-1K pre-trained visual backbones—Swin-MLP, CoAtNet, EfficientNetV2, and DaViT—combined with a multi-stage training procedure in which each backbone is first adapted to the target domain and then frozen while a top-level fusion classifier is trained on their concatenated representations (Wang et al., 16 Sep 2025).
1. Problem setting and design motivation
HDFF is situated in multi-dimensional facial forgery detection, where the central difficulty is not merely distinguishing authentic from manipulated faces, but doing so across a highly diverse manipulation space. The MultiFFDI challenge dataset contains real facial images and forged images produced by more than 50 different forgery methods, including face swapping, attribute editing, identity morphing, GAN synthesis, and related pipelines. The task is still binary classification, but the operational difficulty arises from the breadth of forgery families and from the fact that modern deepfakes often exhibit subtle, spatially localized, or high-frequency artifacts rather than gross visual errors (Wang et al., 16 Sep 2025).
The framework is motivated by four constraints stated explicitly in the challenge setting. First, different generators and editing pipelines introduce different artifacts, including identity blending, boundary inconsistencies, texture and color shifts, and frequency anomalies. Second, detectors trained on a small set of forgery types often overfit to known artifacts and fail on unseen methods. Third, high-quality deepfakes can conceal artifacts in very small regions or implicit frequency patterns. Fourth, the challenge imposes a 200 MB model-size limit, which makes naive ensembling infeasible. HDFF addresses these constraints by combining diverse architectures with complementary inductive biases, emphasizing feature-level ensemble rather than prediction-level averaging, and using a hierarchical training strategy that stabilizes adaptation to the forgery domain while preserving deployability under the competition constraint (Wang et al., 16 Sep 2025).
The term “hierarchical” is used here primarily in the sense of staged adaptation and top-level representation formation. It does not denote a single-backbone feature pyramid. Instead, the hierarchy lies in the training sequence—head-only adaptation, full-network adaptation, and then top-level fusion—and in the emergence of a fused representation above the individual backbone embeddings.
2. Constituent architecture and feature fusion
HDFF integrates four pre-trained models, all originally trained on ImageNet-1K: Swin-MLP, CoAtNet, EfficientNetV2, and DaViT. Each acts as a feature extractor specialized for a different aspect of image representation. Swin-MLP is described as a combination of Swin Transformer and an MLP-based classification head. CoAtNet is a hybrid convolution-plus-attention architecture. EfficientNetV2 is an efficient, scalable CNN. DaViT is a Dual Attention Vision Transformer with dual attention mechanisms (Wang et al., 16 Sep 2025).
After fine-tuning on MultiFFDI, HDFF takes the output of the final feature layer from each backbone—the layer just before the original classifier—and concatenates the resulting vectors into a single fused representation:
A new fully connected fusion layer is then appended on top of this concatenated feature. The final classifier outputs two logits for the real-versus-fake decision:
This design treats the backbone outputs as heterogeneous but compatible descriptors whose joint interactions can be learned explicitly by the fusion classifier, rather than averaged only at the decision level (Wang et al., 16 Sep 2025).
The paper emphasizes complementarity rather than architectural redundancy. Transformer-based backbones—Swin-MLP, DaViT, and the attention component of CoAtNet—are associated with long-range dependency modeling, including global lighting or identity inconsistencies. EfficientNetV2 and the convolutional component of CoAtNet are associated with local, high-frequency artifacts such as edges, compression traces, and blending boundaries. Fusion therefore serves as an interaction model across local-versus-global and CNN-versus-transformer biases.
| Backbone | Architectural characterization | Stated benefit for forgery detection |
|---|---|---|
| Swin-MLP | Swin Transformer with MLP classifier | Patch-level inconsistencies, hierarchical facial layout variation |
| CoAtNet | Hybrid convolution + attention | Local textures plus global context |
| EfficientNetV2 | Efficient, scalable CNN | Texture, frequency sensitivity, compression or upsampling artifacts |
| DaViT | Dual Attention Vision Transformer | Subtle correlations across channels and spatial locations |
The total number of parameters of the combined model, including all four backbones and the fusion fully connected layer, is 180.16M, and the paper states that this keeps the model under the 200 MB competition limit (Wang et al., 16 Sep 2025).
3. Hierarchical multi-stage training strategy
HDFF’s hierarchy is most explicit in its four-stage training pipeline. Stage 1 initializes each backbone from ImageNet-1K pre-trained weights. This gives each model strong generic visual features spanning edges, textures, and shapes. Stage 2 then performs selective fine-tuning by freezing all layers except the final classifier head of each sub-model and training only that final fully connected layer on MultiFFDI. The stated purpose is to adapt the classification boundary quickly while preserving the original feature hierarchy (Wang et al., 16 Sep 2025).
Stage 3 performs comprehensive fine-tuning. After the head has adapted, all layers of each sub-model are unfrozen and trained end-to-end on MultiFFDI with a smaller learning rate, using AdamW and CosineAnnealingLR. This stage is described as crucial. The paper reports that Swin-MLP trained only on the final layer plateaued at approximately 0.75 accuracy, whereas full-network fine-tuning raised performance to competitive levels. This observation is operationally important because it distinguishes HDFF from lightweight transfer-learning recipes that treat backbone features as nearly fixed (Wang et al., 16 Sep 2025).
Stage 4 trains the “grand model.” The four fine-tuned sub-models are integrated into the HDFF architecture, all sub-model weights are frozen, and only the new fusion fully connected layer is trained on the concatenated features. This makes the top-level learning problem efficient and stable: the fusion layer need only learn how to combine already adapted backbone embeddings. In this sense, the framework is hierarchical both within models and between models, culminating in a learned representation over backbone-level features.
The source description also states that HDFF uses standard cross-entropy loss for binary classification, with the predicted probability distribution over produced by the fusion classifier. More generally, the framework’s training logic is to delay joint fusion until after each backbone has been individually specialized for facial forgery detection, rather than attempting simultaneous end-to-end optimization of all components from the start.
4. Multi-dimensional detection, preprocessing, and robustness
In the HDFF formulation, “multi-dimensional” refers to several simultaneous axes of variation: multiple forgery types, multiple generation pipelines, and multiple visual dimensions such as spatial structure, texture and color statistics, local pixel-level anomalies, global consistency, and potentially frequency-domain or implicit artifacts captured by learned filters and attention. The architecture is therefore not tailored to a single manipulation family; it is meant to remain effective across face swapping, attribute editing, identity morphing, GAN synthesis, and other manipulation modes represented in MultiFFDI (Wang et al., 16 Sep 2025).
All sub-models share a unified preprocessing pipeline. Images are resized to a fixed input size, exemplified as , then transformed with AutoAugment, converted to tensors, and normalized with ImageNet mean and standard deviation:
The AutoAugment policy includes operations such as Invert, Rotate, Sharpness, ShearY, TranslateX, Color, and Brightness. These augmentations are described as increasing robustness to pose, lighting, and color variation while preventing overfitting and improving generalization to unseen forgery techniques (Wang et al., 16 Sep 2025).
Robustness in HDFF is attributed to three interacting design choices. Architectural diversity provides sensitivity to different artifact families. Full-network fine-tuning shifts representations away from generic ImageNet semantics toward subtle forensic cues. Strong augmentation prevents the fusion model from collapsing onto superficial correlations. The framework’s stated goal is therefore not only high in-domain performance but broad generalization across many different manipulation methods.
The complementarity claims in the paper are tightly tied to representation type. CNN-like inductive biases assume locality and translational invariance, which favor detection of blending boundaries and local texture noise. Transformer-like inductive biases allow freer modeling of long-range dependencies, which favor global identity, illumination, or consistency anomalies. Fusion allows the final classifier to weight these heterogeneous signals jointly rather than independently.
5. Evaluation, ablations, and operational limits
HDFF achieved a final score of $0.9685225726$ on the private leaderboard of the 2024 Global Deepfake Image Detection Challenge and placed 20th out of 184 teams. The paper presents this as evidence that hierarchical feature fusion can perform competitively under a realistic challenge setting with broad manipulation diversity and explicit deployment constraints (Wang et al., 16 Sep 2025).
Two ablation findings are emphasized. First, CosineAnnealingLR significantly improved performance relative to StepLR. On a phase-1 test set, a single model’s accuracy improved from 0.9949 with StepLR to 0.9967 with CosineAnnealingLR. Second, head-only fine-tuning was insufficient: Swin-MLP plateaued at approximately 0.75 accuracy until all layers were unfrozen and trained end-to-end. These findings frame optimization schedule and depth of adaptation as central to performance, rather than peripheral training details (Wang et al., 16 Sep 2025).
The paper also states that individually fine-tuned backbones are strong, but that fusion of their features via HDFF yields better and more robust performance, as reflected by the competitive leaderboard result. The exact challenge metric is not explicitly specified in the text; the source description states that it is likely a variant of accuracy or ROC-AUC computed over the test set. This suggests caution in interpreting the reported leaderboard score as directly interchangeable with a standard offline metric.
Operationally, the framework is expensive. Each sub-model requires 2–3 days of comprehensive fine-tuning, and the final fusion training adds another 2–3 days. The reported hardware is 4× NVIDIA A100 80GB. The model is also heavy for deployment on edge devices, at approximately 180M parameters, and it remains a single-modality 2D image detector with no explicit use of depth cues or temporal information. The paper further notes or implies that performance is tuned to MultiFFDI and that generalization to entirely different domains may still require further adaptation (Wang et al., 16 Sep 2025).
Suggested future work includes incorporating predicted depth maps or other modalities to detect 3D inconsistencies, exploring more lightweight fusion architectures for deployment, applying domain adaptation to new forgery distributions, and extending the framework to video deepfake detection with temporal consistency models.
6. Position within the broader fusion literature
HDFF, as defined in facial forgery detection, belongs to a broader pattern in which multiple heterogeneous feature extractors are fused above their individual representation spaces rather than only at the prediction level. In medical imaging, “DeepCervix” uses a hybrid deep feature fusion technique in which VGG16, VGG19, ResNet50, and Xception each produce a 1024-dimensional vector, concatenated into a 4096-dimensional fused representation, followed by batch normalization, dropout, and a final dense-plus-softmax layer (Rahaman et al., 2021). In that formulation, the hierarchy lies in the CNN backbones and in the fusion network above them.
Other uses of hierarchical fusion differ in modality and mechanism but preserve the same structural idea of multi-level integration. In SSVEP-based BCIs, hierarchical feature fusion combines spatial dimension fusion and frequency dimension fusion by weighted nonlinear strategies, producing a two-level progression from correlated components to sub-band features and then to final class evidence (Zhang et al., 2018). In RGB-D robotics and scene understanding, hierarchical fusion appears as multi-scale confidence-weighted fusion of RGB and depth features, as in robotic grasping and surface normal estimation, where modality-specific branches are fused at multiple decoder scales and depth features are re-weighted by confidence maps before integration (Song et al., 2019); (Zeng et al., 2019). In guided depth super-resolution, attention-based hierarchical multi-modal fusion combines per-layer gated feature enhancement, feature recalibration, and bi-directional hierarchical feature collaboration across scales (Zhong et al., 2021).
The term has also appeared in double-ensemble settings. In brain tumor MRI classification, a two-level strategy fuses deep features from top-performing ViT models and then aggregates predictions from hyperparameter-optimized machine learning classifiers by majority voting (Ullah et al., 14 Jun 2025). In robotics, a hierarchical representation fusion framework for audio, vision, and proprioception first conditions visual and proprioceptive representations on acoustic cues and then models higher-order cross-modal interactions before action generation (Li et al., 14 Feb 2026). This suggests that “Hierarchical Deep Fusion Framework” is best understood not as a single canonical architecture, but as a recurring design pattern: heterogeneous encoders, staged or multi-level interaction, and a learned fusion mechanism that operates above modality- or backbone-specific embeddings.
Within that broader pattern, the facial forgery version of HDFF is distinguished by three features: explicit competition-driven parameter budgeting, a head-only-to-full-network-to-fusion training hierarchy, and feature-level ensemble across four pre-trained visual backbones for binary detection under many forgery families. Its significance lies less in introducing an entirely new fusion primitive than in demonstrating that carefully staged hierarchical fusion can remain competitive in a difficult, artifact-diverse deepfake image benchmark (Wang et al., 16 Sep 2025).