Robust Deepfake Detection: Mitigating Spatial Attention Drift via Calibrated Complementary Ensembles

Abstract: Current deepfake detection models achieve state-of-the-art performance on pristine academic datasets but suffer severe spatial attention drift under real-world compound degradations, such as blurring and severe lossy compression. To address this vulnerability, we propose a foundation-driven forensic framework that integrates an extreme compound degradation engine with a structurally constrained, multi-stream architecture. During training, our degradation pipeline systematically destroys high-frequency artifacts, optimizing the DINOv2-Giant backbone to extract invariant geometric and semantic priors. We then process images through three specialized pathways: a Global Texture stream, a Localized Facial stream, and a Hybrid Semantic Fusion stream incorporating CLIP. Through analyzing spatial attribution via Score-CAM and feature stability using Cosine Similarity, we quantitatively demonstrate that these streams extract non-redundant, complementary feature representations and stabilize attention entropy. By aggregating these predictions via a calibrated, discretized voting mechanism, our ensemble successfully suppresses background attention drift while acting as a robust geometric anchor. Our approach yields highly stable zero-shot generalization, achieving Fourth Place in the NTIRE 2026 Robust Deepfake Detection Challenge at CVPR. Code is available at https://github.com/khoalephanminh/ntire26-deepfake-challenge.

• The paper presents a calibrated complementary ensemble integrating localized, global, and semantic pathways to robustly detect deepfakes under compound degradations.
• It employs a compound degradation engine simulating up to 15 distinct noise operations to force models to abandon fragile texture shortcuts.
• Experimental results demonstrate improved AUC scores and stability under severe noise, setting a new standard for real-world deepfake forensics.

Robust Deepfake Detection via Calibrated Complementary Ensembles

Motivation and Problem Definition

The paper "Robust Deepfake Detection: Mitigating Spatial Attention Drift via Calibrated Complementary Ensembles" (2604.25889) addresses a critical vulnerability in current deepfake detection regimes: catastrophic spatial attention drift and texture bias when models encounter real-world compound degradations (e.g., blur, lossy compression, occlusion, and multi-modal noise). While foundation models such as ViTs and CLIP provide strong zero-shot generalization on academic datasets, they fail to isolate forgery cues under noisy conditions prevalent in social media and uncontrolled contexts. The central claim is that deepfake artifacts span geometric, contextual, and semantic dimensions, each vulnerable to distinct domain shifts, necessitating robust pipelines that aggregate signals across these spaces.

System Architecture and Compound Degradation

The methodological innovation comprises two principal elements: (1) a compound degradation engine that systematically destroys shortcut texture priors during training, and (2) a multi-stream ensemble architecture integrating specialized expert pathways for local facial geometry, global texture context, and semantic consistency.

The compound degradation engine employs a randomized pipeline invoking up to 15 degradation steps per image, drawn from a pool of 18 distinct noise operations including compression, resampling, sensor noise, blur, and photometric distractors. The pipeline explicitly simulates multi-stage real-world transmission noise, thereby forcing models to abandon fragile artifacts and learn invariant features.

Figure 1: Compound Degradation Engine, demonstrating the impact of isolated and compound operations used to neutralize texture shortcuts during training.

The architecture processes images through three expert streams:

• Localized Facial Stream: targets fine-grained face manipulations, leveraging geometric priors.
• Global Texture Stream: evaluates holistic contextual anomalies, capturing environmental inconsistencies.
• Hybrid Semantic Fusion Stream: integrates CLIP features to detect high-level logical errors, robust to texture ambiguity.

These streams are anchored by DINOv2-Giant, adapted via LoRA for parameter-efficient tuning, and fused through a calibrated 1:2:2 discretized ensemble voting system.

Figure 2: Overview of the architecture, illustrating the three expert streams and multi-modal fusion.

Experimental Evaluation and Numerical Results

The training pool spans 14 datasets balanced across generator diversity, in-the-wild scenarios, and modern high-fidelity forgeries. Strict filtering removes entire face synthesis data to focus on face-swapping and reenactment boundaries. Data balancing and aggressive augmentation yield a 377,343-frame corpus with 52.67% real and 47.33% fake instances.

On the NTIRE 2026 Robust Deepfake Detection Challenge, the ensemble achieves an AUC of 0.8775 (Public Test) and 0.8523 (Private Test), surpassing most competitors and securing fourth place. Ablation experiments rigorously validate the contributions:

• Increasing dataset diversity elevates validation AUC from 0.7452 (single-domain) to 0.9303 (full pool).
• LoRA adaptation preserves zero-shot priors, outperforming full fine-tuning (0.8713 vs. 0.8255 AUC).
• Compound degradation and precise spatial alignment push AUC from 0.8465 (Vanilla) to 0.8713 (Patch-aligned).

Ensemble Design and Complementary Signal Analysis

The ensemble aggregates via discretized probability voting, quantizing raw scores to 0.1 precision and enforcing robust confidence bins. This calibrated fusion outperforms continuous averaging and naive weighting, maintaining stability under severe domain shift. Ablating any stream degrades performance, confirming non-redundant contributions.

Pairwise prediction correlation matrices verify sub-unity values across stream pairs, indicating that each pathway captures complementary decision boundaries rather than redundant signals.

Figure 3: Ensemble voting stability and component ablations, highlighting maximal robustness with the calibrated ensemble.

Figure 4: Prediction correlation matrix demonstrating non-redundant and complementary contributions of each stream.

Explainability: Spatial Attribution and Robustness Visualizations

Explainability analysis utilizes Score-CAM for visualizing attention, and computes spatial attribution entropy and feature cosine similarity. Augmented streams resist entropy collapse and embedding drift under extreme noise, with the Hybrid Fusion stream maintaining sub-9.0 entropy and the Localized Crop stream preserving maximal cosine similarity.

Figure 5: Spatial Attribution Entropy and Cosine Similarity across degradation severity, demonstrating robust attention and feature spaces.

Figure 6: Score-CAM spatial attribution under compound degradation across models, affirming robust facial anchoring and semantic isolation.

Qualitative grids further reveal that the ensemble ignores complex distractors on real images, isolates blending boundaries on synthetic forgeries, and honestly fails under extreme localized blur—a boundary case still unsolved for current architectures.

Figure 7: Qualitative Score-CAM analysis, contrasting correct and incorrect predictions across diverse challenges and noise levels.

Implications and Future Directions

The framework advances practical deepfake forensics by neutralizing texture bias and mitigating attention drift, achieving robust zero-shot generalization amidst realistic degradations. Theoretically, the results emphasize the necessity of multi-scale, multi-modal signal aggregation and parameter-efficient adaptation of foundation models, rather than full fine-tuning or reliance on texture-based shortcuts.

Practically, the ensemble's resilience establishes a new benchmark for real-world deployment scenarios and informs next-generation forensic methodologies. Failure cases—especially extreme blur masking blending boundaries—highlight avenues for integrating higher-order geometric logic or spatio-temporal consistency checks.

Future developments may include:

• Integration with temporal streams for video-centric forensics.
• Leveraging cross-modal priors beyond vision-language (e.g., audio-texture fusion).
• Dynamic ensemble weighting strategies based on contextual signal decay.
• Advanced XAI for causal attribution and actionable forensic evidence.

Conclusion

This work demonstrates that robust forensic detection of deepfakes under real-world noise requires a composite architecture aggregating geometric, contextual, and semantic priors, trained under aggressive compound degradation. Calibrated complementary ensembles negate spatial attention drift and domain-specific shortcut learning, with visual explainability confirming their efficacy. The approach marks a rigorous step forward for both theoretical generalization and practical resilience in AI-driven media forensics.