ReCap: Deepfake Detection via Recovery and Mapping

• ReCap is a deepfake detection framework that leverages ROI-aware masked autoencoding and mapping networks to expose unpredictable tampered traces across facial regions.
• Its recovering stage uses a Vision Transformer-based masked autoencoder with random ROI masking to reveal inconsistencies in forged faces.
• The mapping stage amplifies the real-versus-fake discrepancies, achieving superior cross-domain performance and state-of-the-art detection benchmarks.

ReCAP: Detecting Deepfake Video with Unpredictable Tampered Traces via Recovering Faces and Mapping Recovered Faces

Recap is a deepfake detection framework explicitly designed to address real-world adversarial scenarios in which face manipulation leaves unpredictable and spatially variable tampered traces. Unlike methods that exploit fixed forgery indicators, Recap exposes inconsistencies in arbitrary facial sub-regions via unsupervised recovery, then amplifies the real-vs-fake signal through a mapping network. Its two-stage architecture and ROI-aware masking protocols confer state-of-the-art cross-domain robustness, outperforming prior detectors by significant margins across benchmarks (Hu et al., 2023).

1. Problem Context: Deepfake Detection under Unpredictable Tampering

Recap operates under a threat model in which forgeries are generated by unknown or evolving deepfake engines, resulting in randomized distribution of tampered traces (e.g., blending artifacts, subtle noise discrepancies) across facial regions. Traditional detectors leveraging fixed positional cues or hand-crafted forensics rapidly degrade in cross-domain generalization as the spatial manifestation of artifacts becomes randomized.

Formally, given a face-cropped video $V$ yielding frames $$x_1, \dotsc, x_T \in \mathbb{R}^{H \times W \times 3}$$, with $y_t \in \{0,1\}$ indicating real or fake, the objective is to learn a scoring function $f(x)$ with high values for real frames and low values for fakes, robust to unseen manipulation strategies.

2. Recovering Stage: ROI-Aware Masked Autoencoding

2.1 Random ROI Masking

Each face (224×224) is partitioned via facial landmarks into $K$ semantic ROIs (eyes, cheeks, nose, lips, chin). For each frame, a single ROI $R_k$ is selected at random, and within it, $m=0.75$ of patches are masked (i.e., set to zero). This yields a mask $M \in \{0,1\}^{H \times W}$, applied to the input as $x_{\mathrm{mask}} = M \odot x$.

2.2 Asymmetric ViT-based Masked Autoencoder

A Vision Transformer masked autoencoder $R_\phi$ encodes the unmasked patches and uses a lightweight decoder to reconstruct pixel values in the masked patches. During training (on real faces only), the loss is

$$x_1, \dotsc, x_T \in \mathbb{R}^{H \times W \times 3}$$0

Due to the internal consistency of real facial textures, $$x_1, \dotsc, x_T \in \mathbb{R}^{H \times W \times 3}$$1 recovers masked content effectively. In contrast, fakes exhibit either unrecoverable masked ROIs (if forgery lies within) or contaminated context (if forgery is outside), both yielding higher reconstruction error.

3. Mapping Stage: Discrepancy Amplification in Learned Representation

With the recovery encoder frozen, Recap processes both real and fake (unmasked) frames through $$x_1, \dotsc, x_T \in \mathbb{R}^{H \times W \times 3}$$2, producing corresponding recovered faces $$x_1, \dotsc, x_T \in \mathbb{R}^{H \times W \times 3}$$3. These are then input to a lightweight mapping network $$x_1, \dotsc, x_T \in \mathbb{R}^{H \times W \times 3}$$4, which consists of three residual blocks (ResNet-18 backbone minus final FC) and further convolutional layers, ultimately outputting a mapped face of dimension $$x_1, \dotsc, x_T \in \mathbb{R}^{H \times W \times 3}$$5.

The mapping objective is to minimize mapped distance for reals and maximize for fakes—a binary cross-entropy loss with an auxiliary pixel-wise MSE for training stabilization: $$x_1, \dotsc, x_T \in \mathbb{R}^{H \times W \times 3}$$6

$$x_1, \dotsc, x_T \in \mathbb{R}^{H \times W \times 3}$$7

This meta-learning regime ensures mapping further deteriorates fake representations while enhancing real ones, expanding real/fake separation.

4. Two-Phase Training Paradigm and Optimization

Training proceeds sequentially:

Phase 1: Recovering stage—train $$x_1, \dotsc, x_T \in \mathbb{R}^{H \times W \times 3}$$8 on real data only, minimizing $$x_1, \dotsc, x_T \in \mathbb{R}^{H \times W \times 3}$$9.

Phase 2: Mapping stage—freeze $y_t \in \{0,1\}$0, optimize $y_t \in \{0,1\}$1 on real+fake, minimizing the combined objective

$y_t \in \{0,1\}$2

with $y_t \in \{0,1\}$3, $y_t \in \{0,1\}$4. Fake samples are meta-split within each batch for meta-learning.

Core architectural parameters:

• Recovering: ViT-Base encoder, spatial-temporal attention (VideoMAE style), decoder with four transformer layers.
• Mapping: ResNet-18 (first three blocks) concatenated with three $y_t \in \{0,1\}$5 convolution layers.
• Optimization: AdamW for recovering (lr $y_t \in \{0,1\}$6), SGD for mapping (lr $y_t \in \{0,1\}$7 with decay).

Augmentations involve random flips, color jitter, and low-amplitude Gaussian noise in both training phases.

5. Experimental Results and Ablations

Cross-Domain and Intra-Dataset Results

Recap was trained on FaceForensics++ (encompassing DeepFake, Face2Face, FaceSwap, NeuralTextures) and evaluated zero-shot on Celeb-DF, WildDeepfake, and DFDC datasets. Area under the ROC (AUC) and Equal Error Rate (EER) benchmarks:

• Celeb-DF: 90.1% AUC / 14.2% EER
• WildDF: 83.1% / 24.6%
• DFDC: 83.2% / 24.8%

Recap outperforms state-of-the-art baselines by 3–5 points AUC on cross-dataset tests; when trained on single manipulation types, Recap retains a mean AUC of ~78%, surpassing best competing methods ($y_t \in \{0,1\}$875%).

On intra-dataset evaluation (FF++ C23), Recap achieves near-perfect discrimination: AUC 99.6%–99.0%, rivaling specialized detectors.

Ablation Findings

• Mask ratio $y_t \in \{0,1\}$9 is optimal; lower reduces real/fake distinction, higher impairs both.
• ROI-aware masking yields 4–5 point AUC gain over vanilla MAE/VideoMAE random masks.
• Removing either stage (recovering or mapping) drops AUC by $f(x)$03 points; omitting meta-learning costs 2–3 points.
• Adding Recap's mapping stage to existing MAE-based detectors (MAE, VideoMAE, RECCE) improves them by 4–6 AUC points.

Visualization: Class Activation Maps indicate that fake frames “light up” in masked ROIs (where recovery failed), while real frame attention is contextual. Distribution plots show the mapping stage expands real/fake separability beyond that achieved by recovery alone.

6. Analysis, Robustness, and Underlying Insights

The two-stage design is critical. The recovering stage incentivizes globally consistent inpainting on real faces and exposes any facial region inconsistencies introduced by forgeries as heightened reconstruction error. This error is then operationalized in the mapping stage, where the mapping network is trained to amplify the recoverability discrepancy, further separating reals from fakes in the learned space.

This decomposition counters overfitting to any specific manipulator or artifact type—moving detection away from superficial, potentially short-lived forensics cues toward more robust measures of global facial consistency.

Key limitations:

• Currently, the recovery encoder is frozen during the mapping stage; end-to-end joint optimization may enhance representation but at risk of overfitting.
• ROI selection is hand-crafted from facial landmarks; learned or adaptive regional maskers may better exploit unseen tampering distributions.
• The entire pipeline is frame-wise; extending recovery and mapping across spatio-temporal windows could exploit motion-related artifacts inherent in video forgeries.

7. Impact and Comparative Significance

Recap’s most distinctive technical contribution is the use of random ROI masking and masked autoencoder recovery, independently trained on real faces, with recovery quality then acting as an explicit supervisory signal for a secondary discriminator (mapping network). This two-stage, error-amplification framework provides a generic mechanism for robust fake/real separation regardless of evolving attack surfaces and is highly modular, with clear pathways for integration into other self-supervised or MAE-based architectures.

Benchmarks demonstrate that Recap offers a substantial improvement in cross-domain detection performance (e.g., 3–5 point AUC lift), delivering production-relevant robustness without reliance on forgery-specific artifacts or tuning (Hu et al., 2023). The choice of ROI-aware masking and strict two-phase optimization are empirically and analytically validated for favoring generalization, making Recap a significant advance in the Deepfake detection literature.