GenConViT: Hybrid Deepfake Detection

• The paper introduces a dual-branch architecture that combines Autoencoder and Variational Autoencoder to capture both fine-grained local artifacts and global inconsistencies in deepfake videos.
• The methodology employs convolutional and transformer-based hybrid modules to extract robust features from both original and reconstructed face frames.
• Empirical evaluations demonstrate high accuracy and AUC across major benchmarks like DFDC and Celeb-DFv2, setting new standards for deepfake detection.

GenConViT is a two-branch, hybrid deepfake video detection architecture combining convolutional neural networks (ConvNeXt) with hierarchical vision transformer modules (Swin Transformer), optimized to extract both fine-grained local artifacts and global contextual inconsistencies from manipulated face videos. The model integrates both deterministic (Autoencoder; AE) and probabilistic (Variational Autoencoder; VAE) generative paradigms within its architecture to capture a diverse set of forgery cues. GenConViT was originally proposed for robust detection across a variety of deepfake manipulation techniques and datasets, establishing new performance benchmarks while informing the design of subsequent variants and derivatives (Deressa et al., 2023, Batista, 3 Apr 2025, Monu et al., 2024).

1. Model Architecture

GenConViT consists of two parallel branches—Network A, which employs a plain Autoencoder, and Network B, which utilizes a Variational Autoencoder. Each branch processes both the original input face frame $X$ and its reconstructed version ($I_A$ or $I_B$), and each representation is separately passed through identical ConvNeXt-Swin hybrid modules for feature extraction.

• Autoencoder (AE) Branch (Network A)
  • Encoder: 5 convolutional layers ($$3 \rightarrow 16 \rightarrow 32 \rightarrow 64 \rightarrow 128 \rightarrow 256$$), each with 3×3 kernels, stride 2, ReLU and max pooling (2×2, stride 2). Output latent tensor $Z \in \mathbb{R}^{256 \times 7 \times 7}$.
  • Decoder: 5 transposed-convolutional layers ($$256 \rightarrow 128 \rightarrow 64 \rightarrow 32 \rightarrow 16 \rightarrow 3$$), kernel 2×2, stride 2, ReLU. Output $I_A \in \mathbb{R}^{224 \times 224 \times 3}$ (Deressa et al., 2023).
• Variational Autoencoder (VAE) Branch (Network B)
  • Encoder: 4 convolutional layers ($$3 \rightarrow 16 \rightarrow 32 \rightarrow 64 \rightarrow 128$$), kernel 3×3, stride 2; each with BatchNorm, LeakyReLU; flatten to 12,544-d vector, split into mean $\mu(X)$ and $\log \sigma^2(X)$.
  • Reparameterization: $I_A$0, $I_A$1.
  • Decoder: 4 transposed convolutional layers ($I_A$2), kernel 2×2, stride 2; each with LeakyReLU. Output $I_A$3 (Deressa et al., 2023).
• ConvNeXt-Swin Hybrid Module
  • ConvNeXt_tiny backbone (pretrained on ImageNet-1k) produces feature maps $I_A$4.
  • 1×1 convolution projects $I_A$5, resulting in a sequence of 768-d tokens.
  • Swin_tiny_patch4_window7_224 operates on these tokens, yielding a 1,000-dimensional feature vector per input.

Each original/reconstructed frame pair ($I_A$6; $I_A$7) is mapped via their respective hybrid modules to two 1,000-dim feature vectors, concatenated and passed to a two-way classifier for real/fake decision. The model design allows both branches to contribute equally; empirical results indicate nearly identical per-branch performance, suggesting both the AE’s focus on pixel reconstruction and the VAE’s attention to latent distribution anomalies are essential (Deressa et al., 2023, Batista, 3 Apr 2025).

2. Mathematical Formulation

• AE Loss $I_A$8 where $I_A$9 is the AE reconstruction.
• VAE Loss $I_B$0 $I_B$1

$I_B$2

• Classification Loss Cross-entropy for each branch: $I_B$3 Total loss: $I_B$4 with $I_B$5 and $I_B$6 (Deressa et al., 2023).

3. Training and Evaluation Protocol

• Data and Preprocessing:

GenConViT is evaluated on DFDC, FF++, TM, DeepfakeTIMIT, and Celeb-DF v2. Faces are extracted using OpenCV, face_recognition, and BlazeFace, resized to $I_B$7, and aggressively augmented (Albumentations): RandomRotate, Transpose, Horizontal/Vertical Flip, GaussNoise, ShiftScaleRotate, CLAHE, Sharpen, IAAEmboss, RandomBrightnessContrast, HueSaturationValue, normalized to ImageNet mean/std (Deressa et al., 2023, Batista, 3 Apr 2025).

• Split:

80% train, 15% validation, 5% test at image-level; 3,972 videos held out for video-level testing with 15 frames per video. Final image count: 1,004,810 (Deressa et al., 2023).

• Optimization:

Adam optimizer ($I_B$8, weight decay $I_B$9); batch size: 32 (AE), 16 (VAE), for 30 epochs. Backbone weights are loaded from timm’s convnext_tiny and swin_tiny_patch4_window7_224 (Deressa et al., 2023).

• Metrics:

Video-level predictions via frame-wise aggregation. Performance is reported using accuracy, F1-score, ROC curve, and AUC (Deressa et al., 2023, Batista, 3 Apr 2025).

4. Quantitative Results and Comparative Analysis

GenConViT achieves high detection performance across major benchmarks:

Dataset	Accuracy	AUC	F1
DFDC	98.50%	99.90%	99.10%
FF++	97.00%	99.60%	97.10%
TIMIT	98.28%	—	—
Celeb-DFv2	90.94%	98.10%	—
Average	95.68%	99.33%	—

Per-branch analysis on DFDC, FF++, and Celeb-DFv2 shows identical accuracy (98.5%), AUC (99.9%), and F1 (0.984) for both AE and VAE branches (Deressa et al., 2023).

In broader comparative studies (e.g., DeepSpeak), GenConViT after fine-tuning achieves accuracy 93.82% (AE branch), ROC AUC 0.993, and F1-score 0.938—outperforming Xception, EfficientNet-B4, and Meso4Inc architectures (Batista, 3 Apr 2025). On the DeepfakeBenchmark, GenConViT demonstrates superior generalization and effectiveness after fine-tuning (Batista, 3 Apr 2025).

5. Advances and Variants

Subsequent research has extended GenConViT through targeted modifications:

• Weighted Loss and Restricted Augmentation:

Explicit reweighting of the cross-entropy loss compensates for pronounced class imbalance (e.g., $$3 \rightarrow 16 \rightarrow 32 \rightarrow 64 \rightarrow 128 \rightarrow 256$$0, $$3 \rightarrow 16 \rightarrow 32 \rightarrow 64 \rightarrow 128 \rightarrow 256$$1 for Celeb-DFv2), raising F1 scores by ≈4.5 points. A restricted augmentation policy—limited to horizontal flips and minor rotations—prevents spurious artifacts on real images, further boosting F1 (Monu et al., 2024).

• Masked-Eye Pretraining (MEP) and Hardness-Inspired Curriculum:

Eye regions are occluded during early training epochs, compelling the network to utilize other facial cues (e.g., lip, skin texture, and blending artefacts). A three-stage curriculum (masked-eye pretrain, cropped-face fine-tuning, and full-image fine-tuning) incrementally increases task difficulty, pushing performance from Acc 93.33%/F1 0.8408 to Acc 98.36%/F1 0.9521 on Celeb-DFv2 (Monu et al., 2024).

6. Runtime, Limitations, and Future Directions

• Efficiency:

The AE branch is 3–4× slower at inference than the VAE; VAE checkpoints are significantly larger (5.2 GB vs 450 MB). Joint operation maximizes accuracy but is resource-intensive (Batista, 3 Apr 2025).

• Failures and Limitations:

Some subtle manipulations outside the trained distribution evade detection. The model’s generalization can decline for out-of-distribution attacks absent targeted domain adaptation or further regularization. No formal OOD ablation was reported in the original work—though “zero-shot” results on DeepfakeTIMIT and Celeb-DFv2 remain strong (98.3% and 90.9% accuracy, respectively) (Deressa et al., 2023).

• Potential Improvements:

Future work could implement explicit KL annealing schedules, frequency-domain autoencoding, adversarial fine-tuning, dynamic branch selection for efficiency, and add temporal or multimodal modeling for continual adaptation to evolving forgeries (Deressa et al., 2023, Batista, 3 Apr 2025).

7. Implementation, Code Availability, and Practical Usage

Reference implementations are available at https://github.com/erprogs/GenConViT. PyTorch instantiation (per the code):

$$3 \rightarrow 16 \rightarrow 32 \rightarrow 64 \rightarrow 128 \rightarrow 256$$2 (Deressa et al., 2023)

The model’s dual-branch, data-augmented, and curriculum-trained design yields state-of-the-art video-level deepfake detection, offering robust detection on large-scale and emerging benchmarks, and forms the basis for further advances in digital media forensics (Deressa et al., 2023, Batista, 3 Apr 2025, Monu et al., 2024).