---
title: 'GenConViT: Hybrid Deepfake Detection'
url: https://www.emergentmind.com/topics/genconvit
type: topic
---

# GenConViT: Hybrid Deepfake Detection

GenConViT is a two-branch, hybrid deepfake video detection architecture combining convolutional neural networks (ConvNeXt) with hierarchical vision transformer modules (Swin Transformer), optimized to extract both fine-grained local artifacts and global contextual inconsistencies from manipulated face videos. The model integrates both deterministic (Autoencoder; AE) and probabilistic (Variational Autoencoder; VAE) generative paradigms within its architecture to capture a diverse set of forgery cues. GenConViT was originally proposed for robust detection across a variety of deepfake manipulation techniques and datasets, establishing new performance benchmarks while informing the design of subsequent variants and derivatives [2307.07036, 2504.02900, 2410.05466].

## 1. Model Architecture

GenConViT consists of two parallel branches—Network A, which employs a plain Autoencoder, and Network B, which utilizes a Variational Autoencoder. Each branch processes both the original input face frame $X$ and its reconstructed version ($I_A$ or $I_B$), and each representation is separately passed through identical ConvNeXt-Swin hybrid modules for feature extraction.

- **Autoencoder (AE) Branch (Network A)**  
  - Encoder: 5 convolutional layers ($3 \rightarrow 16 \rightarrow 32 \rightarrow 64 \rightarrow 128 \rightarrow 256$), each with 3×3 kernels, stride 2, ReLU and max pooling (2×2, stride 2). Output latent tensor $Z \in \mathbb{R}^{256 \times 7 \times 7}$.
  - Decoder: 5 transposed-convolutional layers ($256 \rightarrow 128 \rightarrow 64 \rightarrow 32 \rightarrow 16 \rightarrow 3$), kernel 2×2, stride 2, ReLU. Output $I_A \in \mathbb{R}^{224 \times 224 \times 3}$ [2307.07036].

- **Variational Autoencoder (VAE) Branch (Network B)**  
  - Encoder: 4 convolutional layers ($3 \rightarrow 16 \rightarrow 32 \rightarrow 64 \rightarrow 128$), kernel 3×3, stride 2; each with BatchNorm, LeakyReLU; flatten to 12,544-d vector, split into mean $\mu(X)$ and $\log \sigma^2(X)$.
  - Reparameterization: $z = \mu + \sigma \odot \epsilon$, $\epsilon \sim \mathcal{N}(0,I)$.
  - Decoder: 4 transposed convolutional layers ($256 \rightarrow 64 \rightarrow 32 \rightarrow 16 \rightarrow 3$), kernel 2×2, stride 2; each with LeakyReLU. Output $I_B \in \mathbb{R}^{112 \times 112 \times 3}$ [2307.07036].

- **ConvNeXt-Swin Hybrid Module**  
  - ConvNeXt_tiny backbone (pretrained on ImageNet-1k) produces feature maps $F_{conv}$.
  - 1×1 convolution projects $C \rightarrow 768$, resulting in a sequence of 768-d tokens.
  - Swin_tiny_patch4_window7_224 operates on these tokens, yielding a 1,000-dimensional feature vector per input.

Each original/reconstructed frame pair ($X, I_A$; $X, I_B$) is mapped via their respective hybrid modules to two 1,000-dim feature vectors, concatenated and passed to a two-way classifier for real/fake decision. The model design allows both branches to contribute equally; empirical results indicate nearly identical per-branch performance, suggesting both the AE’s focus on pixel reconstruction and the VAE’s attention to latent distribution anomalies are essential [2307.07036, 2504.02900].

## 2. Mathematical Formulation

- **AE Loss**  
  $$L_\mathrm{rec}^{AE} = \| X - X' \|_2^2$$  
  where $X'$ is the AE reconstruction.

- **VAE Loss**  
  $$L_\mathrm{rec}^{VAE} = \mathbb{E}_\epsilon [ \| X - X'' \|_2^2 ]$$  
  $$D_{KL} = \frac{1}{2}\sum_{i=1}^K [ \mu_i^2 + \sigma_i^2 - \log \sigma_i^2 - 1 ]$$  
  $$L_{VAE} = L_\mathrm{rec}^{VAE} + \beta \cdot D_{KL},\ \beta=1$$

- **Classification Loss**  
  Cross-entropy for each branch:
  $$L_\mathrm{CE}(p, y) = -\sum_{c \in \{0,1\}} \mathbb{1}_{[y = c]}\log p_c$$  
  Total loss:
  $$L_\mathrm{total} = L_A + L_B$$  
  with $L_A=L_\mathrm{CE}(p_A, y)$ and $L_B = L_\mathrm{CE}(p_B, y) + \lambda_\mathrm{rec} L_\mathrm{rec}^{VAE} + \lambda_{KL} D_{KL}$ [2307.07036].

## 3. Training and Evaluation Protocol

- **Data and Preprocessing:**  
  GenConViT is evaluated on DFDC, FF++, TM, DeepfakeTIMIT, and Celeb-DF v2. Faces are extracted using OpenCV, face_recognition, and BlazeFace, resized to $224 \times 224$, and aggressively augmented (Albumentations): RandomRotate, Transpose, Horizontal/Vertical Flip, GaussNoise, ShiftScaleRotate, CLAHE, Sharpen, IAAEmboss, RandomBrightnessContrast, HueSaturationValue, normalized to ImageNet mean/std [2307.07036, 2504.02900].

- **Split:**  
  80% train, 15% validation, 5% test at image-level; 3,972 videos held out for video-level testing with 15 frames per video. Final image count: 1,004,810 [2307.07036].

- **Optimization:**  
  Adam optimizer ($\mathrm{lr}=1 \times 10^{-4}$, weight decay $1 \times 10^{-4}$); batch size: 32 (AE), 16 (VAE), for 30 epochs. Backbone weights are loaded from timm’s convnext_tiny and swin_tiny_patch4_window7_224 [2307.07036].

- **Metrics:**  
  Video-level predictions via frame-wise aggregation. Performance is reported using accuracy, F1-score, ROC curve, and AUC [2307.07036, 2504.02900].

## 4. Quantitative Results and Comparative Analysis

GenConViT achieves high detection performance across major benchmarks:

| Dataset         | Accuracy | AUC     | F1   |
|-----------------|----------|---------|------|
| DFDC            | 98.50%   | 99.90%  | 99.10% |
| FF++            | 97.00%   | 99.60%  | 97.10% |
| TIMIT           | 98.28%   | —       | —  |
| Celeb-DFv2      | 90.94%   | 98.10%  | —   |
| **Average**     | 95.68%   | 99.33%  | —   |

Per-branch analysis on DFDC, FF++, and Celeb-DFv2 shows identical accuracy (98.5%), AUC (99.9%), and F1 (0.984) for both AE and VAE branches [2307.07036].

In broader comparative studies (e.g., DeepSpeak), GenConViT after fine-tuning achieves accuracy 93.82% (AE branch), ROC AUC 0.993, and F1-score 0.938—outperforming Xception, EfficientNet-B4, and Meso4Inc architectures [2504.02900]. On the DeepfakeBenchmark, GenConViT demonstrates superior generalization and effectiveness after fine-tuning [2504.02900].

## 5. Advances and Variants

Subsequent research has extended GenConViT through targeted modifications:

- **Weighted Loss and Restricted Augmentation:**  
  Explicit reweighting of the cross-entropy loss compensates for pronounced class imbalance (e.g., $w_\mathrm{real}=1.85$, $w_\mathrm{fake}=1$ for Celeb-DFv2), raising F1 scores by ≈4.5 points.  
  A restricted augmentation policy—limited to horizontal flips and minor rotations—prevents spurious artifacts on real images, further boosting F1 [2410.05466].

- **Masked-Eye Pretraining (MEP) and Hardness-Inspired Curriculum:**  
  Eye regions are occluded during early training epochs, compelling the network to utilize other facial cues (e.g., lip, skin texture, and blending artefacts). A three-stage curriculum (masked-eye pretrain, cropped-face fine-tuning, and full-image fine-tuning) incrementally increases task difficulty, pushing performance from Acc 93.33%/F1 0.8408 to Acc 98.36%/F1 0.9521 on Celeb-DFv2 [2410.05466].

## 6. Runtime, Limitations, and Future Directions

- **Efficiency:**  
  The AE branch is 3–4× slower at inference than the VAE; VAE checkpoints are significantly larger (5.2 GB vs 450 MB). Joint operation maximizes accuracy but is resource-intensive [2504.02900].  

- **Failures and Limitations:**  
  Some subtle manipulations outside the trained distribution evade detection. The model’s generalization can decline for out-of-distribution attacks absent targeted domain adaptation or further regularization. No formal OOD ablation was reported in the original work—though “zero-shot” results on DeepfakeTIMIT and Celeb-DFv2 remain strong (98.3% and 90.9% accuracy, respectively) [2307.07036].

- **Potential Improvements:**  
  Future work could implement explicit KL annealing schedules, frequency-domain autoencoding, adversarial fine-tuning, dynamic branch selection for efficiency, and add temporal or multimodal modeling for continual adaptation to evolving forgeries [2307.07036, 2504.02900].

## 7. Implementation, Code Availability, and Practical Usage

Reference implementations are available at https://github.com/erprogs/GenConViT. PyTorch instantiation (per the code):

```python
from timm import create_model
convnext = create_model('convnext_tiny', pretrained=True)
swin     = create_model('swin_tiny_patch4_window7_224', pretrained=True)
ae       = Autoencoder()  # see Sec. 3.3 in code
vae      = VariationalAE()  # see Sec. 3.4 in code
genconv  = GenConViT(ae, vae, convnext, swin)
```
[2307.07036]

The model’s dual-branch, data-augmented, and curriculum-trained design yields state-of-the-art video-level deepfake detection, offering robust detection on large-scale and emerging benchmarks, and forms the basis for further advances in digital media forensics [2307.07036, 2504.02900, 2410.05466].

Source: https://www.emergentmind.com/topics/genconvit