---
title: 'FakePartsBench: Partial Deepfake Benchmark'
url: https://www.emergentmind.com/topics/fakepartsbench
type: topic
---

# FakePartsBench: Partial Deepfake Benchmark

Searching arXiv for FakePartsBench and closely related benchmark papers to ground the article.
FakePartsBench is a benchmark dataset introduced in the paper "FakeParts: a New Family of AI-Generated DeepFakes" [2508.21052]. It is designed for the evaluation of **partial deepfakes**: videos in which only specific spatial regions or temporal segments are manipulated while the remaining content stays authentic. The benchmark is presented as the first large-scale dataset specifically designed to capture the full spectrum of partial deepfakes, and it combines recent full-video generators with localized editing methods, together with pixel-level and frame-level manipulation annotations. In the paper’s own reporting, the abstract describes the benchmark as comprising "over 25K videos," while the comparison table lists **16,000 real videos** and **25,000 fake videos**, implying **41,000 videos** in total [2508.21052].

## 1. Conceptual basis and threat model

FakePartsBench is organized around the paper’s distinction between **Full Deepfakes** and **FakeParts**. Full deepfakes are described as fully synthetic or heavily modified videos, typically produced by **text-to-video (T2V)**, **image-to-video (I2V)**, and **text-image-to-video (TI2V / IT2V)** systems. FakeParts, by contrast, are fine-grained manipulations that affect only part of a video [2508.21052].

The paper divides FakeParts into three subtypes. **Spatial FakeParts** are confined to specific image regions and include **FaceSwap**, **Inpainting**, and **Outpainting**. **Temporal FakeParts** are localized in time and are represented by **Frame interpolation**. **Style FakeParts** are appearance edits that preserve structure and semantics and are represented by **Style / color change**. The defining property is that the majority of the video remains real while only selected regions or segments are altered [2508.21052].

This threat model differs from standard deepfake benchmarks that emphasize global synthesis or full-scene modification. The paper argues that partial manipulations are especially deceptive because authentic surrounding context acts as camouflage. It reports that FakeParts reduce human detection accuracy by **over 30%** compared to traditional deepfakes in the abstract, and that detection rates drop by **over 40%** in the introduction. It also states that state-of-the-art detection models show degradation of up to **43%** when moving from fully synthetic content to partial manipulations [2508.21052].

## 2. Dataset composition and generation pipeline

The benchmark includes both recent full-video deepfakes and localized edits generated from authentic videos. The paper reports that the fake portion is nearly balanced between **12,000 full deepfakes** and **13,000 FakeParts**, giving **25,000 fake videos** overall [2508.21052].

| Category | Methods | Count |
|---|---|---:|
| T2V | Sora, Veo2, Allegro AI | 6,000 |
| I2V | Sora, Veo2, Allegro AI | 3,000 |
| TI2V / IT2V | Sora, Veo2, Allegro AI | 3,000 |
| FaceSwap | InsightFace | 3,000 |
| Inpainting | DiffuEraser, ProPainter | 4,000 |
| Outpainting | AKiRa | 2,000 |
| Interpolation | Framer | 2,000 |
| Style change | RAVE | 2,000 |

The sources of authentic video include **DAVIS 2016**, **DAVIS 2017**, **YouTube-VOS 2019**, **MOSE**, and **LVD-2M**. The method table also lists **Real - ytb** with **16,000 videos**, resolution **1280×720**, **28 FPS**, and duration **7 s**. Additional source datasets are used for specific manipulations: **Celeb-DF** as target videos for FaceSwap, **CelebA** and additional celebrity face images as source identities, and **Animal Kingdom** for style-change videos [2508.21052].

The generation pipeline is split into full deepfakes and FakeParts. For full deepfakes, prompts are extracted using a VLM; for I2V and TI2V, the **first frame** from a real-video segment is used as the condition; and both the prompt and the conditioning frame are stored as metadata. The default VLM is **PaLI-Gemma 2 (3B)** fine-tuned on **VQAv2**. For FaceSwap, the method is **InsightFace**, and gender consistency is enforced using VLM prompting and filtering of unmatched pairs. For inpainting, the pipeline is: extract an initial frame, use a VLM to identify a salient object, segment the object across frames with **Grounded-SAM-2**, and then inpaint the masked region using **DiffuEraser** or **ProPainter**. The outpainting method is **AKiRa**, which uses camera-controlled coherent extrapolation conditioned with backwards camera tracking. Interpolation uses **Framer**, which takes the first and last frames and generates **21 intermediate frames**. Style change uses **RAVE** with the prompt "Change the color of the animal in the video" [2508.21052].

The paper notes a minor inconsistency in method counting. The prose states that content is generated using **ten** state-of-the-art models, whereas the benchmark comparison table reports **#Meth. = 9**; the table explicitly enumerates **Sora, Veo2, Allegro AI, Framer, RAVE, DiffuEraser, ProPainter, InsightFace, and AKiRa** [2508.21052].

## 3. Annotation structure and benchmark tasks

FakePartsBench provides **pixel-level manipulation annotations** and **frame-level manipulation annotations**. The paper also describes these as **fine-grained spatial and temporal annotations**, with pixel-level precision for region-based manipulations and explicit frame-level labels for manipulated video segments [2508.21052].

For spatial manipulations, the benchmark supports pixel-level masks, which is consistent with the use of **Grounded-SAM-2** in the inpainting pipeline and with the paper’s claim of pixel-level precision. For temporal manipulations, the paper states that fine-grained spatial and temporal annotations are available, although it does not formalize a temporal-boundary annotation schema or provide quantitative statistics such as average manipulated span length [2508.21052].

The benchmark directly supports several tasks: **video-level real/fake classification**, **frame-level fake detection**, **spatial localization / segmentation of manipulated regions**, **temporal localization of manipulated segments**, and **category-wise robustness analysis across manipulation types**. At the same time, the experiments in the paper mainly instantiate **video-level classification** and some **frame-wise/image-level detector evaluation**. Dedicated segmentation or temporal localization baselines are not reported, despite the annotation granularity [2508.21052].

The benchmark also stores metadata associated with generation, including prompts and conditioning frames for I2V and TI2V. A plausible implication is that FakePartsBench can function not only as a detector benchmark but also as a controlled test bed for studying the interaction between manipulation type, conditioning signal, and detector failure mode.

## 4. Evaluation protocol, baselines, and reported metrics

The paper evaluates both image-level and video-level deepfake detectors. The image-level baselines are **CNNDetection**, **UnivFD**, **NPR**, **FatFormer**, and **C2P-CLIP**. All process video frames independently as **224×224** image patches. Two resizing strategies are tested: **uniform resize** to **224×224**, and **aspect-preserving resize + center crop**, in which the shorter side is resized to **224**. The video-level baselines are **DeMamba** and **AIGVDet** [2508.21052].

The main benchmark table reports, for each category, the **mean predicted probability that the sample is classified as fake**. The categories are **T2V**, **I2V**, **IT2V**, **Stylechange**, **Faceswap**, **Real**, **Interpolation**, **Inpainting**, and **Outpainting**. The paper also reports **Acc. on orig.**, **All Avg.**, **Full Fake Avg.**, and **FakeParts Avg.**. In Appendix Table 5, the reported classification metrics are **F1 Score**, **Average Precision (AP)**, and **Accuracy** [2508.21052].

For dataset realism and quality, the comparison table reports **FVD\_W = 240.8**, **FVD\_FP = 211.5**, and VBench metrics of **Consistency = 0.940**, **Temporal Flicker = 0.970**, and **Quality = 0.623**. The paper does not give explicit formulas for these metrics, and it likewise does not define explicit formulas for localization-oriented quantities such as **IoU**, **mIoU**, **temporal IoU**, **mask AP**, or temporal-boundary losses. This indicates that the benchmark’s current evaluation is primarily **classification-oriented**, even though the annotations support finer-grained tasks [2508.21052].

Within the reported detector summary, the best automated aggregate result in Appendix Table 5 is obtained by **C2P-CLIP**, with **F1 = 0.467**, **AP = 0.987**, and **Accuracy = 0.651**. Human detection is reported at **F1 = 0.750**, **AP = 0.755**, and **Accuracy = 0.751** [2508.21052].

## 5. Empirical findings

The paper’s main empirical conclusion is that all evaluated detectors degrade substantially on FakePartsBench. This applies to both image-level and video-level methods. The most extreme case is **CNNDetection**, which collapses to near-zero output across essentially all categories, despite **0.997** original accuracy on its native benchmark. In the benchmark table, its **All Avg.** is **0.001**, with a drop of **(-0.996)** [2508.21052].

The reported failure pattern is differentiated by detector family. **AIGVDet**, **NPR**, and **DeMamba** perform relatively better on **T2V**, **I2V**, and **IT2V**, but perform poorly on **inpainting**, **outpainting**, and often **faceswap**. **UnivFD**, **FatFormer**, and **C2P-CLIP** generalize better to several partial-manipulation categories, but often underperform on recent high-fidelity full-video generation from **Sora**, **Veo2**, and **Allegro**. The paper interprets this as a trade-off: artifact-centric detectors help on full synthetic videos, whereas foundation-model or semantic detectors help on partial edits, and neither strategy is robust across both regimes [2508.21052].

Human evaluation shows a similar vulnerability. The paper reports slightly inconsistent participant counts across sections: the main text states **80 participants**, the introduction and abstract say **over 60 participants**, and the appendix states **60 individuals**. Each participant labels **20** randomly selected clips in a Streamlit-based survey. The reported overall human accuracy is about **75.3%** in the main text, while the appendix gives **Accuracy = 0.751**. Category-wise mean fake probabilities show that humans detect **Stylechange** most easily at **0.983**, while **Inpainting** at **0.588**, **Faceswap** at **0.612**, and **Interpolation** at **0.676** are harder [2508.21052].

The paper also includes a branch analysis of **AIGVDet**. In Appendix Table 4, the **spatial-domain branch** dominates for most categories, while the **optical-flow branch** helps particularly for **Faceswap** and somewhat for **Inpainting**. For example, on **Faceswap**, the combined score is **0.216**, the spatial branch is **0.067**, and the optical-flow branch is **0.366**. This supports the claim that motion cues are useful for some FakeParts categories but insufficient as a general solution [2508.21052].

A broader significance claim follows directly from these results: detectors that appear strong on prior deepfake benchmarks can still fail badly on localized manipulations. The benchmark therefore exposes a blind spot that is not captured when evaluation is restricted to full synthetic videos.

## 6. Benchmark position, internal inconsistencies, and limitations

FakePartsBench is motivated by two gaps identified in prior diffusion-era video deepfake datasets: the absence of dedicated coverage for partial manipulations and insufficient perceptual quality or realism. In the paper’s landscape table, FakePartsBench is the only dataset marked positive for **Spatial**, **Temporal**, and **Style** FakeParts support. It is also presented as having stronger realism statistics than competing datasets such as **GVD**, **VidProM**, **GenVidBench**, and **DeMamba**, as reflected in the reported **FVD** and **VBench** scores [2508.21052].

The benchmark nonetheless has several stated and implied limitations. The authors explicitly say that dataset size is still limited for **training-scale use**, since generation is costly and complex, and they describe the current release as more suited to **evaluation** than to massive detector training. They also identify finer control over manipulation extent as future work, specifically mentioning **varying inpainting surface areas** and studying how size affects detectability [2508.21052].

The paper also contains a number of numerical inconsistencies. The abstract’s "over **25K videos**" does not align directly with the comparison table’s **16,000 real** plus **25,000 fake**. Participant counts vary between **60** and **80**. Method counts vary between **9** and **10**. In addition, the paper does not clearly specify exact **train/val/test counts**, per-category split breakdowns, or subject- or source-disjoint protocols in the provided text. These are not merely editorial details; they matter for strict reproducibility and for comparative benchmark design [2508.21052].

Another limitation is methodological rather than quantitative. Although FakePartsBench includes rich pixel-level and frame-level annotations, the paper does not yet provide localization or segmentation baselines that exploit them. This suggests that the benchmark’s present contribution is strongest as a **classification and robustness benchmark**, with the annotation structure already in place for future work on joint classification, localization, temporal boundary detection, and mask-aware deepfake forensics [2508.21052].

Source: https://www.emergentmind.com/topics/fakepartsbench