FakePartsBench: Partial Deepfake Benchmark
- FakePartsBench is a benchmark dataset specifically designed to evaluate partial deepfakes, including spatial, temporal, and style manipulations with precise annotations.
- It comprises over 41,000 videos with a balanced mix of authentic and manipulated content generated via 9–10 state-of-the-art models.
- The benchmark supports multiple tasks such as video-level classification, spatial segmentation, and temporal localization, revealing significant performance drops in current detectors.
Searching arXiv for FakePartsBench and closely related benchmark papers to ground the article. FakePartsBench is a benchmark dataset introduced in the paper "FakeParts: a New Family of AI-Generated DeepFakes" (Brison et al., 28 Aug 2025). It is designed for the evaluation of partial deepfakes: videos in which only specific spatial regions or temporal segments are manipulated while the remaining content stays authentic. The benchmark is presented as the first large-scale dataset specifically designed to capture the full spectrum of partial deepfakes, and it combines recent full-video generators with localized editing methods, together with pixel-level and frame-level manipulation annotations. In the paper’s own reporting, the abstract describes the benchmark as comprising "over 25K videos," while the comparison table lists 16,000 real videos and 25,000 fake videos, implying 41,000 videos in total (Brison et al., 28 Aug 2025).
1. Conceptual basis and threat model
FakePartsBench is organized around the paper’s distinction between Full Deepfakes and FakeParts. Full deepfakes are described as fully synthetic or heavily modified videos, typically produced by text-to-video (T2V), image-to-video (I2V), and text-image-to-video (TI2V / IT2V) systems. FakeParts, by contrast, are fine-grained manipulations that affect only part of a video (Brison et al., 28 Aug 2025).
The paper divides FakeParts into three subtypes. Spatial FakeParts are confined to specific image regions and include FaceSwap, Inpainting, and Outpainting. Temporal FakeParts are localized in time and are represented by Frame interpolation. Style FakeParts are appearance edits that preserve structure and semantics and are represented by Style / color change. The defining property is that the majority of the video remains real while only selected regions or segments are altered (Brison et al., 28 Aug 2025).
This threat model differs from standard deepfake benchmarks that emphasize global synthesis or full-scene modification. The paper argues that partial manipulations are especially deceptive because authentic surrounding context acts as camouflage. It reports that FakeParts reduce human detection accuracy by over 30% compared to traditional deepfakes in the abstract, and that detection rates drop by over 40% in the introduction. It also states that state-of-the-art detection models show degradation of up to 43% when moving from fully synthetic content to partial manipulations (Brison et al., 28 Aug 2025).
2. Dataset composition and generation pipeline
The benchmark includes both recent full-video deepfakes and localized edits generated from authentic videos. The paper reports that the fake portion is nearly balanced between 12,000 full deepfakes and 13,000 FakeParts, giving 25,000 fake videos overall (Brison et al., 28 Aug 2025).
| Category | Methods | Count |
|---|---|---|
| T2V | Sora, Veo2, Allegro AI | 6,000 |
| I2V | Sora, Veo2, Allegro AI | 3,000 |
| TI2V / IT2V | Sora, Veo2, Allegro AI | 3,000 |
| FaceSwap | InsightFace | 3,000 |
| Inpainting | DiffuEraser, ProPainter | 4,000 |
| Outpainting | AKiRa | 2,000 |
| Interpolation | Framer | 2,000 |
| Style change | RAVE | 2,000 |
The sources of authentic video include DAVIS 2016, DAVIS 2017, YouTube-VOS 2019, MOSE, and LVD-2M. The method table also lists Real - ytb with 16,000 videos, resolution 1280×720, 28 FPS, and duration 7 s. Additional source datasets are used for specific manipulations: Celeb-DF as target videos for FaceSwap, CelebA and additional celebrity face images as source identities, and Animal Kingdom for style-change videos (Brison et al., 28 Aug 2025).
The generation pipeline is split into full deepfakes and FakeParts. For full deepfakes, prompts are extracted using a VLM; for I2V and TI2V, the first frame from a real-video segment is used as the condition; and both the prompt and the conditioning frame are stored as metadata. The default VLM is PaLI-Gemma 2 (3B) fine-tuned on VQAv2. For FaceSwap, the method is InsightFace, and gender consistency is enforced using VLM prompting and filtering of unmatched pairs. For inpainting, the pipeline is: extract an initial frame, use a VLM to identify a salient object, segment the object across frames with Grounded-SAM-2, and then inpaint the masked region using DiffuEraser or ProPainter. The outpainting method is AKiRa, which uses camera-controlled coherent extrapolation conditioned with backwards camera tracking. Interpolation uses Framer, which takes the first and last frames and generates 21 intermediate frames. Style change uses RAVE with the prompt "Change the color of the animal in the video" (Brison et al., 28 Aug 2025).
The paper notes a minor inconsistency in method counting. The prose states that content is generated using ten state-of-the-art models, whereas the benchmark comparison table reports #Meth. = 9; the table explicitly enumerates Sora, Veo2, Allegro AI, Framer, RAVE, DiffuEraser, ProPainter, InsightFace, and AKiRa (Brison et al., 28 Aug 2025).
3. Annotation structure and benchmark tasks
FakePartsBench provides pixel-level manipulation annotations and frame-level manipulation annotations. The paper also describes these as fine-grained spatial and temporal annotations, with pixel-level precision for region-based manipulations and explicit frame-level labels for manipulated video segments (Brison et al., 28 Aug 2025).
For spatial manipulations, the benchmark supports pixel-level masks, which is consistent with the use of Grounded-SAM-2 in the inpainting pipeline and with the paper’s claim of pixel-level precision. For temporal manipulations, the paper states that fine-grained spatial and temporal annotations are available, although it does not formalize a temporal-boundary annotation schema or provide quantitative statistics such as average manipulated span length (Brison et al., 28 Aug 2025).
The benchmark directly supports several tasks: video-level real/fake classification, frame-level fake detection, spatial localization / segmentation of manipulated regions, temporal localization of manipulated segments, and category-wise robustness analysis across manipulation types. At the same time, the experiments in the paper mainly instantiate video-level classification and some frame-wise/image-level detector evaluation. Dedicated segmentation or temporal localization baselines are not reported, despite the annotation granularity (Brison et al., 28 Aug 2025).
The benchmark also stores metadata associated with generation, including prompts and conditioning frames for I2V and TI2V. A plausible implication is that FakePartsBench can function not only as a detector benchmark but also as a controlled test bed for studying the interaction between manipulation type, conditioning signal, and detector failure mode.
4. Evaluation protocol, baselines, and reported metrics
The paper evaluates both image-level and video-level deepfake detectors. The image-level baselines are CNNDetection, UnivFD, NPR, FatFormer, and C2P-CLIP. All process video frames independently as 224×224 image patches. Two resizing strategies are tested: uniform resize to 224×224, and aspect-preserving resize + center crop, in which the shorter side is resized to 224. The video-level baselines are DeMamba and AIGVDet (Brison et al., 28 Aug 2025).
The main benchmark table reports, for each category, the mean predicted probability that the sample is classified as fake. The categories are T2V, I2V, IT2V, Stylechange, Faceswap, Real, Interpolation, Inpainting, and Outpainting. The paper also reports Acc. on orig., All Avg., Full Fake Avg., and FakeParts Avg.. In Appendix Table 5, the reported classification metrics are F1 Score, Average Precision (AP), and Accuracy (Brison et al., 28 Aug 2025).
For dataset realism and quality, the comparison table reports FVD_W = 240.8, FVD_FP = 211.5, and VBench metrics of Consistency = 0.940, Temporal Flicker = 0.970, and Quality = 0.623. The paper does not give explicit formulas for these metrics, and it likewise does not define explicit formulas for localization-oriented quantities such as IoU, mIoU, temporal IoU, mask AP, or temporal-boundary losses. This indicates that the benchmark’s current evaluation is primarily classification-oriented, even though the annotations support finer-grained tasks (Brison et al., 28 Aug 2025).
Within the reported detector summary, the best automated aggregate result in Appendix Table 5 is obtained by C2P-CLIP, with F1 = 0.467, AP = 0.987, and Accuracy = 0.651. Human detection is reported at F1 = 0.750, AP = 0.755, and Accuracy = 0.751 (Brison et al., 28 Aug 2025).
5. Empirical findings
The paper’s main empirical conclusion is that all evaluated detectors degrade substantially on FakePartsBench. This applies to both image-level and video-level methods. The most extreme case is CNNDetection, which collapses to near-zero output across essentially all categories, despite 0.997 original accuracy on its native benchmark. In the benchmark table, its All Avg. is 0.001, with a drop of (-0.996) (Brison et al., 28 Aug 2025).
The reported failure pattern is differentiated by detector family. AIGVDet, NPR, and DeMamba perform relatively better on T2V, I2V, and IT2V, but perform poorly on inpainting, outpainting, and often faceswap. UnivFD, FatFormer, and C2P-CLIP generalize better to several partial-manipulation categories, but often underperform on recent high-fidelity full-video generation from Sora, Veo2, and Allegro. The paper interprets this as a trade-off: artifact-centric detectors help on full synthetic videos, whereas foundation-model or semantic detectors help on partial edits, and neither strategy is robust across both regimes (Brison et al., 28 Aug 2025).
Human evaluation shows a similar vulnerability. The paper reports slightly inconsistent participant counts across sections: the main text states 80 participants, the introduction and abstract say over 60 participants, and the appendix states 60 individuals. Each participant labels 20 randomly selected clips in a Streamlit-based survey. The reported overall human accuracy is about 75.3% in the main text, while the appendix gives Accuracy = 0.751. Category-wise mean fake probabilities show that humans detect Stylechange most easily at 0.983, while Inpainting at 0.588, Faceswap at 0.612, and Interpolation at 0.676 are harder (Brison et al., 28 Aug 2025).
The paper also includes a branch analysis of AIGVDet. In Appendix Table 4, the spatial-domain branch dominates for most categories, while the optical-flow branch helps particularly for Faceswap and somewhat for Inpainting. For example, on Faceswap, the combined score is 0.216, the spatial branch is 0.067, and the optical-flow branch is 0.366. This supports the claim that motion cues are useful for some FakeParts categories but insufficient as a general solution (Brison et al., 28 Aug 2025).
A broader significance claim follows directly from these results: detectors that appear strong on prior deepfake benchmarks can still fail badly on localized manipulations. The benchmark therefore exposes a blind spot that is not captured when evaluation is restricted to full synthetic videos.
6. Benchmark position, internal inconsistencies, and limitations
FakePartsBench is motivated by two gaps identified in prior diffusion-era video deepfake datasets: the absence of dedicated coverage for partial manipulations and insufficient perceptual quality or realism. In the paper’s landscape table, FakePartsBench is the only dataset marked positive for Spatial, Temporal, and Style FakeParts support. It is also presented as having stronger realism statistics than competing datasets such as GVD, VidProM, GenVidBench, and DeMamba, as reflected in the reported FVD and VBench scores (Brison et al., 28 Aug 2025).
The benchmark nonetheless has several stated and implied limitations. The authors explicitly say that dataset size is still limited for training-scale use, since generation is costly and complex, and they describe the current release as more suited to evaluation than to massive detector training. They also identify finer control over manipulation extent as future work, specifically mentioning varying inpainting surface areas and studying how size affects detectability (Brison et al., 28 Aug 2025).
The paper also contains a number of numerical inconsistencies. The abstract’s "over 25K videos" does not align directly with the comparison table’s 16,000 real plus 25,000 fake. Participant counts vary between 60 and 80. Method counts vary between 9 and 10. In addition, the paper does not clearly specify exact train/val/test counts, per-category split breakdowns, or subject- or source-disjoint protocols in the provided text. These are not merely editorial details; they matter for strict reproducibility and for comparative benchmark design (Brison et al., 28 Aug 2025).
Another limitation is methodological rather than quantitative. Although FakePartsBench includes rich pixel-level and frame-level annotations, the paper does not yet provide localization or segmentation baselines that exploit them. This suggests that the benchmark’s present contribution is strongest as a classification and robustness benchmark, with the annotation structure already in place for future work on joint classification, localization, temporal boundary detection, and mask-aware deepfake forensics (Brison et al., 28 Aug 2025).