X-AVFake: Multimodal Forensics Benchmark

• X-AVFake is a fine-grained multimodal benchmark for explainable video forensics that advances beyond binary deepfake detection by providing detailed annotations.
• It employs spatial-temporal localization and explicit reasoning categories to annotate manipulation type, region, temporal anchors, and justification.
• The benchmark supports tasks like detection, manipulation type classification, localization, and reasoning, setting a new standard for evaluating complex deepfake identifications.

X-AVFake is a fine-grained multimodal benchmark for explainable video forensics introduced alongside FakeHunter. It is designed for audio-video authenticity analysis that goes beyond binary real-versus-fake judgment by annotating manipulation type, region or entity, temporal anchors, violated reasoning category, and free-form justification. In that formulation, X-AVFake shifts the evaluation target from conventional deepfake detection toward structured forensic reasoning over both modalities, with emphasis on localization in space and time and on explicit explanation of why a clip violates physical, semantic, or narrative expectations (Chen et al., 20 Aug 2025).

1. Origins and conceptual scope

X-AVFake was proposed to address three limitations attributed to earlier datasets: a predominantly unimodal focus, binary labels only, and limited support for reasoning or explanations. The benchmark is therefore framed as a testbed for multimodal forensics in which a model must determine not only whether content is manipulated, but also what was manipulated, where it occurs, and why the resulting clip is implausible or inconsistent (Chen et al., 20 Aug 2025).

This orientation distinguishes X-AVFake from earlier audio-visual deepfake work centered on talking-head detection. For example, FTFDNet with the Audio-Visual Attention Mechanism (AVAM) was designed for speech-driven fake talking faces and reported a detection rate above 97% on FTFDD, but its problem setting remained binary detection of fake talking-face videos rather than fine-grained explanation or structured reasoning (Wang et al., 2022). A later line of work reformulated audio-visual deepfake detection as prompted multimodal question answering, such as AV-LMMDetect, which asks whether a clip is real or fake and reports strong results on FakeAVCeleb and MAVOS-DD; however, its output space is still fundamentally a constrained binary verdict rather than the richer annotation space emphasized by X-AVFake (Cao et al., 25 Feb 2026).

The benchmark’s central idea is therefore forensic explicability. Instead of evaluating only discriminative accuracy, it evaluates whether a model can connect multimodal evidence to explicit reasoning categories and to localized manipulated content. This suggests a broader conception of detection in which evidential grounding is part of the task definition rather than an after-the-fact diagnostic layer.

2. Dataset composition and construction

X-AVFake is described as containing 5.7k+ video sessions and 950+ minutes of content. In the comparative dataset table, its modality is A+V, its number of real videos is 5700, its number of fake videos is 5700, and its manipulation types are listed as 2. Every real video has a paired manipulated version, so the dataset is balanced by construction (Chen et al., 20 Aug 2025).

Property	Value
Modality	A+V
Manipulation types	2
# Real	5700
# Fake	5700
Duration	950+ minutes

The two manipulation families are visual object removal and audio content replacement. The visual subset contains videos in which the tampering is in the image or video stream only; the audio subset contains videos in which the tampering is in the audio stream only; and the full benchmark combines both into a multimodal evaluation setting (Chen et al., 20 Aug 2025).

Visual object removal is produced by selecting a target entity or object, tracking it across frames with Grounded SAM 2, generating pixel-level masks, and then removing it with ProPainter. The manipulated result is intended to remain visually clean while becoming semantically inconsistent, such as when a critical object disappears although other cues imply that it should remain present. The manipulation label for this family is often visual-delete (Chen et al., 20 Aug 2025).

Audio manipulation is produced as audio replacement or dubbing according to a JSON manipulation plan. The pipeline uses Seeing-and-Hearing diffusion-based tools to generate new audio segments guided by visual cues. The resulting manipulations can create voice mismatches, semantic mismatch between speech and visible events, or temporal misalignment between audio and video. The manipulation label for this family is often audio-replace (Chen et al., 20 Aug 2025).

The benchmark description does not provide explicit train, validation, and test percentages. It is characterized primarily as an evaluation benchmark for FakeHunter, although the associated memory bank is built from real samples during the FakeHunter pipeline (Chen et al., 20 Aug 2025).

3. Annotation schema and reasoning structure

X-AVFake is richly annotated at several levels. Each sample includes file-level identifiers, a binary authenticity label, a manipulation type label, a modality flag, region or entity annotations, temporal localization, and a reasoning trace stored in JSON. For visual manipulations, the annotations include pixel-level masks derived from Grounded SAM 2 plus tracking. For audio manipulations, the annotations include precise time segments for replaced audio (Chen et al., 20 Aug 2025).

A central component is the taxonomy of reasoning violations. The benchmark defines categories labeled A–F: Physical Laws, Time/Season, Location/Culture, Role/Profession, Causality/Order, and Narrative Context. The description also notes a discrepancy: the table header refers to “7 Categories of Reasoning Violations,” but only six labels, A–F, are printed. The operational use of the benchmark nevertheless centers on these six named categories (Chen et al., 20 Aug 2025).

These categories specify why a manipulated sample is implausible. Physical Laws concerns violations such as an effect appearing without a visible cause. Time/Season captures conflict between spoken or visual cues and temporal context. Location/Culture covers inconsistency between cultural or locational references and the visible scene. Role/Profession captures mismatch between an apparent role and the accompanying statement. Causality/Order formalizes breaks in event sequencing. Narrative Context captures failures of referential coherence, such as dialogue referring to an entity that has been visually removed (Chen et al., 20 Aug 2025).

The reasoning trace is itself structured. For each manipulated sample, the stored JSON includes manipulation_type, target_entity, violated_category, justification, and temporal_anchor. This trace is used both during data generation and as supervision for explainable models that are expected to output similar structured analyses at inference time (Chen et al., 20 Aug 2025).

A plausible implication is that X-AVFake operationalizes explanation as a supervised prediction target rather than a free-form byproduct. That design choice makes it suitable for evaluating systems whose outputs must be inspectable, auditable, and tied to explicit evidence.

4. Tasks and evaluation protocols

X-AVFake supports four core tasks: binary deepfake detection, manipulation type classification, localization, and explainability or reasoning. Binary detection requires deciding whether a video is real or manipulated. Manipulation type classification requires identifying whether the manipulation is visual or audio and, for fake samples, distinguishing visual-delete from audio-replace. Localization concerns manipulated regions in video and manipulated time segments in audio. Explainability requires structured verdicts in JSON containing fields such as label, type, region or timestamp, and explanation (Chen et al., 20 Aug 2025).

The reported main metric is accuracy, evaluated over the overall dataset and over the audio and visual subsets separately. On this benchmark, the task is difficult. FTCN, a video-only detector, reports 0.00% on the visual subset; AASIST, an audio-only anti-spoofing model, reports 18.69% on the audio subset; Qwen2.5-Omni-7B reports 18.68% overall, 12.27% on the audio subset, and 24.83% on the visual subset; MiniCPM-o-2.6 reports 9.19% overall, 17.88% on the audio subset, and 0.78% on the visual subset (Chen et al., 20 Aug 2025).

FakeHunter is the principal model evaluated on X-AVFake. With a Qwen2.5 backbone, it reports 34.75% overall, 23.00% on the audio subset, and 46.50% on the visual subset. With a MiniCPM backbone, it reports 27.00% overall, 25.50% on the audio subset, and 28.50% on the visual subset. The low absolute values, including for large multimodal models, indicate that the benchmark is challenging in a way not captured by conventional binary talking-face benchmarks (Chen et al., 20 Aug 2025).

The ablation results further expose that difficulty. Raw Qwen2.5-Omni-7B yields 18.68% overall. FakeHunter with memory and without tool use yields 21.00% overall, 1.50% on the audio subset, and 40.50% on the visual subset. FakeHunter without memory and with tool use yields 27.00% overall, with 27.00% on both audio and visual subsets. The full system yields 34.75% overall, 23.00% on the audio subset, and 46.50% on the visual subset (Chen et al., 20 Aug 2025).

The benchmark therefore measures not only whether a detector can separate real from fake, but whether it can reason over subtle multimodal inconsistencies, identify manipulated content at the correct granularity, and express that reasoning in a structured form.

5. Relation to FakeHunter and to the broader research landscape

X-AVFake is tightly coupled to FakeHunter, a multimodal deepfake detection framework based on memory-guided retrieval, chain-of-thought-style Observation–Thought–Action reasoning, and tool-augmented verification. FakeHunter encodes visual content with CLIP and audio with CLAP, forms 1024-dimensional concatenated audio-visual embeddings, retrieves similar real exemplars from a FAISS-indexed memory bank, and emits structured JSON verdicts specifying what was modified, where it occurs, and why it is judged fake. Its memory bank is built from real videos, uses up to 10,000 samples, and is clustered into 300 prototypes via K-Means (Chen et al., 20 Aug 2025).

The system-level implementation details also reveal the benchmark’s operating assumptions. In the FakeHunter pipeline, videos are sampled at 1 FPS and limited to either 128 frames or 30 seconds of audio, while audio is split into chunks aligned with keyframes. The pipeline processes a 10-minute clip in 8 minutes on a single NVIDIA A800 or 2 minutes on four GPUs, corresponding to 0.8x real-time and 0.2x, respectively (Chen et al., 20 Aug 2025).

Within the broader audio-visual forensics literature, X-AVFake occupies a distinct niche. FakeAVCeleb provides multimodal labels for real versus manipulated audio and video and is widely used for AV detectors, but its emphasis is on human-centric deepfakes and multimodal classification rather than explainable reasoning with structured justifications (Khalid et al., 2021). SAVe and AVPF push generalization through real-only or pseudo-fake training, focusing on visual artifacts and audio-visual alignment, yet their primary objective remains robust detection rather than explanation (Shahzad et al., 26 Mar 2026, Wei et al., 10 Apr 2026). AVFakeBench later expands the benchmark landscape toward general-subject audio-video forgery with 12K questions, seven forgery types, and four annotation levels, but it is positioned as a broader AV-LMM benchmark rather than the specific reasoning-centered corpus introduced with FakeHunter (Xia et al., 26 Nov 2025).

This comparison places X-AVFake between classic deepfake datasets and later large-scale AV-LMM benchmarks: it is narrower in manipulation taxonomy than some later resources, but richer in reasoning supervision than most earlier ones.

6. Significance, limitations, and prospective development

X-AVFake’s significance lies in its treatment of explainability as a first-class benchmark target. The benchmark is designed around realism and diversity of reasoning failure modes rather than around face-swap or lip-sync artifacts alone. Its visual manipulations use Grounded SAM 2 and ProPainter; its audio manipulations use Seeing-and-Hearing; and its annotation pipeline stores explicit reasoning traces aligned with manipulation type and localization (Chen et al., 20 Aug 2025).

At the same time, the benchmark has clear limitations. It focuses on two primary tampering families, visual object removal and audio content replacement. It does not provide explicit train, validation, and test split percentages. The description does not report inter-annotator agreement or a detailed human quality-control study for the explanations. The benchmark is also introduced primarily as an evaluation set, and the reported results show that even advanced systems remain far from saturation (Chen et al., 20 Aug 2025).

A plausible implication is that X-AVFake should be read less as a general-purpose deepfake corpus and more as a targeted stress test for multimodal reasoning under semantically grounded manipulations. Its difficulty stems from the fact that many examples are not face-centric and do not reduce to obvious artifact spotting. Instead, successful models must connect local edits to global semantic violations across time and modality.

In that sense, X-AVFake marks a transition in audio-visual forensics: from benchmarks centered on perceptual realism or binary detection toward benchmarks that ask whether a model can produce a forensic account of a manipulation. That shift aligns it with the emerging role of multimodal large models in deepfake detection, but also exposes how far current systems remain from reliable, fine-grained, and grounded explanation (Cao et al., 25 Feb 2026).