DDL-AV Dataset: Audio-Visual Deepfake Benchmark

• DDL-AV is the audio-visual subset of the DDL benchmark, comprising over 0.3M generated audio-video samples for deepfake detection and localization.
• The dataset features diverse forgery scenarios including voice cloning, face swapping, and asynchronous manipulations across both segmented and full-length clips.
• DDL-AV provides fine-grained temporal annotations for both audio and video streams, enabling precise segment-level analysis and interpretable forensic evaluation.

Searching arXiv for the DDL and DDL-AV papers to ground the article in the cited sources. DDL-AV is the audio-visual subset of the DDL benchmark for interpretable deepfake detection and localization in real-world scenarios. Within the broader DDL dataset, which contains over $1.8\text{M}$ forged samples and supports classification, temporal localization, and spatial localization, DDL-AV contributes more than $0.3\text{M}$ audio-video samples and is defined by multimodal forgery diversity, varied manipulation modes, and fine-grained temporal annotations for both audio and video streams (Miao et al., 29 Jun 2025). It is also described as containing both segmented and full-length video clips, enabling evaluation beyond isolated short segments and toward more realistic deployment conditions (Zhang et al., 24 Aug 2025).

1. Position within the DDL benchmark

DDL-AV is not an independent benchmark in isolation; it is the audio-visual component of the larger DDL dataset. The parent DDL resource is organized into an image unimodal subset, DDL-I, with more than $1.5\text{M}$ images, and an audio-visual subset, DDL-AV, with more than $0.3\text{M}$ audio-video samples covering audio, video, and audio-video forgeries (Miao et al., 29 Jun 2025). The DDL paper frames this design around four key innovations: Diverse Forgery Scenarios, Comprehensive Deepfake Methods, Varied Manipulation Modes, and Fine-grained Forgery Annotations.

For DDL as a whole, the benchmark draws on 14+ real datasets for source diversity, including FF++, Celeb-DF, DFDC, FFIW, CelebA, FFHQ, and CelebV-HQ. In the original-source statistics reported for the audio-video branch, the train/valid partition uses VoxCeleb2 (72K), while the test partition uses VoxCeleb2 (10K) and CelebV-HQ (15K), for a total of 97K source samples (Miao et al., 29 Jun 2025). The distinction between 97K source samples and more than $0.3\text{M}$ DDL-AV samples indicates that the released benchmark is a generated and annotated deepfake corpus rather than a mere repackaging of raw source media.

Aspect	DDL-AV
Role in DDL	Audio-visual subset
Scale	$>0.3\text{M}$ audio-video samples
Source-data total	97K original audio-video samples
Primary source datasets	VoxCeleb2, CelebV-HQ
Supported task types in DDL context	Detection and localization

2. Data composition and modality coverage

DDL-AV is explicitly multimodal. It contains both the audio and the visual channel, and its benchmark logic includes cases in which one modality is manipulated while the other remains authentic (Zhang et al., 24 Aug 2025). The DDL paper further specifies that DDL-AV covers audio, video, and audio-video samples, situating it within a broader program of multimodal deepfake detection and localization (Miao et al., 29 Jun 2025).

The benchmark is designed around realistic forgery variability rather than purely synchronized, fully manipulated clips. In the ERF-BA-TFD+ description, DDL-AV includes both segmented and full-length video clips, and each video can include both manipulated and non-manipulated segments (Zhang et al., 24 Aug 2025). This structure matters technically because it turns the task from simple clip-level binary classification into a joint detection-and-localization problem over long temporal contexts.

Public descriptions emphasize modality coverage and annotation granularity more strongly than exhaustive clip-count statistics. The ERF-BA-TFD+ paper states that the dataset contains enough long-duration and full-length video samples to challenge detectors on temporally dispersed manipulations, but the exact number of clips, average length, and number of segments are not given verbatim there (Zhang et al., 24 Aug 2025).

3. Forgery scenarios and manipulation taxonomy

DDL-AV is characterized by a broad forgery space spanning both audio and video. In the benchmark description, audio forgeries include state-of-the-art techniques such as text-to-speech, voice cloning, and voice swapping, while visual forgeries include face swapping, facial animation, and text-to-video (AIGC) generation (Zhang et al., 24 Aug 2025). This places DDL-AV in direct contact with both classical face-manipulation pipelines and newer generative-content workflows.

Three main forgery modes are identified: fake audio & fake video, fake audio & real video, and real audio & fake video (Zhang et al., 24 Aug 2025). The dataset also includes audio-video misalignment, described as asynchronous temporal forgery, such as manipulated audio with real video or the reverse. In the DDL paper’s terminology, forged audio and video content can be fabricated at non-aligned/independent time sequences, and the temporal-domain manipulation repertoire includes content replacement, content deletion, content insertion, and hybrid / asynchronous forgeries (Miao et al., 29 Jun 2025).

These properties distinguish DDL-AV from benchmarks centered on globally forged clips or tightly synchronized manipulations. A common misconception is to treat audio-visual deepfake datasets as if they test only whether a whole clip is fake. DDL-AV is designed for a harder regime in which manipulation may be partial, modality-specific, or temporally non-overlapping. That design is central to its value as a benchmark for interpretable and forensic-style systems.

4. Annotation design and supported tasks

The defining technical feature of DDL-AV is its fine-grained temporal annotation scheme. The DDL paper states that, for audio-visual forgeries, the dataset provides temporal segment labels specifying the precise timing, including start/end of fake segments, for both audio and video streams (Miao et al., 29 Jun 2025). It further notes that audio and video may be forged in partially overlapping or non-overlapping intervals.

The ERF-BA-TFD+ description complements this by stating that DDL-AV provides frame-level binary labels for whether a frame or segment is real or fake, as well as segment-level ground truths to support both classification and localization (Zhang et al., 24 Aug 2025). Taken together, these descriptions place DDL-AV squarely in the category of temporally localized multimodal forensics datasets.

The parent DDL initiative is motivated by interpretability. The DDL paper argues that binary classification alone is insufficient in critical domains such as law, and that practical interpretability requires localization evidence (Miao et al., 29 Jun 2025). For DDL-AV, that interpretability is temporal rather than pixel-wise: the benchmark asks not only whether a clip is manipulated, but also when the manipulated content occurs in each modality.

5. Evaluation protocol and benchmark usage

The reported evaluation on DDL-AV is multi-level and localization-oriented. The ERF-BA-TFD+ paper evaluates both segment-level and full-video performance and reports Average Precision at multiple IoU thresholds—[email protected], [email protected], and [email protected]—as well as Average Recall at different proposal budgets such as AR@100, AR@50, AR@20, and AR@10 (Zhang et al., 24 Aug 2025). It also references accuracy, precision, recall, and F1 score as part of the evaluation framework.

The metrics are given in the usual forms:

$\mathrm{AP} = \int_0^1 p(r)\,dr$

and

$$\mathrm{AR}(K)=\frac{1}{N}\sum_{i=1}^N \max_{j\in S_K}\mathrm{IoU}(g_i,p_j).$$

Reported benchmark results illustrate the dataset’s difficulty. In a baseline comparison, scores after further training dropped from [email protected] = 0.9630 and [email protected] = 0.8498 on LAV-DF to [email protected] = 0.5228 and [email protected] = 0.3884 on DDL-AV, with AR@100 = 0.5200 on DDL-AV (Zhang et al., 24 Aug 2025). After integrating the UMMA attention mechanism, the same work reports DDL-AV fusion-modality scores of [email protected] = 0.9243, [email protected] = 0.8050, [email protected] = 0.0451, and AR@90 = 0.8246. On a sampled validation set, adding the ERF module increased [email protected] from 0.6472 to 0.8214, [email protected] from 0.5431 to 0.7287, AR@100 from 0.6513 to 0.7886, and AR@10 from 0.5836 to 0.7397; the competition-set overall score was 0.78.

DDL-AV was also used in the “Workshop on Deepfake Detection, Localization, and Interpretability,” Track 2: Audio-Visual Detection and Localization (DDL-AV), where ERF-BA-TFD+ won first place (Zhang et al., 24 Aug 2025). This competition context is significant because it operationalizes the dataset not merely as a static corpus but as a challenge benchmark for multimodal detection and temporal localization.

6. Research significance, difficulty profile, and limitations

The main significance of DDL-AV lies in how it combines multimodal scope with interpretable supervision. The DDL paper positions the broader DDL benchmark as a response to the limitations of prior datasets that provide mostly binary labels, restricted forgery scenarios, limited deepfake diversity, and insufficient scale for complex real-world scenarios (Miao et al., 29 Jun 2025). Within that program, DDL-AV supplies the temporal-localization component for audio-visual content.

The difficulty profile reported in downstream work is consistent and specific. The ERF-BA-TFD+ paper highlights high diversity in forgery techniques, long-duration and full-length video analysis, audio-video misalignment, sparse or subtle manipulations, and the need for segment-level localization as primary challenges when using DDL-AV (Zhang et al., 24 Aug 2025). These are not incidental benchmark nuisances; they define the dataset’s research role. DDL-AV is intended to penalize methods that overfit local artifacts, assume synchronous multimodal tampering, or collapse localization into clip-level classification.

A plausible implication is that DDL-AV is especially useful for studying robustness under modality discordance and temporal sparsity. Because the benchmark includes manipulated and non-manipulated segments within the same clip, and because audio and video can be forged independently, it creates a setting in which multimodal fusion, temporal proposal generation, and interpretable evidence extraction must all function correctly at once.

The official DDL project page is listed at: https://deepfake-workshop-ijcai2025.github.io/main/index.html (Miao et al., 29 Jun 2025).