Ditto-1M: Synthetic Video Editing Dataset

• Ditto-1M is a large-scale synthetic dataset comprising one million triplets of source video, natural language instruction, and edited output for precise video modifications.
• It employs a robust automated pipeline that integrates vision-language models, advanced filtering mechanisms, and multi-stage curation to ensure high fidelity and scalability.
• Ditto-1M underpins the Editto model, which achieves state-of-the-art performance across automatic and human evaluations in instruction-based video editing.

Ditto-1M is a large-scale, high-fidelity synthetic dataset expressly constructed to advance instruction-based video editing by providing one million video editing triplets. Each triplet in Ditto-1M consists of a source video $V_s$, a natural language editing instruction $p$, and the corresponding edited output video $V_e$. The dataset is engineered to support fine-grained supervised training of models that learn to perform diverse and precise video edits specified by human-readable instructions. Ditto-1M is the result of a highly automated, scalable framework that synthesizes, filters, and curates high-quality video editing data at scale, addressing the prior scarcity of such resources. Developed by leveraging an integrated suite of vision-LLMs, state-of-the-art image and video editing architectures, and rigorous data curation mechanisms, Ditto-1M underpins the development and evaluation of the Editto model, which demonstrates state-of-the-art performance in instruction-based video editing (Bai et al., 17 Oct 2025).

1. Dataset Structure and Statistics

Ditto-1M comprises 1,000,000 synthetic video editing triplets $(V_s, p, V_e)$. Each source and edited video is a 101-frame clip sampled at 20 FPS (approximately 5 seconds in duration) with a resolution of 1280 × 720 pixels (HD). Source material is drawn from over 200,000 unique, professional-grade clips licensed from Pexels. Deduplication is performed using DINOv2 feature embeddings and pairwise similarity thresholding to eliminate near duplicates, while CoTracker3-driven motion-scale filtering removes static or low-motion content, yielding a final filtered pool of approximately 200,000 unique clips.

Editing instructions are generated via a two-step Vision-LLM (VLM) process using Qwen-2.5-VL. First, each video is captioned; second, a diverse natural language instruction is synthesized using the source video and its caption. Instructions fall broadly into two categories: approximately 700,000 global edits (style transformations, lighting or environment changes) and 300,000 local edits (object addition/removal, color changes, texture modifications). Lexical diversity in the instruction set is high, as evidenced by a word cloud showing hundreds of distinct action verbs such as “make vintage,” “add fireworks,” and “replace car.”

2. Data Generation Pipeline

The Ditto-1M pipeline comprises three major phases: pre-processing, core synthesis, and post-processing.

2.1 Pre-processing

Input videos are deduplicated (via DINOv2 embedding similarity) and filtered by motion characteristics (mean per-frame displacement from CoTracker3 trajectories). The surviving clips are then resized, and frame rates standardized to the 1280×720, 20 FPS format.

2.2 Core Synthesis

The core synthesis phase operates as follows (see Figure 1 and provided pseudocode):

• For each filtered video $V_s$:
  1. Generate a caption $c = \mathrm{VLM}(V_s, p_{\mathrm{caption}})$.
  2. Produce an editing instruction $p = \mathrm{VLM}(V_s, c, p_{\mathrm{instruct}})$.
  3. Sample a key frame $f_k$ from $V_s$.
  4. Apply a state-of-the-art instructed image editor (Qwen-Image) to obtain the appearance-modified anchor frame $f_k' = \mathcal{E}_{\mathrm{img}}(f_k, p)$.
  5. Predict a dense depth video $p$0 for $p$1 using VideoDepthAnything.
  6. Synthesize the edited output video $p$2 using VACE, an in-context video generator with multi-head attention fusion across depth, edited keyframe, and textual instruction.

Efficiency improvements are achieved through distillation (from CausVid to VACE) and 8-bit quantization, which collectively reduce compute to approximately 20% of the original VACE cost, allowing full dataset synthesis in approximately 12,000 GPU-days.

2.3 Post-processing and Quality Control

VLM-based curation automatically rejects outputs failing edit fidelity (correspondence to $p$3), source fidelity (preservation of original motion/semantics), visual appeal (absence of artifacts), or safety constraints (no inappropriate content). A two-stage denoising process based on Wan2.2 introduces and then removes slight Gaussian noise, further enhancing output while avoiding semantic drift.

3. Instruction Generation and Spectrum

Editing instructions are algorithmically generated to maximize diversity and coverage. Qwen-2.5-VL first produces a semantic caption and then, conditioned on both the source video and its caption, generates a unique instruction. This two-step process ensures both contextual relevance and compositional diversity in synthesized edits. Instructions span canonical global transformations—style, illumination, background—and more intricate local manipulations, such as detailed object insertions/removals, fine color/texture operations, and compositional scene edits. Empirical analysis (word cloud) confirms lexical and semantic breadth across the entire corpus.

4. Model Training: Curriculum and Architecture

The Editto model, trained exclusively on Ditto-1M, utilizes a Modality Curriculum Learning (MCL) regime. Conditioning inputs consist of a visual reference ($p$4) and/or text instruction ($p$5). The curriculum schedule $p$6 (probability of providing the reference frame at step $p$7) controls reliance on visual guidance:

• For warm-up steps ($p$8), $p$9
• Annealed to zero over $V_e$0 steps:

$V_e$1

• After $V_e$2, only the text instruction is fed

Training proceeds with a batch size of 64 videos, AdamW optimizer (β₁=0.9, β₂=0.999, weight decay=0.01), a constant learning rate of $V_e$3, and total of 16,000 steps (first 5,000 for warm-up). Model weights are fine-tuned only for the linear projections in context blocks; all other parameters remain frozen.

The architecture is based on a pre-trained VACE backbone:

• Context Branch: frozen, except for linear projections; extracts spatiotemporal structure from $V_e$4 and optionally $V_e$5
• Main Branch: DiT-style U-Net decoder for denoising latent representations based on joint text and context conditioning

The flow-matching loss is used as the training objective:

$V_e$6

where $V_e$7 is the clean edited latent, $V_e$8 is the noised latent at time $V_e$9, $(V_s, p, V_e)$0 is the conditioning vector, and $(V_s, p, V_e)$1 is the denoising vector field.

5. Evaluation Methodology and Empirical Results

Evaluation encompasses both automatic and human-centric metrics. Automatic assessment includes:

• CLIP-T: cosine similarity between CLIP text and video embeddings (measuring adherence to instructions; higher is better).
• CLIP-F: mean inter-frame CLIP similarity (quantifying temporal coherence; higher is better).
• VLM score: an aggregate Vision-LLM judge that assesses accuracy, fidelity, and aesthetics.

Human evaluation is conducted via 1,000 pairwise comparisons on Edit-Acc (edit accuracy), Temp-Con (temporal consistency), and Overall impression.

Method	CLIP-T↑	CLIP-F↑	VLM↑	Edit-Acc↑	Temp-Con↑	Overall↑
TokenFlow	23.63	98.43	7.10	1.70	1.97	1.70
InsV2V	22.49	97.99	6.55	2.17	1.96	2.07
InsViE	23.56	98.78	7.35	2.28	2.30	2.36
Editto (ours)	25.54	99.03	8.10	3.85	3.76	3.86

The results indicate Editto, trained on Ditto-1M, substantially outperforms prior models on all metrics, achieving visually sharper, temporally stable, and instruction-compliant outputs. User studies strongly prefer Editto across all axes. The Sim2Real experiment demonstrates that Editto trained on Ditto-1M can invert stylized clips back to photorealistic content, supporting the dataset’s fidelity and Editto's generalization capabilities.

Ablation studies reveal that scaling from 100k to 1M samples yields steady improvements. Removing curriculum learning degrades CLIP-T by more than 2 points and lowers Edit-Acc by approximately 20%.

6. Significance, Limitations, and Implications

Ditto-1M provides an unprecedented resource for instruction-based video editing at scale, overcoming prior data bottlenecks by combining scalable synthesis, rigorous curation, and broad semantic coverage. The underlying approach—leveraging modular VLMs, automated filtering, and efficient generative pipelines—demonstrates a principled path for constructing large supervised benchmarks in rich visual editing domains.

A plausible implication is that similar pipelines could scale to other generative tasks (e.g., multimodal manipulation, long-form video storytelling) if matched by corresponding VLM advances and sufficiently diverse source corpora. However, the dataset’s exclusive reliance on synthetic edits, rather than user-provided edits, may impose domain shift constraints for certain downstream applications. Further, the significant computational investment (≈12,000 GPU-days) highlights ongoing efficiency challenges intrinsic to scaling high-fidelity generative datasets. Overall, Ditto-1M establishes both a practical benchmark and a methodological template for future research in large-scale, instruction-driven video transformation (Bai et al., 17 Oct 2025).