Editto: Instruction-Based Video Editing Framework

• Editto is an instruction-based video editing framework that uses advanced diffusion models and curriculum learning to achieve precise, scalable, and photorealistic edits.
• It leverages a pretrained video generator (VACE) with a dedicated temporal enhancer and a massive synthetic dataset (Ditto-1M) to support both global and local video manipulation.
• The framework employs flow-matching loss and curriculum training on 1M curated triplets, achieving state-of-the-art fidelity, temporal coherence, and instruction adherence.

Editto is an instruction-based video editing framework designed to enable precise, scalable, and photorealistic video edits from natural language instructions. It combines a distilled diffusion model adapted from the VACE in-context video generator with a curriculum-based training strategy, a large-scale synthetic dataset (Ditto-1M), and a temporal enhancement module, establishing a new state-of-the-art for text-driven video manipulation in terms of fidelity, temporal coherence, and instruction-following ability (Bai et al., 17 Oct 2025).

1. Architectural Foundations

Editto builds directly upon a pretrained in-context video generator (VACE), adapted to instruction-based editing by integrating hierarchical multimodal conditioning and temporal enhancement modules. The architecture consists of three primary components:

• Context Branch (Frozen): This branch encodes the source video $V_s$ and (optionally) a reference key-frame $f_k'$, using a stack of temporal-spatial attention blocks. Each block uses

$$\mathrm{Attn}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$

with $Q$, $K$, $V$ computed from projected input features with learnable matrices $W_q$, $W_k$, $W_v$.

• Main Branch (DiT Transformer Backbone): This branch processes text embeddings of the instruction $p$ alongside outputs of the Context Branch via alternating self- and cross-attention layers. Cross-attention fuses text/visual context by

$f_k'$0

where $f_k'$1 is the concatenated context vector.

• Temporal Enhancer (Lightweight MoE Denoiser): Derived from the Wan2.2 MoE denoiser, this refinements step injects Gaussian noise $f_k'$2 into the preliminary output video $f_k'$3 and denoises via a specialized expert:

$f_k'$4

where $f_k'$5 is an MoE expert, improving temporal and local detail.

The pipeline is highly efficient: all modules are post-training quantized (8-bit) and distilled (per Yin et al., CausVid), enabling batch inference at approximately 20% of the original cost. During fine-tuning, only the context block's linear projections (≈5% of the parameters) are updated, permitting rapid adaptation to instruction-based tasks.

2. Dataset Construction and Curriculum Training

Ditto-1M is a synthetic dataset comprised of 1,000,000 triplets $f_k'$6, constructed from approximately 200,000 curated high-resolution videos sourced from Pexels. The processing pipeline is:

1. Deduplication: DINOv2 cosine similarity threshold removes near-duplicates.
2. Motion Filtering: CoTracker3 ensures sufficient motion.
3. Instruction Generation: Video-LLMs (VLMs) generate a dense caption $f_k'$7 for $f_k'$8, then produce the instruction $f_k'$9 using both $$\mathrm{Attn}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$0 and $$\mathrm{Attn}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$1.
4. Key-frame and Depth Augmentation: Qwen-Image samples a reference frame $$\mathrm{Attn}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$2, VideoDepthAny produces a depth sequence $$\mathrm{Attn}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$3.
5. Edit Synthesis: A distilled VACE model synthesizes the target video $$\mathrm{Attn}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$4 conditioned on $$\mathrm{Attn}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$5, $$\mathrm{Attn}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$6, and $$\mathrm{Attn}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$7.
6. Automated Filtering: A VLM-based agent filters outputs for instruction fidelity, motion consistency, visual quality, and safety.

Of the 1M triplets, approximately 700k are global (style/color) edits and 300k are local (object-level) edits. Each sequence comprises 101 frames at resolution $$\mathrm{Attn}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$8 and 20 FPS.

Curriculum Learning: A modality curriculum linearly anneals the probability $$\mathrm{Attn}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$9 of providing the reference key-frame $Q$0 over 5,000 steps:

$Q$1

Initially both video and reference frame are provided; after $Q$2, only $Q$3 and $Q$4 are available.

Training: The model is trained for 16,000 steps (5,000 curriculum, 11,000 pure text), using AdamW with $Q$5, $Q$6, and weight decay $Q$7, a constant learning rate $Q$8, on 64 NVIDIA H-series GPUs, totaling approximately 12,000 GPU-days.

3. Loss Formulation and Optimization

Training employs the flow-matching loss for latent video diffusion:

$Q$9

where $K$0 is the clean target video, $K$1 is the noisy version at time $K$2, and $K$3 is text/visual context.

Notably, no adversarial or auxiliary perceptual losses are included. Weight decay is applied as a stability regularizer exclusively to the learnable context block projections.

4. Evaluation Methodologies and Results

Performance is quantitatively assessed using both automatic metrics and human evaluations:

• CLIP-T: Text-video similarity

$K$4

• CLIP-F: Inter-frame consistency

$K$5

• VLM-score: Holistic judgment by a VLM (1–10 scale).
• Human evaluation: 1–4 scale for Edit-Accuracy (Edit-Acc), Temporal Consistency (Temp-Con), Overall.

Method	CLIP-T ↑	CLIP-F ↑	VLM-score ↑	Edit-Acc ↑	Temp-Con ↑	Overall ↑
TokenFlow	23.63	98.43	7.10	1.70	1.97	1.70
InsViE	23.56	98.78	7.35	2.28	2.30	2.36
Editto	25.54	99.03	8.10	3.85	3.76	3.86

Qualitative samples (as shown in Figure 1 of the source) display marked improvements in both global stylization (e.g., color grading) and local edits (e.g., precise alterations such as “paint the suit black”), without introducing identity drift. Sim-to-real transfer experiments indicate the dataset’s photorealism by enabling the model to invert stylized sequences back to their original form.

Ablation studies demonstrate the impact of scale (raising training size from 100k to 1M improves CLIP-T by ≈1.5 points) and curriculum (removing reference-drop annealing reduces instruction adherence by ≈2 CLIP-T).

5. Strengths, Limitations, and Prospective Research

Strengths:

• Fully scalable data synthesis and training regime, achieving 1M video edits and balancing fidelity, efficiency, and diversity via distillation and quantization.
• Consistent state-of-the-art performance across both automatic and human evaluation benchmarks.
• A curriculum learning strategy enabling evolution from visual-text conditioning to pure-text.

Limitations:

• Quality remains constrained by the original VACE model’s motion priors, with a ceiling on achievable realism.
• Generation is limited to static, 101-frame, 720p video; extending to very long or ultra-high-resolution sequences is not directly supported.
• Quantization and distillation, while computationally advantageous, introduce a minor fidelity degradation compared to full-precision, multi-step inference.

Future Work:

• Design and implementation of temporal attention modules tailored for editing tasks (e.g., cross-clip reference attention).
• Support for multi-turn, interactive editing workflows, where edit sequences iteratively refine outputs.
• Expansion to other modalities, such as audio-driven edit instructions or editing of 3D scenes.
• Development of advanced, nuanced safety filters to further improve ethical compliance.

In summary, Editto demonstrates that large-scale, high-quality synthetic datasets, combined with efficient diffusion modeling and curriculum-driven training, enable substantial advances in instruction-based video editing, narrowing the gap between image and video manipulation (Bai et al., 17 Oct 2025).