ImVideoEdit: Image-learning Video Editing via 2D Spatial Difference Attention Blocks

Abstract: Current video editing models often rely on expensive paired video data, which limits their practical scalability. In essence, most video editing tasks can be formulated as a decoupled spatiotemporal process, where the temporal dynamics of the pretrained model are preserved while spatial content is selectively and precisely modified. Based on this insight, we propose ImVideoEdit, an efficient framework that learns video editing capabilities entirely from image pairs. By freezing the pre-trained 3D attention modules and treating images as single-frame videos, we decouple the 2D spatial learning process to help preserve the original temporal dynamics. The core of our approach is a Predict-Update Spatial Difference Attention module that progressively extracts and injects spatial differences. Rather than relying on rigid external masks, we incorporate a Text-Guided Dynamic Semantic Gating mechanism for adaptive and implicit text-driven modifications. Despite training on only 13K image pairs for 5 epochs with exceptionally low computational overhead, ImVideoEdit achieves editing fidelity and temporal consistency comparable to larger models trained on extensive video datasets.

• The paper introduces a novel image-based training method that decouples spatial and temporal editing using Predict-Update Spatial Difference Attention.
• It leverages pre-trained 3D Diffusion Transformers and text-guided dynamic gating to produce semantically precise and consistent video edits without paired video data.
• Quantitative evaluations show that ImVideoEdit outperforms video-trained baselines in instruction adherence and temporal stability while using significantly less computational resources.

ImVideoEdit: Video Editing via Image-Based Spatial Difference Attention

Motivation and Problem Formulation

ImVideoEdit introduces an efficient, video-free paradigm for generative video editing by reframing the task as a decoupled spatiotemporal process. The method leverages only image pairs for training, treating them as single-frame videos to learn precise spatial feature transformations, while the temporal dynamics from pre-trained 3D Diffusion Transformers (DiTs) remain intact. This design circumvents the prohibitive data acquisition and computational cost associated with paired video datasets, a major bottleneck in existing approaches.

Current video editing pipelines suffer from instability when fine-tuned directly on highly coupled spatiotemporal attention modules, frequently resulting in background drift and temporal flickering. These issues are often mitigated by using external masks or segmentation models, but such solutions limit zero-shot text-driven interaction and fail with complex non-rigid deformations. ImVideoEdit sidesteps these constraints through adaptive, implicit text-guided mechanisms and spatial difference modeling.

Dataset Construction and Supervision

The data pipeline synthesizes high-quality paired images via scene-conditioned prompt construction, semantic validation (Gemini 3.1 Pro), and automated/human filtering. Each sample consists of a source and edited image, both generated using text-to-image models (Qwen-Image, Qwen-Image-Edit) with editing instructions and comprehensive visual descriptions. This yields a dataset of approximately 13K pairs, densely covering task categories such as style transfer, object addition/removal, color transformation, and background replacement.

Figure 1: Dataset pipeline stages are illustrated, encompassing prompt construction, paired synthesis, and filtering for robust supervision.

Figure 2: Editing task distribution in the dataset, supporting broad spatial manipulation scenarios.

Architecture: Predict-Update Spatial Difference Attention

The ImVideoEdit architecture is built upon a frozen Wan-T2V-1.3B backbone, preserving strong pre-trained spatiotemporal priors. Image pairs are formatted as single-frame videos; spatial feature reorganization is performed via the Predict-Update Spatial Difference Attention module, which operates parallel to each attention block in the 3D DiT.

Predict Phase: The 2D spatial difference operator extracts an initial coarse spatial alignment between source and target representations. This is fused with the original spatiotemporal attention output through a linear projection initialized at zero.

Update Phase: The spatial conflict is estimated by subtracting the initial 2D observation from the predictive state, isolating high-frequency spatial residuals. This offset is refined via a second interaction block, yielding the structural difference for spatial edits.

Text-Guided Dynamic Semantic Gating: To ensure prompt-aware modification rather than indiscriminate spatial recalibration, text embeddings modulate the spatial residuals through cross-attention and an MLP-projected gating matrix. This gating modulates update intensity, enabling localized, instruction-faithful edits.

Figure 3: Architectural overview of ImVideoEdit, highlighting the Predict-Update module with spatiotemporal preservation and text-driven gating.

Training and Evaluation Protocol

ImVideoEdit is trained for 5 epochs on only image pairs, leveraging a flow-matching objective for conditional editing. The training is memory-efficient, with VRAM consumption around 20 GB per GPU—orders of magnitude lower than video-based pipelines.

Out-of-the-box zero-shot generalization is achieved for diverse video editing tasks, with no explicit temporal supervision. The evaluation protocol uses VLM-based metrics (Gemini 3.1 Pro) and the VBench suite, capturing instruction adherence, temporal consistency, visual fidelity, artifact absence, subject/background consistency, and motion smoothness.

Quantitative and Qualitative Performance

ImVideoEdit demonstrates strong numerical results, outperforming VACE-1.3B and aligning closely with Kiwi-Edit (which uses full video data) on both VLM and VBench metrics. On VLM, ImVideoEdit achieves a Total Score of 65.24 (vs. 56.79 for VACE-1.3B). ImVideoEdit consistently excels in instruction adherence and temporal stability, even in challenging scenarios demanding fine localization or non-rigid object replacement.

Qualitative comparisons reveal that baseline models frequently fail in prompt alignment or introduce undesirable global shifts and structural inconsistencies. ImVideoEdit produces precise, semantically complete edits that strictly follow instructions, fully replacing backgrounds and modifying color/texture as required, while maintaining coherence and realism.

Figure 4: Examples of basic editing tasks handled by ImVideoEdit, illustrating localized object manipulation, background replacement, and style transfer.

Figure 5: Qualitative results highlight ImVideoEdit’s precision and temporal fidelity relative to baseline models.

Ablation and Robustness Analysis

The ablation studies validate the necessity of all core components. Removal of text gating or the update module significantly degrades performance, especially in instruction adherence and spatial completeness. Most notably, the Predict-Update design outperforms naive parallel 2D extraction strategies by a substantial margin, as shown in both VLM-based and qualitative evaluation.

Figure 6: Qualitative ablation results demonstrate degradation when omitting architectural elements or text-driven gating.

Implications, Limitations, and Future Directions

The principal implication is that video editing can be effectively learned from image data, challenging the assumption that extensive paired video sequences are required for instruction-faithful editing with temporal consistency. This paradigm unlocks scalable training and accessibility, drastically reducing computational resources and accelerating model development for open-domain video editing.

ImVideoEdit’s decoupled spatiotemporal modeling offers a transferable foundation for diverse edit types, but inherently relies on the robustness of underlying temporal priors from pre-trained backbones. Potential limitations may emerge in scenes involving rare or extreme motion, where spatial alignment learned from images is insufficient to generalize across frames.

Directions for future research include:

• Extending this paradigm to multi-frame image supervision for more intricate temporal edits.
• Generalizing Predict-Update mechanisms to other modalities (audio, volumetric data).
• Integrating broader text-image datasets and expanding semantic gating for compositional and interactive editing.
• Exploring lightweight fine-tuning for targeted domains (e.g., medical videos, scientific visualization).

Figure 7: Dataset visualization samples, confirming the diversity and quality of training supervision.

Figure 8: Extended qualitative results, exemplifying ImVideoEdit’s generalization across editing subtasks.

Conclusion

ImVideoEdit establishes a high-efficiency, video-free approach to instruction-guided video editing, driven by spatial difference attention and text-guided gating. The method achieves top-tier editing fidelity and temporal consistency, outperforming several video-trained baselines, with minimal data requirements and computational overhead. This work validates the transferability of image-based spatial supervision to complex video editing tasks and sets the stage for future developments in scalable generative frameworks for video synthesis and manipulation.