EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing

Abstract: Video object removal aims to eliminate dynamic target objects and their visual effects, such as deformation, shadows, and reflections, while restoring seamless backgrounds. Recent diffusion-based video inpainting and object removal methods can remove the objects but often struggle to erase these effects and to synthesize coherent backgrounds. Beyond method limitations, progress is further hampered by the lack of a comprehensive dataset that systematically captures common object effects across varied environments for training and evaluation. To address this, we introduce VOR (Video Object Removal), a large-scale dataset that provides diverse paired videos, each consisting of one video where the target object is present with its effects and a counterpart where the object and effects are absent, with corresponding object masks. VOR contains 60K high-quality video pairs from captured and synthetic sources, covers five effects types, and spans a wide range of object categories as well as complex, dynamic multi-object scenes. Building on VOR, we propose EffectErase, an effect-aware video object removal method that treats video object insertion as the inverse auxiliary task within a reciprocal learning scheme. The model includes task-aware region guidance that focuses learning on affected areas and enables flexible task switching. Then, an insertion-removal consistency objective that encourages complementary behaviors and shared localization of effect regions and structural cues. Trained on VOR, EffectErase achieves superior performance in extensive experiments, delivering high-quality video object effect erasing across diverse scenarios.

• The paper introduces a reciprocal learning framework, EffectErase, alongside the comprehensive VOR dataset for effect-aware video object editing.
• It employs task-aware region guidance and effect consistency loss to achieve superior spatiotemporal accuracy and state-of-the-art quantitative metrics.
• The framework robustly handles both object removal and insertion tasks, demonstrating impressive generalization on synthetic and real-world benchmarks.

EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing

Introduction

The problem of video object removal encompasses not only the elimination of foreground objects from dynamic video sequences, but also the seamless erasure of their visual side effects, including occlusion, shadows, lighting perturbations, reflections, and deformations. Traditional inpainting and recent diffusion-based techniques have improved object removal, but their reliance on input masks and lack of explicit modeling for object-induced environmental alterations limits their robustness, particularly in real-world scenarios featuring intricate visual effects.

This work introduces two primary contributions: the VOR dataset, a large-scale, effect-focused, hybrid video dataset, and EffectErase, a reciprocal learning framework addressing both removal and insertion. The methodology incorporates new cross-modal conditioning approaches and maximizes spatiotemporal congruence in object–effect localization, outperforming prior art under both synthetic and real-world conditions.

Figure 1: Limitations of existing video object removal methods. Existing approaches often fail to remove object-induced effects such as reflections.

VOR Dataset: A Comprehensive Benchmark for Effect-Aware Removal

The VOR dataset is designed to overcome previous limitations regarding realism, scale, and effect coverage. It is constructed as a hybrid corpus blending camera-captured and 3D-rendered videos. Real-world data collection utilizes custom capture software to produce tightly aligned object-present/absent pairs, addressing issues of exposure, duration, and motion registration.

Figure 2: Pipeline for VOR dataset construction, combining synthetic Blender-generated scenes with real captures, Ken Burns-based camera simulation, and mask annotation.

The dataset structure covers five effect categories critical to realistic video removal:

• Occlusion (opaque, semi-transparent, transparent)
• Shadow
• Lighting
• Reflection
• Deformation

These are represented across 60,000 triplet video pairs, with 366 object classes and 443 scenes, spanning indoor and outdoor domains and supporting diverse dynamics in object and camera motion. Masks are generated and verified using a combination of SAM2-assisted propagation and manual annotation.

Figure 3: Representative side effects (occlusion, shadow, lighting, reflection, deformation) as captured in VOR.

Additionally, VOR provides two core benchmarks for robust evaluation: VOR-Eval (curated with GT) and VOR-Wild (in-the-wild, no GT).

EffectErase: Reciprocal Learning for Video Object Removal and Insertion

EffectErase models video object removal and insertion as inverse, complementary tasks sharing a common structural backbone. The core architecture extends a VAE-based, DiT-driven denoising model with three salient design components:

1. Removal–Insertion Joint Learning: Both tasks use paired latent encodings, addressing the same affected regions and enabling cross-domain supervision to stabilize region correspondence.
2. Task-Aware Region Guidance (TARG): This module leverages a CLIP-based image-text pipeline and explicit task tokens to encode spatiotemporal correlations, facilitating precise effect-related region prediction and dynamic task switching.
3. Effect Consistency Loss (EC): Attention maps from both tasks are projected via a learned mapper and aligned using a KL-divergence-based loss against a soft difference prior, enforcing consistent focus on effect regions.

   Figure 4: Removal–Insertion tasks are inverse operations, defined over identical video regions.

   

   Figure 5: The EffectErase joint learning framework integrates removal/insertion pair encoding via VAE, fuses features through an adaptor, and supervises maximal attention to reinforce effect region consistency.

During training, the model is optimized for denoising on both removal and insertion tasks, regularized by the effect consistency objective.

Experimental Results

Quantitative Comparison

EffectErase was evaluated on ROSE and VOR benchmarks. It demonstrates state-of-the-art performance across all primary metrics:

• VOR-Eval: PSNR 23.75, SSIM 0.806, LPIPS 0.170, FVD 342.87
• VOR-Wild: QScore 9.28, User Score 7.20

EffectErase shows marked improvement in both temporal consistency (FVD) and perceptual quality (LPIPS) compared to prior SOTA methods, confirming its efficacy in effect-aware object removal.

Qualitative Analysis

Figure 6: Qualitative results on VOR-Eval. EffectErase achieves artifact-free removal of objects and effects, outperforming inpainting and prior removal approaches.

Figure 7: Qualitative results on VOR-Wild demonstrate robust generalization to in-the-wild effects, including complex occlusions, lighting changes, and reflections.

Ablation studies confirm the additive contribution of each core design: EC loss, TARG, and hybrid data training.

Reciprocal Insertion Task

EffectErase is natively extensible to the video object insertion task with no additional finetuning, accurately synthesizing context-sensitive effects—such as shadows and reflections—for inserted objects, maintaining scene integrity.

Figure 8: Video object insertion by EffectErase yields contextually consistent, high-quality composite scenes with realistic effects.

Analysis and Dataset Insights

• The VOR dataset encompasses a broader distribution of scene categories than previous resources, which is visualized using a custom taxonomy.

  Figure 9: VOR scene taxonomy covers 67 indoor and outdoor types.

• Capture software and annotation pipelines are engineered for reliability and quality, ensuring minimal spatiotemporal bias.

  Figure 10: Custom app ensures highly aligned video pairs by automating exposure, matching, and triggering.

Implications and Future Work

The EffectErase framework presents a generalizable reciprocal learning paradigm for bidirectional effect-aware video editing. The integration of cross-modal guidance and effect-region consistency is broadly applicable to tasks requiring precise control over object–environment interactions—future work may involve mask-free, multimodal interaction (e.g., text-guided target specification).

The VOR dataset establishes a new benchmark standard, expected to drive further advances in effect-centric video object manipulation.

Conclusion

EffectErase and the VOR dataset jointly advance the methodological and empirical frontier for effect-aware video object removal and insertion. The reciprocal task design, region-aware guidance, and soft consistency regularization enable superior temporal, structural, and perceptual quality across diverse scenarios. Open challenges remain in user-interaction modalities and effect disambiguation, but this framework provides a robust foundation for expanded research and practical deployment in post-production, editing, and video understanding systems.