From Understanding to Erasing: Towards Complete and Stable Video Object Removal

Abstract: Video object removal aims to eliminate target objects from videos while plausibly completing missing regions and preserving spatio-temporal consistency. Although diffusion models have recently advanced this task, it remains challenging to remove object-induced side effects (e.g., shadows, reflections, and illumination changes) without compromising overall coherence. This limitation stems from the insufficient physical and semantic understanding of the target object and its interactions with the scene. In this paper, we propose to introduce understanding into erasing from two complementary perspectives. Externally, we introduce a distillation scheme that transfers the relationships between objects and their induced effects from vision foundation models to video diffusion models. Internally, we propose a framewise context cross-attention mechanism that grounds each denoising block in informative, unmasked context surrounding the target region. External and internal guidance jointly enable our model to understand the target object, its induced effects, and the global background context, resulting in clear and coherent object removal. Extensive experiments demonstrate our state-of-the-art performance, and we establish the first real-world benchmark for video object removal to facilitate future research and community progress. Our code, data, and models are available at: https://github.com/WeChatCV/UnderEraser.

• The paper presents a novel understanding-centric framework that integrates object-induced relation distillation (OIRD) and framewise context cross-attention (FCCA) for complete and stable video object removal.
• OIRD leverages vision foundation models to accurately capture object-induced side effects such as shadows and reflections, thereby enhancing removal fidelity.
• FCCA injects localized, frame-specific context during denoising, ensuring temporal consistency and realistic background synthesis, as demonstrated by improved metrics on CAMERA-Bench.

Understanding-Centric Diffusion for Complete and Stable Video Object Removal

Introduction

Video object removal mandates eliminating target objects and seamlessly synthesizing plausible, temporally coherent backgrounds. Diffusion models have recently shown significant promise due to their powerful generative priors, but existing methods reveal a critical deficiency: insufficient semantic and physical understanding of the object's interactions with its environment, specifically failing to address object-induced side effects such as shadows, reflections, and illumination changes. The work "From Understanding to Erasing: Towards Complete and Stable Video Object Removal" (2604.01693) introduces a new understanding-centric framework that augments video diffusion models via targeted understanding guidance—both externally (object-induced relation distillation from vision foundation models) and internally (framewise context cross-attention on local background). This essay overviews the methodological advances, quantitative/qualitative results, the first real-world benchmark for the task, and implications for future video editing paradigms.

Methodology

The proposed architecture extends a DiT-based video diffusion backbone with two principal contributions: Object-Induced Relation Distillation (OIRD) and Framewise Context Cross-Attention (FCCA). These enhancements address the core limitation of prior art—the lack of side effect–aware object understanding in generative models.

Figure 1: The framework leverages OIRD from vision foundation models and FCCA with unmasked backgrounds for side-effect-aware object removal.

Object-Induced Relation Distillation

Vision foundation models (VFMs), pretrained on large-scale data, reliably localize objects and their induced side effects. The architecture explicitly distills the spatial and semantic relationships between the target object and its induced effects from VFMs (specifically DINOv2), aligning these relations within the diffusion model's latent space. The distillation is formulated via an OIRD loss, which minimizes the discrepancy in cross-region similarity (between object and side-effect regions) between the VFM and the generative backbone. This cross-modal relational transfer empowers the model to attend to and eliminate not just the masked region, but also secondary side effects that generic diffusion denoisers overlook.

Figure 2: VFM-derived attention maps more accurately capture object regions and side effects (e.g., shadows) compared to standard diffusion models.

Framewise Context Cross-Attention

Standard cross-attention mechanisms in video diffusion models are not designed to inject frame-specific visual cues, resulting in suboptimal exploitation of local background context. FCCA addresses this gap by encoding visual tokens from the unmasked regions (using a CLIP vision encoder) and injecting them at the frame level via a simple reshaping and batching strategy. This provides each denoising step with precise, contextually relevant information, thereby enhancing inpainting fidelity and spatio-temporal consistency.

Keyframe-Guided Propagation

Long video sequences compound temporal inconsistency due to memory and distributional mismatch (training on short clips, inference on longer ones). The Keyframe-Guided Propagation (KGP) module introduces anchor frames (keyframes) generated with ground truth guidance and aligns the subsequent clipwise processing to these anchors, alleviating color artifacts and flicker observed in naïve long-sequence inference.

Figure 3: The keyframe-guided propagation mechanism ensures temporal consistency in long video object removal.

Experimental Results

Datasets and Metrics

A new real-world benchmark (CAMERA-Bench) comprising 40 paired videos with precise ground truth was established, complementing ROSE-Bench (synthetic) and Scene-Bench (web videos with SAM2-generated masks). Metrics include PSNR, SSIM, and LPIPS against ground truth, as well as GPT-4o–based video evaluation for unpaired settings.

Figure 4: Samples from the CAMERA-Bench real-world dataset, highlighting diverse side effects (e.g., reflections, shadows).

Quantitative Evaluation

The proposed model achieves superior PSNR/SSIM and lower LPIPS on all benchmarks, surpassing diffusion and transformer baselines. For instance, on CAMERA-Bench, the method scores 28.09 PSNR, 0.9448 SSIM, and 0.0726 LPIPS, exceeding ROSE and MiniMax-Remover. Human evaluation corroborates the numerical improvements, with top average human rankings for removal completeness, background consistency, and overall quality.

Qualitative Evaluation

Visual comparisons show the model cleanly removes objects and all associated effects (e.g., shadows, reflections), unlike DiffuEraser or ROSE, which often leave artifacts or miss physical side effects. Backgrounds are both spatially plausible and temporally stable. In complex real-world and Internet videos, the approach generalizes effectively.

Figure 5: Comparison of competing methods: only the proposed approach removes both the target object and reflection/shadow side effects fully.

Ablation Studies

OIRD, when introduced independently, yields substantial improvements in reconstruction quality. Adding FCCA on top of OIRD further enhances all metrics, demonstrating the complementarity of semantic/physical relation guidance and precise local conditioning.

Figure 6: Visualization of ablation: OIRD (left) boosts side-effect removal; FCCA (right) improves background completion and temporal coherence.

Analysis of distillation depth reveals optimal alignment when OIRD is applied at intermediate layers (block 15) of the backbone. Additionally, adjusting the OIRD regularization weight $\lambda$ empirically validates 0.1 as the optimal trade-off for stable convergence.

Implications and Future Developments

This framework significantly advances video object editing by incorporating explicit physical/semantic understanding via foundation models, a direction immediately extensible to generic object interaction editing, video compositing, and multimodal inpainting. The separation of external relational distillation and internal visual context conditioning sets a modular blueprint for foundation model–driven editing tasks.

The CAMERA-Bench dataset provides, for the first time, realistic benchmarks featuring natural side effects, catalyzing community progress and fair comparison.

Practical applications include surveillance video redaction, cinema postproduction, AR, and VR content manipulation, where side-effect-aware editing is mandatory. The approach’s modularity also means future integration with more advanced foundation models and cross-modal supervision (e.g., physics simulators) is straightforward. Potential developments include generic physical-consistency regularization, extension to open-domain object editing, and tighter integration with large multimodal models for instruction-driven, robust video manipulation.

Conclusion

The paper presents a robust approach for complete and coherent video object removal by complementing generative diffusion models with explicit object–side-effect relational understanding and contextual background conditioning. By bridging physical/semantic perception from foundation models to generative editing and providing infrastructure for standardized benchmarking, this work marks a significant advance toward controllable, side-effect-aware spatiotemporal video editing. Future advances in foundation model capabilities and compositional reasoning will likely further strengthen such pipelines for broader, more challenging video editing tasks.