ROSE: Remove Objects with Side Effects in Videos

Abstract: Video object removal has achieved advanced performance due to the recent success of video generative models. However, when addressing the side effects of objects, e.g., their shadows and reflections, existing works struggle to eliminate these effects for the scarcity of paired video data as supervision. This paper presents ROSE, termed Remove Objects with Side Effects, a framework that systematically studies the object's effects on environment, which can be categorized into five common cases: shadows, reflections, light, translucency and mirror. Given the challenges of curating paired videos exhibiting the aforementioned effects, we leverage a 3D rendering engine for synthetic data generation. We carefully construct a fully-automatic pipeline for data preparation, which simulates a large-scale paired dataset with diverse scenes, objects, shooting angles, and camera trajectories. ROSE is implemented as an video inpainting model built on diffusion transformer. To localize all object-correlated areas, the entire video is fed into the model for reference-based erasing. Moreover, additional supervision is introduced to explicitly predict the areas affected by side effects, which can be revealed through the differential mask between the paired videos. To fully investigate the model performance on various side effect removal, we presents a new benchmark, dubbed ROSE-Bench, incorporating both common scenarios and the five special side effects for comprehensive evaluation. Experimental results demonstrate that ROSE achieves superior performance compared to existing video object erasing models and generalizes well to real-world video scenarios. The project page is https://rose2025-inpaint.github.io/.

• The paper introduces a novel diffusion transformer model that uses a reference-based input to effectively remove objects and their side effects from videos.
• It presents an automatic 3D-rendered synthetic dataset with 16,678 video pairs and six distinct side-effect categories to enable robust training and evaluation.
• Extensive experiments demonstrate state-of-the-art performance with enhanced PSNR, SSIM, and LPIPS metrics, validating improved realism in video inpainting.

ROSE: Remove Objects with Side Effects in Videos

Introduction and Motivation

The ROSE framework addresses the video object removal problem with a focus on eliminating not only the target objects but also their correlated side effects—shadows, reflections, lighting changes, mirror images, and translucency. Prior video inpainting methods, including flow-based and diffusion-based approaches, have achieved notable performance in object removal but typically fail to handle these side effects due to the lack of physically accurate paired training data. ROSE introduces a systematic approach to dataset construction and model design, enabling robust removal of objects and their environmental interactions.

Figure 1: Video object removal results generated by ROSE, demonstrating the elimination of objects and their side effects across diverse scenarios.

Synthetic Dataset Construction via 3D Rendering

A key contribution of ROSE is the fully-automatic pipeline for generating paired video data using Unreal Engine. This pipeline samples diverse 3D scenes and objects, generates multi-view videos with precise object masks, filters valid views to avoid occlusions, and renders aligned video pairs with and without the target object. The resulting dataset comprises 16,678 synthetic video pairs, each 90 frames at 1920×1080 resolution, covering urban, rural, and natural scenes with dynamic lighting and object interactions.

Figure 2: Paired video preparation pipeline using 3D data, including scene/object sampling, multi-view generation, view filtering, and rendering.

The dataset is explicitly categorized into six side-effect classes: common (minimal interaction), light source, mirror, reflection, shadow, and translucent. This categorization enables targeted evaluation and training for complex object-environment relationships.

Figure 3: Illustration of the various side-effect categories studied in the ROSE dataset.

Model Architecture and Methodology

ROSE is implemented as a diffusion transformer-based video inpainting model, extending the Wan2.1 backbone. The model input consists of the full original video and object masks, diverging from the conventional "mask-and-inpaint" paradigm. This reference-based approach allows the model to leverage the entire video context, facilitating accurate localization and removal of side effects.

Figure 4: The ROSE framework, concatenating noisy latents, input video, and masks, with an auxiliary difference mask predictor for side effect localization.

Mask Augmentation

To enhance robustness to real-world mask imperfections, ROSE employs diverse mask augmentation strategies during training: original (precise), point-wise (sparse), bounding box (coarse), dilated, and eroded masks. This exposure to varied mask types improves generalization to user-provided inputs.

Figure 5: Visualization of various mask augmentation strategies adopted in training.

Difference Mask Prediction

A novel difference mask predictor is introduced as an auxiliary branch, trained to identify all regions affected by object removal. The predictor computes binary masks by thresholding pixel-wise differences between original and edited videos, and is supervised via MSE loss. This explicit supervision guides the model to attend to subtle side effects, improving semantic correctness and realism.

Experimental Results

ROSE is evaluated on ROSE-Bench, a comprehensive benchmark comprising synthetic paired, realistic paired, and realistic unpaired video subsets. Quantitative metrics include PSNR, SSIM, and LPIPS for paired data, and VBench metrics for unpaired data.

ROSE achieves state-of-the-art results across all side-effect categories, with mean PSNR/SSIM/LPIPS of 31.12/0.917/0.077 on synthetic paired data, outperforming prior methods such as DiffuEraser, ProPainter, and FuseFormer. On realistic paired benchmarks, ROSE maintains high fidelity (PSNR 31.34, SSIM 0.923, LPIPS 0.092). VBench evaluation confirms superior motion smoothness, background consistency, and subject consistency.

Figure 6: Qualitative comparison between ROSE and existing approaches, highlighting superior handling of shadows, reflections, and illumination changes.

Ablation studies demonstrate the effectiveness of reference-based input, mask augmentation, and difference mask prediction, each contributing to improved performance on side-effect categories.

Implementation Considerations

• Computational Requirements: Training utilizes 4 NVIDIA H800 GPUs, with 80,000 optimization steps and a learning rate of 2e-5. Inference time scales with video length, which may limit efficiency for long sequences.
• Data Alignment: The use of 3D rendering ensures pixel-wise alignment of video pairs, critical for supervised learning of side effects.
• Generalization: Mask augmentation and diverse synthetic data enable robust generalization to real-world videos and imperfect user masks.
• Limitations: ROSE may exhibit flickering artifacts under large motion and reduced efficiency for long videos.

Implications and Future Directions

ROSE advances the state of video object removal by systematically addressing side effects through synthetic data generation and explicit model supervision. The framework sets a new benchmark for semantic and physical correctness in video inpainting, with practical applications in video editing, post-production, and AR/VR content creation. Future research may focus on real-time optimization, further bridging synthetic and real-world domains, and extending side effect modeling to broader environmental interactions.

Conclusion

ROSE presents a unified solution for video object removal with side effect handling, leveraging synthetic paired data and a diffusion transformer architecture with explicit difference mask supervision. Extensive experiments validate its superiority over prior methods in both quantitative and qualitative metrics. The framework and benchmark contribute to the advancement of video editing technologies, with potential for further development in efficiency and generalization.