DiffuEraser: A Diffusion Model for Video Inpainting (2501.10018v1)

Abstract: Recent video inpainting algorithms integrate flow-based pixel propagation with transformer-based generation to leverage optical flow for restoring textures and objects using information from neighboring frames, while completing masked regions through visual Transformers. However, these approaches often encounter blurring and temporal inconsistencies when dealing with large masks, highlighting the need for models with enhanced generative capabilities. Recently, diffusion models have emerged as a prominent technique in image and video generation due to their impressive performance. In this paper, we introduce DiffuEraser, a video inpainting model based on stable diffusion, designed to fill masked regions with greater details and more coherent structures. We incorporate prior information to provide initialization and weak conditioning,which helps mitigate noisy artifacts and suppress hallucinations. Additionally, to improve temporal consistency during long-sequence inference, we expand the temporal receptive fields of both the prior model and DiffuEraser, and further enhance consistency by leveraging the temporal smoothing property of Video Diffusion Models. Experimental results demonstrate that our proposed method outperforms state-of-the-art techniques in both content completeness and temporal consistency while maintaining acceptable efficiency.

• The paper introduces DiffuEraser, a novel diffusion model that addresses video inpainting by improving known pixel propagation, unknown pixel generation, and temporal consistency.
• It leverages stable diffusion and temporal smoothing techniques to overcome issues like blurring and frame inconsistencies in large mask scenarios.
• Experimental results show that DiffuEraser outperforms state-of-the-art methods in content completeness and computational efficiency, paving the way for advanced video editing applications.

DiffuEraser: Advancements in Video Inpainting through Diffusion Models

The paper "DiffuEraser: A Diffusion Model for Video Inpainting" presents a novel approach to video inpainting using diffusion models, addressing some limitations of existing methods by enhancing generative capabilities and temporal consistency. The model, termed DiffuEraser, utilizes stable diffusion mechanisms to improve the quality of masked region restoration in videos, particularly when dealing with large mask sizes.

Recent advancements in video inpainting have leveraged flow-based pixel propagation and transformer-based methods. However, existing approaches often encounter issues such as blurring and temporal inconsistencies, particularly in scenarios involving large masks. The challenges inherent in these methods necessitate the development of more robust generative models, with diffusion models emerging as a promising avenue due to their demonstrated efficacy in image and video generation tasks.

DiffuEraser is developed to address three core sub-problems of video inpainting: the propagation of known pixels, the generation of unknown pixels, and ensuring temporal consistency of completed content.

1. Propagation of Known Pixels: The model enhances the propagation of known pixels using motion modules and by leveraging propagation capabilities from the prior model, Propainter. This incorporation ensures that known pixels are accurately and consistently propagated across frames, aligning the completed content with unmasked regions.
2. Generation of Unknown Pixels: Utilizing the powerful generative capabilities inherent within stable diffusion models, DiffuEraser is designed to generate plausible and detailed content for unknown pixels (those that have not previously appeared in any masked frames), effectively addressing issues of structural integrity and detail.
3. Temporal Consistency: To address temporal inconsistencies that arise during long-sequence inference, the model expands the temporal receptive fields and employs the temporal smoothing properties of Video Diffusion Models (VDMs). This ensures consistency across frame transitions and enhances the smoothness of content at clip intersections.

The paper details the network architecture influenced by AnimateDiff, integrating the diffusion-based image inpainting model BrushNet that incorporates additional feature extraction layers. This multitask architecture facilitates seamless noise reduction and feature correlation through temporal attention mechanisms, thereby enhancing temporal consistency.

To mitigate noisy artifacts frequently encountered in diffusion models, the paper ascribes importance to the incorporation of priors, derived from performing DDIM Inversion on outputs from a lightweight inpainting model. This approach stabilizes the inpainting outcomes, reducing the manifestation of noisy artifacts and unwanted visual elements.

The research implications are significant in both practical and theoretical contexts, suggesting potential advancements and application scenarios in video editing domains, such as object removal and scene reconstruction, where temporal consistency is crucial. Furthermore, by enhancing the video inpainting framework using diffusion models, this research outlines potential pathways for integrating advanced motion models in other temporal generative tasks within the AI landscape.

The experimental results presented demonstrate that DiffuEraser surpasses current state-of-the-art techniques in both content completeness and temporal consistency, while maintaining computational efficiency through Phased Consistency Models (PCM). While promising, the authors hint at possible further explorations into video editing tasks beyond inpainting, suggesting substantial progressions in AI-driven video content manipulation. The methodologies and theoretical underpinnings conveyed in this work may serve to inspire subsequent investigations and model developments within the broader AI and computer vision communities.