BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion

Abstract: Image inpainting, the process of restoring corrupted images, has seen significant advancements with the advent of diffusion models (DMs). Despite these advancements, current DM adaptations for inpainting, which involve modifications to the sampling strategy or the development of inpainting-specific DMs, frequently suffer from semantic inconsistencies and reduced image quality. Addressing these challenges, our work introduces a novel paradigm: the division of masked image features and noisy latent into separate branches. This division dramatically diminishes the model's learning load, facilitating a nuanced incorporation of essential masked image information in a hierarchical fashion. Herein, we present BrushNet, a novel plug-and-play dual-branch model engineered to embed pixel-level masked image features into any pre-trained DM, guaranteeing coherent and enhanced image inpainting outcomes. Additionally, we introduce BrushData and BrushBench to facilitate segmentation-based inpainting training and performance assessment. Our extensive experimental analysis demonstrates BrushNet's superior performance over existing models across seven key metrics, including image quality, mask region preservation, and textual coherence.

• The paper introduces a dual-branch diffusion model where masked image features and noisy latents are processed separately to improve inpainting coherence.
• It employs a blurred blending strategy with an adjustable control scale to preserve unmasked regions and enhance image quality.
• Benchmark tests across multiple datasets show that BrushNet outperforms prior methods in key metrics like image quality and semantic consistency.

BrushNet: A Detailed Overview

The paper "BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion" introduces an innovative approach to image inpainting using diffusion models. BrushNet is designed to overcome limitations in traditional inpainting methods by introducing a dual-branch architecture that separately processes masked image features and noisy latents, improving semantic coherence and quality in restored images.

Introduction to BrushNet

The primary objective of image inpainting is to fill corrupted or missing regions in an image while maintaining the visual coherence with the surrounding areas. Traditional diffusion model adaptations for inpainting often face semantic inconsistencies and require a complex melding of image structures and noise. BrushNet advances this by decomposing the processing of masked images into separate branches, allowing for efficient incorporation of image features with improved coherence.

Figure 1: Performance comparisons of BrushNet and previous image inpainting methods across various inpainting tasks: random mask (with less than and more than 50% masked) and segmentation masks.

Architectural Design

Dual-Branch Model

BrushNet employs a dual-branch design where the masked image features and noisy latents are processed separately before being combined for final rendering. This decoupling dramatically reduces the learning complexity, facilitating more accurate integration of image details. The architecture outputs an inpainted image by first downsampling the mask to fit the latent size and aligning the masked image's feature distribution with a VAE encoder for hierarchical feature extraction.

Figure 2: Model overview of BrushNet. The architecture allows for pixel-level integration of masked image features with pre-trained diffusion models.

Flexible Blending and Control

To ensure robust unmasked region preservation, BrushNet introduces a blurred blending strategy, enhancing the coherence at the boundaries of the mask while maintaining high fidelity in the unmasked regions. The model also allows for adjustable control over the influence of the masked image through an adjustable control scale parameter, offering more precise control over the inpainting output.

Figure 3: Flexible control scale of BrushNet illustrating the gradual adaptation from precise to rough control as the adjustment parameter varies.

Comparative Performance Analysis

Evaluation Benchmark

The paper evaluates BrushNet's performance using proposed benchmark datasets, BrushData and BrushBench, as well as established datasets like EditBench. These provide a comprehensive evaluation landscape covering both synthetic and natural images across different inpainting scenarios, such as inside-outside mask categories.

Numerical and Qualitative Results

BrushNet outperforms existing methods across seven key metrics, including image quality, masked region preservation, and textual coherence, highlighting both quantitative superiority and qualitative enhancements in visual consistency and detail preservation.

Figure 4: Comparison between previous inpainting architectures and BrushNet, showing significant improvements in preserved detail and image fidelity.

BrushNet's results show superior coherence in style, content, color, and prompt alignment across diverse datasets, establishing new state-of-the-art benchmarks.

Insights and Future Directions

BrushNet demonstrates the potential of hierarchical feature incorporation in diffusion models, opening up avenues for improved model architectures capable of better handling complex inpainting tasks. However, the dependency on the base diffusion model and challenges with uniquely shaped masks remain. Future work could explore more generalized architectures or adaptive models to further enhance flexibility and reduce limitations.

Conclusion

BrushNet represents a significant step towards efficient and accurate image inpainting using diffusion models. By introducing a dual-branch architecture, the model improves upon existing designs by offering better image coherence and high-quality restoration. As the field advances, BrushNet's architectural insights will likely influence a broader range of applications and subsequent models in image processing and generation tasks.