BrushEdit: All-In-One Image Inpainting and Editing (2412.10316v2)

Abstract: Image editing has advanced significantly with the development of diffusion models using both inversion-based and instruction-based methods. However, current inversion-based approaches struggle with big modifications (e.g., adding or removing objects) due to the structured nature of inversion noise, which hinders substantial changes. Meanwhile, instruction-based methods often constrain users to black-box operations, limiting direct interaction for specifying editing regions and intensity. To address these limitations, we propose BrushEdit, a novel inpainting-based instruction-guided image editing paradigm, which leverages multimodal LLMs (MLLMs) and image inpainting models to enable autonomous, user-friendly, and interactive free-form instruction editing. Specifically, we devise a system enabling free-form instruction editing by integrating MLLMs and a dual-branch image inpainting model in an agent-cooperative framework to perform editing category classification, main object identification, mask acquisition, and editing area inpainting. Extensive experiments show that our framework effectively combines MLLMs and inpainting models, achieving superior performance across seven metrics including mask region preservation and editing effect coherence.

BrushEdit: All-In-One Image Inpainting and Editing

The paper presents BrushEdit, an advanced interactive framework for image editing that integrates inpainting techniques with LLMs. Unlike inversion-based or instruction-based methods prevalent in image editing models, BrushEdit incorporates a novel approach by leveraging multimodal LLMs (MLLMs) in conjunction with a dual-branch image inpainting model to enable user-friendly, instruction-guided image editing.

Background and Motivation

Traditionally, image editing tasks have been approached using diffusion models that excel in generating high-quality visuals when the output is generated from textual descriptions. However, in cases where modifications are grounded in a source image, either through inversion-based or instruction-based methodologies, there are limitations. Inversion-based models tend to struggle with large modifications due to the structure-preserving constraints of latent inversions, affecting flexibility and user interactivity. Instruction-based models, while more flexible, rely heavily on curated data and often function as black boxes, reducing user control over specific edits.

Methodology

BrushEdit provides an innovative solution to these challenges through its dual-branch architecture that facilitates image editing via an agent-cooperative framework which includes both multimodal LLMs and a sophisticated inpainting model. This approach significantly advances the current capabilities of image editing by allowing free-form manipulation through precise mask control and category classification.

Key Components:

1. Editing Instructor: The system uses pre-trained MLLMs to interpret user instructions and identify necessary editing operations, classify editing categories, locate the primary objects for modification, and produce necessary masks for editing. This ensures that the system accurately captures the user’s intent and generates detailed guidance for the editing process.
2. Inpainting Model: BrushEdit utilizes a dual-branch image inpainting model, BrushNet, to accomplish the actual editing tasks. This model processes the masked regions based on the generated masks and captions from the editing instructor. BrushNet is trained using an enhanced dataset (BrushData-v2), allowing it to support arbitrary mask shapes efficiently without requiring retraining for different mask types, contributing to its versatile applicability.

Experimental Results

BrushEdit is benchmarked against various established methods in image inpainting and editing. It excels in background fidelity and image-text alignment across multiple datasets, demonstrating marked improvements over traditional models both in angel-level benchmarks and practical applications.

Numerical Performance:

• Achieves superior scores in PSNR, MSE, and SSIM, achieving substantial improvements in preserving background fidelity.
• Demonstrates high CLIP similarity scores which indicate better alignment between edited images and natural language instructions.

The experimental setup also included user studies which confirmed that BrushEdit significantly enhances editing quality and usability compared to existing methods, particularly in maintaining coherence between edited and unedited regions.

Implications and Future Directions

BrushEdit’s primary contribution lies in establishing a robust framework for natural language-guided image editing that balances user control with computational efficiency. Practically, this could facilitate more seamless image manipulation in creative industries where personalized and precise edits are essential. Theoretically, this paradigm opens up further exploration of integrating MLLM capabilities with visual full-stack edits.

However, it also outlines areas for future research, particularly in refining the interaction between LLMs and visual generation networks, potentially improving other dimensions such as speed and flexibility via more granular user-control processes. Future research could explore expanded applications in video editing and dynamic image manipulations.

In conclusion, BrushEdit showcases a formidable step forward in image inpainting and editing, providing a scalable, flexible, and user-centric solution that adeptly bridges textual instructions with complex visual alterations.