MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing (2306.10012v3)

Abstract: Text-guided image editing is widely needed in daily life, ranging from personal use to professional applications such as Photoshop. However, existing methods are either zero-shot or trained on an automatically synthesized dataset, which contains a high volume of noise. Thus, they still require lots of manual tuning to produce desirable outcomes in practice. To address this issue, we introduce MagicBrush (https://osu-nlp-group.github.io/MagicBrush/), the first large-scale, manually annotated dataset for instruction-guided real image editing that covers diverse scenarios: single-turn, multi-turn, mask-provided, and mask-free editing. MagicBrush comprises over 10K manually annotated triplets (source image, instruction, target image), which supports trainining large-scale text-guided image editing models. We fine-tune InstructPix2Pix on MagicBrush and show that the new model can produce much better images according to human evaluation. We further conduct extensive experiments to evaluate current image editing baselines from multiple dimensions including quantitative, qualitative, and human evaluations. The results reveal the challenging nature of our dataset and the gap between current baselines and real-world editing needs.

• The paper introduces MagicBrush, the first large-scale manually annotated dataset designed to overcome limitations of synthetic text-guided image editing.
• It details a rigorous construction process using iterative DALL-E 2 generations and strict quality controls to ensure precise instruction-image alignment.
• Empirical evaluations show that models fine-tuned on MagicBrush achieve superior text-image alignment and perceptual similarity with improved editing accuracy.

An Analysis of MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing

MagicBrush represents the first large-scale dataset specifically curated for instruction-guided image editing, addressing the limitations of existing text-guided techniques that suffer from noise due to automatic synthesis. The dataset offers over 10,000 manually annotated triplets (source image, instruction, and target image), facilitating the training of models for more accurate and diverse image editing tasks.

Introduction

The paper underscores the necessity of semantic edits on images for various applications, positing natural language as an intuitive medium for such tasks. Text-guided approaches are categorized mainly as zero-shot or end-to-end editing, both of which rely on synthetic datasets that potentially fail to reflect real-world complexities and needs. MagicBrush, contrarily, aims to fill this gap by providing a manually annotated corpus that mimics realistic editing scenarios, encompassing single-turn, multi-turn, mask-provided, and mask-free scenarios.

Dataset Construction

MagicBrush is constructed using rigorous quality control measures for its annotators, who are tasked with devising natural language instructions for image edits. These instructions are then used to generate images using DALL-E 2, repeating the process iteratively until satisfactory changes are made or replacing failed attempts with new instructions. This ensures high fidelity between instructions and image transformations and captures the nuance of real-world editing needs.

Empirical Evaluation

The paper evaluates existing image editing models on MagicBrush using a variety of metrics, including L1, L2, CLIP, and human evaluations, establishing baseline performances with and without mask guidance.

For mask-free settings, InstructPix2Pix fine-tuned on MagicBrush notably improved its performance, outperforming other models in text-image alignment and perceptual similarity with target outputs. This advancement highlights MagicBrush’s strength in refining model outputs to align with human-instructed edits without excessive alterations. Among mask-provided models, although showing promise in overall perceptual similarity, they fall short in targeted adjustments as seen in mask-free capabilities, indicating that fine-tuning with quality data like MagicBrush can bridge this gap effectively.

Implications and Future Directions

The results of MagicBrush’s applicability suggest significant practical and theoretical implications: the dataset elevates baseline capabilities of current text-based models, prompting further development of nuanced and less error-prone AI models. From a practical standpoint, models trained on such a robust dataset hold promise for more intuitive user interfaces in consumer editing software, potentially democratizing complex image editing.

Future research might explore leveraging MagicBrush for user-specific fine-tuning, or implementing comprehensive model evaluation strategies by combining MagicBrush with innovative metrics for deeper inspection of model robustness and edit credibility. Moreover, the paper calls for development in dataset extensions accommodating broad changes (global edits) or incorporating generative aspects beyond rigid transformations.

In summary, MagicBrush, crafted with meticulous attention to annotation quality and diversity, stands as a crucial resource in advancing text-guided image editing, laying groundwork for future explorations that merge intricate AI capabilities with human-like understanding and execution of semantic image edits.