OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision (2411.07199v2)

Abstract: Instruction-guided image editing methods have demonstrated significant potential by training diffusion models on automatically synthesized or manually annotated image editing pairs. However, these methods remain far from practical, real-life applications. We identify three primary challenges contributing to this gap. Firstly, existing models have limited editing skills due to the biased synthesis process. Secondly, these methods are trained with datasets with a high volume of noise and artifacts. This is due to the application of simple filtering methods like CLIP-score. Thirdly, all these datasets are restricted to a single low resolution and fixed aspect ratio, limiting the versatility to handle real-world use cases. In this paper, we present \omniedit, which is an omnipotent editor to handle seven different image editing tasks with any aspect ratio seamlessly. Our contribution is in four folds: (1) \omniedit is trained by utilizing the supervision from seven different specialist models to ensure task coverage. (2) we utilize importance sampling based on the scores provided by large multimodal models (like GPT-4o) instead of CLIP-score to improve the data quality. (3) we propose a new editing architecture called EditNet to greatly boost the editing success rate, (4) we provide images with different aspect ratios to ensure that our model can handle any image in the wild. We have curated a test set containing images of different aspect ratios, accompanied by diverse instructions to cover different tasks. Both automatic evaluation and human evaluations demonstrate that \omniedit can significantly outperform all the existing models. Our code, dataset and model will be available at https://tiger-ai-lab.github.io/OmniEdit/

• The paper presents OmniEdit’s specialist-to-generalist supervision, leveraging seven specialist models to achieve comprehensive image editing tasks.
• The paper introduces the innovative EditNet diffusion-transformer hybrid, enhancing training quality via importance sampling with GPT-4o-scored data.
• The paper demonstrates that OmniEdit processes images of any aspect ratio, yielding a notable 20% improvement in perceptual quality over baseline models.

Insights into OmniEdit: Building an Image Editing Generalist Model

The paper under review presents OmniEdit, a comprehensive framework for instruction-based image editing, intending to unify and enhance the versatility of image manipulation methods. By leveraging the supervision from multiple specialist models, this approach tackles a multitude of editing tasks while overcoming limitations inherent in traditional methods. OmniEdit claims notable advances across several aspects of image editing, particularly in its capability to process images of any aspect ratio and resolution. This work holds substantial implications for both practical image editing applications and theoretical developments in computer vision and artificial intelligence.

Key Innovations and Methodological Approaches

OmniEdit is designed as an omnipotent editor, addressing key challenges in existing image editing models: biased data synthesis, inadequate training quality control, and restricted dataset resolutions. The paper outlines four primary contributions of the proposed model:

1. Specialist-to-Generalist Supervision: By training OmniEdit to learn from seven specialist models, each excelling in distinct editing tasks, the framework ensures broad task coverage. This method contrasts with prior models' reliance on single, often biased expert approaches, thus promoting a more balanced and comprehensive editing skill set.
2. Importance Sampling for Data Quality: The use of large multimodal models like GPT-4o to score data samples, as opposed to the less reliable CLIP-score, introduces a sophisticated feedback loop for importance sampling. The distilled modeling from GPT-4o to the InternVL2 model improves the curation of training data, significantly enhancing the overall quality of the training sets.
3. EditNet Architecture: The introduction of the EditNet architecture showcases an innovative diffusion-transformer hybrid capable of handling diverse editing tasks with improved precision. This architecture facilitates effective interaction between control and original branches, thereby increasing the model's comprehension of editing instructions.
4. Support for Diverse Aspect Ratios: Training with varied aspect ratios elevates OmniEdit's capability to process a wider array of real-world images without loss of quality. This aspect significantly broadens the model's applicability in practical scenarios.

Evaluation and Results

Comprehensive evaluations demonstrate OmniEdit's superiority over existing models across several domains, including automatic metrics and human assessments. Notably, the model outperforms others in perceptual quality and semantic consistency by a substantial margin. Specifically, the human evaluation highlights a notable 20% improvement on average over the best baseline models like CosXL-Edit.

Implications and Future Directions

The advancements presented by OmniEdit suggest several implications for the field. Practically, this model could be crucial for enhancing user experiences in applications requiring high-precision image editing, such as media production and digital marketing. Theoretically, OmniEdit’s architecture and training strategy could inspire future research directions in model generalization across varied image manipulation tasks.

From a broader perspective, the reliance on specialist models to train a generalist suggests a promising strategy for other AI model domains, potentially influencing approaches to multitask learning and reducing biases in machine learning pipelines. Looking forward, integrating more potent underlying base models could further amplify OmniEdit’s capabilities and extend its current framework, setting new benchmarks in image editing standards.

In summary, OmniEdit represents a substantial step forward in the evolution of image editing tools, both broadening the scope of tasks these models can handle and improving the quality and reliability of their outputs. As the landscape of AI continues to evolve, methodologies like those in OmniEdit will likely shape future innovations in image manipulation and beyond.