PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions (2409.15278v4)

Abstract: This paper presents a versatile image-to-image visual assistant, PixWizard, designed for image generation, manipulation, and translation based on free-from language instructions. To this end, we tackle a variety of vision tasks into a unified image-text-to-image generation framework and curate an Omni Pixel-to-Pixel Instruction-Tuning Dataset. By constructing detailed instruction templates in natural language, we comprehensively include a large set of diverse vision tasks such as text-to-image generation, image restoration, image grounding, dense image prediction, image editing, controllable generation, inpainting/outpainting, and more. Furthermore, we adopt Diffusion Transformers (DiT) as our foundation model and extend its capabilities with a flexible any resolution mechanism, enabling the model to dynamically process images based on the aspect ratio of the input, closely aligning with human perceptual processes. The model also incorporates structure-aware and semantic-aware guidance to facilitate effective fusion of information from the input image. Our experiments demonstrate that PixWizard not only shows impressive generative and understanding abilities for images with diverse resolutions but also exhibits promising generalization capabilities with unseen tasks and human instructions. The code and related resources are available at https://github.com/AFeng-x/PixWizard

• The paper introduces PixWizard, a unified image-to-image assistant that leverages a 30-million-instance dataset to support a wide range of vision tasks.
• It demonstrates competitive performance in image restoration, editing, grounding, and dense prediction, often matching or surpassing specialized models.
• The model integrates innovative techniques such as structure-aware guidance, semantic-aware fusion, and task-aware dynamic sampling to handle varied resolutions and tasks effectively.

PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions

The paper "PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions" introduces PixWizard, a comprehensive image-to-image visual assistant designed to handle a variety of vision tasks through natural language instructions. This system integrates various vision tasks within a unified image-text-to-image generation framework, leveraging an extensive Omni Pixel-to-Pixel Instruction-Tuning Dataset (OPPIT) consisting of 30 million instances across a wide spectrum of visual tasks.

Unified Framework and Dataset Construction

PixWizard is built upon the Diffusion Transformer (DiT) foundation model, enhanced with capabilities to handle any resolution and aspect ratio, closely mimicking human perceptual processes. The model incorporates structure-aware and semantic-aware guidance mechanisms for effective information fusion from input images.

The OPPIT dataset is meticulously curated to cover a broad array of vision tasks, including:

1. Image Generation: Tasks like text-to-image generation, controllable generation, inpainting, and outpainting.
2. Image Editing: Such as object removal, object replacement, background replacement, and style transfer.
3. Image Restoration: Covering deraining, desnowing, deblurring, super-resolution, and more.
4. Image Grounding: Locating objects based on user prompts.
5. Dense Image Prediction: Encompassing depth estimation, surface normal estimation, semantic segmentation, and various image-to-image translations.

Model Enhancements and Training

PixWizard incorporates several innovative components:

• Text Encoders: Utilizing both Gemma-2B and the CLIP text encoder to generate task embeddings, significantly aiding the model's ability to follow task-specific instructions accurately.
• Structural-Aware Guidance: Enhancing the model’s capacity to capture the overall structure of input images.
• Semantic-Aware Guidance: Introducing a zero-initialized gating mechanism to integrate image and text information effectively.
• Task-Aware Dynamic Sampler: Selectively sampling relevant semantic tokens to improve efficiency and accuracy.
• Any Resolution Mechanism: The dynamic partitioning and padding scheme, along with NTK-Aware Scaled RoPE and sandwich normalization, enable handling of varied image resolutions effectively.

The training process involves a two-stage strategy with data balancing, ensuring robust performance across diverse tasks.

Experimental Results and Performance

PixWizard showcases impressive performance, often comparable or superior to both task-specific models and existing vision generalists:

• Image Restoration: PixWizard demonstrates competitive performance in image denoising, deraining, and other restoration tasks, outperforming many established methods.
• Image Grounding: It significantly surpasses other generalist models like InstructDiffusion in referring segmentation tasks.
• Dense Image Prediction: Achieves competitive accuracy in tasks such as depth estimation, surface normal estimation, and semantic segmentation.
• Image Editing: On par with state-of-the-art methods such as Emu Edit, PixWizard excels in various editing tasks, accurately following user instructions.
• Image Generation: Shows strong results in conditional generation, inpainting, and outpainting, with high-quality image synthesis capabilities.

Implications and Future Directions

The versatility and solid performance of PixWizard suggest its potential applicability in practical real-world scenarios, catering to diverse user requirements with its free-form instruction-following capability. The incorporation of advanced text encoders and innovative model components enhances its interaction and usability.

Future developments could focus on supporting multi-image input conditions and further improving performance on complex tasks like segmentation. Integrating more advanced foundational models, such as SD3 and FLUX, could also enhance the model's generative and predictive capabilities, fostering advancements toward achieving a highly capable, general-purpose visual assistant.

In conclusion, PixWizard represents a significant step in the development of versatile, interactive visual assistants capable of handling a wide range of tasks through natural, free-form instructions.