Instruct-IPT: All-in-One Image Processing Transformer via Weight Modulation

Abstract: Due to the unaffordable size and intensive computation costs of low-level vision models, All-in-One models that are designed to address a handful of low-level vision tasks simultaneously have been popular. However, existing All-in-One models are limited in terms of the range of tasks and performance. To overcome these limitations, we propose Instruct-IPT -- an All-in-One Image Processing Transformer (IPT) that could effectively address manifold image restoration tasks with large inter-task gaps, such as denoising, deblurring, deraining, dehazing, and desnowing. While most research propose feature adaptation methods, we reveal their failure in addressing highly distinct tasks, and suggest weight modulation that adapts weights to specific tasks. Firstly, we search for task-sensitive weights and introduce task-specific biases on top of them. Secondly, we conduct rank analysis for a good compression strategy and perform low-rank decomposition on the biases. Thirdly, we propose synchronous training that updates the task-general backbone model and the task-specific biases simultaneously. In this way, the model is instructed to learn both general and task-specific knowledge. Via our simple yet effective method that instructs the IPT to be task experts, Instruct-IPT could better cooperate between tasks with distinct characteristics at humble costs. As an additional feature, we enable Instruct-IPT to receive human prompts. We have conducted experiments on Instruct-IPT to demonstrate the effectiveness of our method on manifold tasks, and we have effectively extended our method to diffusion denoisers as well. The code is available at https://github.com/huawei-noah/Pretrained-IPT.

• The paper presents a novel weight modulation technique that directly adjusts model weights to tackle diverse image restoration tasks.
• It employs low-rank bias decomposition and synchronous training to boost computational efficiency while maintaining high task performance.
• The model integrates multi-modal inputs via human prompts, enabling versatile and natural interaction for unified image processing.

Instruct-IPT: An Innovative Approach to All-in-One Image Processing

The paper presents Instruct-IPT, a transformative approach to achieving efficient and effective results in low-level image processing tasks using transformers. Traditionally, different image restoration tasks such as denoising, deblurring, deraining, dehazing, and desnowing have necessitated distinct models, each optimized for specific input properties and output requirements. In contrast, Instruct-IPT offers a unified framework capable of addressing multiple tasks, simplifying the model deployment and maintenance landscape significantly.

Core Contributions

Instruct-IPT introduces a novel weight modulation technique that adjusts model weights by overlaying task-specific biases onto a shared, task-general backbone. This is accomplished through several key innovations:

1. Task-Sensitive Weight Modulation: This method diverges from conventional feature modulation techniques typically employed in multitask frameworks, which alter the feature space rather than modifying the model's inherent parameters. The authors argue that modifying weights, rather than features, is more effective for handling tasks with large inter-task differences.
2. Low-Rank Decomposition of Biases: Recognizing the computational and memory burdens typical of large-scale models, the authors incorporate a low-rank decomposition of the task-specific biases, thereby reducing the overhead while maintaining performance. This careful consideration of computational efficiency enables the deployment of Instruct-IPT in resource-constrained environments.
3. Synchronous Training Methodology: Unlike the standard two-phase training (initial general training followed by task-specific fine-tuning), synchronous training updates both the task-general and task-specific components concurrently. This unified process is purported to facilitate better knowledge transfer between related tasks and avoid the performance bottlenecks often encountered in segmented training workflows.
4. Integration of Multi-Modal Inputs via Human Prompts: The model can interpret and act on text instructions, positioning it as a multi-modal framework compatible with human language inputs. This feature not only extends the range of interaction but also enhances its usability in practical scenarios, allowing it to be directed using plain language commands.

Results and Evaluation

The empirical results are robust, demonstrating that Instruct-IPT achieves high performance across a spectrum of image restoration benchmarks while maintaining computational efficiency. The paper reports competitive or even superior performance to existing specialized models on each task, highlighting not only the model’s versatility but also its capability to retain task-specific quality. The introduction of a tiny version of Instruct-IPT allows for further scalability, offering capabilities comparable to larger models with reduced computation and memory demands.

Implications and Future Directions

Instruct-IPT represents a significant step forward in the integration of multiple low-level vision tasks within a single architectural framework. The implications of this work extend to both practical applications and theoretical advancements in AI. The ability to share a general backbone across multiple tasks suggests pathways for future exploration in reducing redundancy in model architectures, potentially leading to further breakthroughs in unified AI models.

In terms of future developments, this paper sets a precedent in multi-task learning, encouraging further examination into the blend of different model architectures, such as incorporating capabilities for high-level vision tasks or extending into domains beyond image processing, such as video or sensor data assimilation. Additionally, the integration of human prompts within the processing workflow opens up avenues for expanding the adaptability of AI models to more naturalistic and interactive forms of human-machine communication.

Overall, Instruct-IPT offers a novel and efficient paradigm for unified image processing tasks, underscored by its innovative efforts in weight modulation, low-rank adaptation, and multi-modal capabilities. Its introduction may likely inspire subsequent research avenues in the quest to refine multitask learning and integration approaches.