TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions (2111.14813v2)

Abstract: Removing adverse weather conditions like rain, fog, and snow from images is an important problem in many applications. Most methods proposed in the literature have been designed to deal with just removing one type of degradation. Recently, a CNN-based method using neural architecture search (All-in-One) was proposed to remove all the weather conditions at once. However, it has a large number of parameters as it uses multiple encoders to cater to each weather removal task and still has scope for improvement in its performance. In this work, we focus on developing an efficient solution for the all adverse weather removal problem. To this end, we propose TransWeather, a transformer-based end-to-end model with just a single encoder and a decoder that can restore an image degraded by any weather condition. Specifically, we utilize a novel transformer encoder using intra-patch transformer blocks to enhance attention inside the patches to effectively remove smaller weather degradations. We also introduce a transformer decoder with learnable weather type embeddings to adjust to the weather degradation at hand. TransWeather achieves improvements across multiple test datasets over both All-in-One network as well as methods fine-tuned for specific tasks. TransWeather is also validated on real world test images and found to be more effective than previous methods. Implementation code can be accessed at https://github.com/jeya-maria-jose/TransWeather .

• The paper presents an efficient transformer-based architecture that restores images degraded by various adverse weather conditions.
• It employs a hierarchical encoder with intra-patch transformer blocks and learnable weather embeddings in the decoder to adapt restoration based on weather type.
• Experiments demonstrate significant improvements in PSNR and SSIM with fewer parameters and faster inference compared to previous methods.

An Assessment of "TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions"

This paper presents TransWeather, a transformer-based architecture designed for restoring images affected by adverse weather conditions such as rain, fog, and snow. Unlike existing methodologies that often specialize in mitigating one specific type of weather degradation, this work proposes an end-to-end solution that addresses multiple weather conditions using a unified model.

Core Contributions

The primary contribution of this research lies in the development of an efficient transformer framework that employs a singular encoder-decoder system. The encoder utilizes intra-patch transformer blocks to bolster attention within patches, effectively targeting smaller weather-induced degradations. This is complemented by a transformer decoder that incorporates learnable weather type embeddings to adjust dynamically based on the specific degradation encountered. The encoder is noteworthy for its ability to extract high- and low-level features through a hierarchical architecture, enhancing the model's adaptability across varied conditions.

Methodology

The proposed setup involves a single stage hierarchical transformer encoder. The encoder's innovation lies in the intra-patch mechanism, which is designed to concentrate on removing small-scale features within the patches by subdividing them further, ensuring computational efficiency. The decoder advances this by applying learnable weather type queries that tailor the restoration approach based on the weather condition identified, culminating in the conversion of hierarchical features into a clean image using convolutional blocks.

TransWeather's effectiveness is quantified through experiments across diverse datasets. It achieved significant performance boosts on synthetic datasets, involving complex conditions like combined rain and fog, as well as real-world test datasets, such as the RainDrop and Snow100K-L. Notably, TransWeather outperformed the All-in-One method, which was the prior state-of-the-art for simultaneous adverse weather removal, both in PSNR and SSIM metrics.

Performance and Efficiency

By attaining 31 M parameters compared to All-in-One's 44 M, TransWeather not only demonstrates a reduced computational load but also exhibits faster inference times, approximately 0.14 seconds per image, showcasing its potential utility in real-time applications. These results underlie the effectiveness of leveraging transformer-based solutions with a focused intra attention mechanism, positioning TransWeather as a scalable and generalizable solution for weather-related image restoration tasks.

Theoretical and Practical Implications

The paper opens discussions on employing transformer architectures for low-level vision tasks traditionally dominated by CNN-based methodologies. It suggests a path forward for incorporating intra-patch attention mechanisms and learnable query embeddings in transformers to expand their application scope. The proposal that a singular model can adapt to various weather conditions challenges the status quo of maintaining separate models for differing degradations, thereby reducing model complexity and deployment overheads which is crucial in applications such as autonomous vehicle navigation and surveillance systems.

Future Developments

While TransWeather offers remarkable improvements, real-world performance, particularly with high-intensity rain and complex splattering effects, indicates avenues for further research. Potential advancements could involve integrating physics-based models of weather phenomena with learned embeddings to increase the robustness and generalization capabilities of these frameworks.

In conclusion, TransWeather presents an advanced and efficient method for multi-weather image restoration using transformer architecture. Its novel approach and the significant improvements observed have important implications for the deployment of real-time image restoration systems across various domains afflicted by adverse weather conditions.