ARNIQA: Learning Distortion Manifold for Image Quality Assessment (2310.14918v2)

Abstract: No-Reference Image Quality Assessment (NR-IQA) aims to develop methods to measure image quality in alignment with human perception without the need for a high-quality reference image. In this work, we propose a self-supervised approach named ARNIQA (leArning distoRtion maNifold for Image Quality Assessment) for modeling the image distortion manifold to obtain quality representations in an intrinsic manner. First, we introduce an image degradation model that randomly composes ordered sequences of consecutively applied distortions. In this way, we can synthetically degrade images with a large variety of degradation patterns. Second, we propose to train our model by maximizing the similarity between the representations of patches of different images distorted equally, despite varying content. Therefore, images degraded in the same manner correspond to neighboring positions within the distortion manifold. Finally, we map the image representations to the quality scores with a simple linear regressor, thus without fine-tuning the encoder weights. The experiments show that our approach achieves state-of-the-art performance on several datasets. In addition, ARNIQA demonstrates improved data efficiency, generalization capabilities, and robustness compared to competing methods. The code and the model are publicly available at https://github.com/miccunifi/ARNIQA.

ARNIQA: Learning Distortion Manifold for Image Quality Assessment

The paper "ARNIQA: Learning Distortion Manifold for Image Quality Assessment" introduces a self-supervised approach for No-Reference Image Quality Assessment (NR-IQA), a subfield tasked with assessing image quality without the need for a high-quality reference image. This work, known as ARNIQA (leArning distoRtion maNifold for Image Quality Assessment), strives to capture the intrinsic nature of image distortions by modeling them as a manifold and seeks to improve upon current methodologies by aligning image quality assessments more closely with human perception.

Methodological Overview

1. Distortion Manifold Learning: The central idea of this paper is to learn a manifold that represents various image distortion patterns without being reliant on a content-dependent understanding of the images themselves. By utilizing a pre-trained encoder in conjunction with a self-supervised learning framework, ARNIQA seeks to maximize the similarity of patch representations from different images that have undergone identical distortion types, thus focusing on the learned distortion manifold.
2. Image Degradation Model: ARNIQA introduces a degradation model that randomly generates an extensive variety of distortion compositions (approximately 1.9 billion) by applying synthetically-induced distortions to pristine images. These degradation operations are applied in varying sequences and intensities, allowing the model to explore a diverse range of potential distortions that might be encountered in real-world scenarios.
3. Self-Supervised Training Protocol: The self-supervised learning strategy capitalizes on a contrastive learning approach where image crops from different images, degraded in the same fashion, are encouraged to produce similar embeddings. This method diverges from existing strategies that generate embeddings from different crops of the same image, thus potentially entangling image content with distortion patterns. The use of hard negative samples, specifically half-scale versions of the original images, enhances the learning by demanding a more fine-grained discrimination.
4. Linear Regression for Quality Prediction: Once the encoder has been trained to map distortions to representation areas on the manifold, the final quality scores are derived via a simple linear regressor, facilitating an efficient mapping from manifold representations to perceptual quality scores without further fine-tuning of the encoder.

Experimental Validation

The performance evaluation of ARNIQA demonstrates exceptional results across both synthetic and real-world distortion datasets, with the model achieving state-of-the-art results on benchmarks such as LIVE, CSIQ, TID2013, KADID10k, and more. Notably, ARNIQA excels in data efficiency, often requiring only a fraction of the training data relative to competitor methods like CONTRIQUE and Re-IQA, while still achieving superior or comparable performance metrics.

ARNIQA's cross-dataset generalization capabilities were further highlighted through evaluation on different datasets, where it outperformed other methods by better modeling the consistent structure of the quality distortion manifold. The gMAD performance test also exhibited superior robustness when pitted against other methodologies, identifying less visible discrepancies when ARNIQA was used as the defender.

Implications and Future Directions

The implications of this model are substantial both in practical and theoretical realms. Practically, the capacity to learn such a vast and comprehensive distortion manifold means that this representation can be effectively leveraged in diverse applications from image restoration efforts in multimedia systems to evaluating image uploads in social media platforms. Theoretically, the approach reflects a shift toward understanding and utilizing image distortions through manifold learning, potentially influencing the development of new models that capitalize on this approach for other computer vision tasks.

For future developments, ARNIQA opens pathways toward further refining image quality metrics to bridge the gap between algorithmic assessment and human perceptual judgment. Additionally, the utilization of the distortion manifold for the design of blind image enhancement and restoration frameworks could be an intriguing area of paper, as well as the exploration of more complex manifold structures than current linear regression mappings. This work reflects a significant step forward in the endeavor to understand and reproduce human-like image quality evaluation in an automated manner.