Exploring CLIP for Assessing the Look and Feel of Images (2207.12396v2)

Abstract: Measuring the perception of visual content is a long-standing problem in computer vision. Many mathematical models have been developed to evaluate the look or quality of an image. Despite the effectiveness of such tools in quantifying degradations such as noise and blurriness levels, such quantification is loosely coupled with human language. When it comes to more abstract perception about the feel of visual content, existing methods can only rely on supervised models that are explicitly trained with labeled data collected via laborious user study. In this paper, we go beyond the conventional paradigms by exploring the rich visual language prior encapsulated in Contrastive Language-Image Pre-training (CLIP) models for assessing both the quality perception (look) and abstract perception (feel) of images in a zero-shot manner. In particular, we discuss effective prompt designs and show an effective prompt pairing strategy to harness the prior. We also provide extensive experiments on controlled datasets and Image Quality Assessment (IQA) benchmarks. Our results show that CLIP captures meaningful priors that generalize well to different perceptual assessments. Code is avaliable at https://github.com/IceClear/CLIP-IQA.

• The paper introduces a novel antonym prompt pairing strategy to overcome linguistic ambiguity in CLIP for image quality and aesthetic assessment.
• The study finds that CLIP achieves competitive performance in no-reference image quality evaluation and abstracts emotional content in images, correlating well with human perception.
• Its customizable prompt mechanism enables fine-grained evaluations of attributes like brightness and mood, paving the way for nuanced AI-driven media analysis.

Exploring CLIP for Assessing the Look and Feel of Images

The paper "Exploring CLIP for Assessing the Look and Feel of Images" presents an innovative approach to the evaluation of visual content using Contrastive Language-Image Pre-training (CLIP). This method leverages the rich visual-language priors encapsulated in CLIP models to assess both quantifiable quality perceptions, such as noise and exposure, and abstract perceptions like emotion and aesthetics, all without the necessity for explicit task-specific training.

Methodology and Contributions

The authors introduce a novel antonym prompt pairing strategy to tackle the challenges posed by linguistic ambiguity and prompt sensitivity in CLIP models. The effectiveness of CLIP in visual perception assessment hinges on the careful design of prompts, as their approach demonstrates noticeable improvements in correlation with human perception benchmarks. They explore prompts such as "Good photo" versus "Bad photo" to capture the quality perception and extend this to abstract attributes by using pairs like "Happy" versus "Sad".

Key Experiments and Results

A comprehensive suite of experiments is conducted to evaluate CLIP's ability to assess both the look (quality perception) and feel (abstract perception):

1. Quality Perception: The paper reports competitive performance of CLIP in No-Reference Image Quality Assessment (NR-IQA) compared to established models. Remarkably, CLIP-IQA achieved comparable or superior correlation scores to human perception without task-specific training on datasets such as KonIQ-10k and SPAQ.
2. Abstract Perception: By expanding CLIP's application to include abstract notions like "happy" or "sad", the study shows that CLIP can effectively discern emotional content, achieving a level of understanding similar to human evaluators.
3. Customizability: CLIP's ability to handle arbitrary prompts allows for customizable quality assessments, enabling fine-grained evaluations with prompts specifically tailored to distinct visual attributes like brightness and colorfulness.

Implications and Future Directions

The results indicate that CLIP, despite its inherent architecture designed primarily for semantic tasks, can be effectively adapted to interpret aesthetic and emotional nuances in images. This work lays groundwork for the potential extension of CLIP's capabilities beyond traditional applications, suggesting a trajectory towards more holistic AI systems capable of nuanced media analysis.

Looking forward, it is crucial to address the limitations identified, such as prompt sensitivity and the integration of domain-specific terminology. The exploration of more sophisticated prompt design methods or training CLIP with diverse prompts could further enhance its perceptual accuracy. Additionally, incorporating the vision-language priors into more task-specific architectures may bridge the performance gap observed in some tested benchmarks.

Conclusion

The paper makes a significant contribution by demonstrating the utility of CLIP in visual perception assessment, broadening the scope of its application. By circumventing the need for large, annotated datasets through the use of language-vision priors, the approach not only enhances our understanding of image quality and abstraction but also paves the way for more efficient and versatile AI assessment tools in computer vision.