Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language (2306.16410v1)

Abstract: We propose LENS, a modular approach for tackling computer vision problems by leveraging the power of LLMs. Our system uses a LLM to reason over outputs from a set of independent and highly descriptive vision modules that provide exhaustive information about an image. We evaluate the approach on pure computer vision settings such as zero- and few-shot object recognition, as well as on vision and language problems. LENS can be applied to any off-the-shelf LLM and we find that the LLMs with LENS perform highly competitively with much bigger and much more sophisticated systems, without any multimodal training whatsoever. We open-source our code at https://github.com/ContextualAI/lens and provide an interactive demo.

• The paper introduces the LENS approach which leverages language models as reasoning modules integrated with standard vision components to perform visual tasks without additional pretraining.
• The methodology employs vision modules such as CLIP and BLIP to generate rich textual descriptions, enabling effective object recognition and question answering.
• Evaluations reveal that LENS achieves competitive performance in zero-shot and few-shot settings, surpassing models like Flamingo on benchmarks like VQA 2.0 while noting limitations on datasets like OK-VQA.

Towards LLMs That Can See: Computer Vision Through the Lens of Natural Language

The paper presents a modular approach, termed LENS, aiming to enhance the capabilities of LLMs by enabling them to tackle computer vision tasks via natural language descriptions. The authors propose using a LLM to interpret outputs from independently operating vision modules, which provide comprehensive textual information regarding images. This method stands out due to its pretraining-free nature, allowing application to any standard LLM without necessitating additional multimodal datasets.

Overview and Methodology

The research introduces LENS, which leverages LLMs as reasoning modules and integrates them with predefined vision components. This integration eliminates the conventional multimodal pretraining stages typically required for aligning visual and textual modalities, as observed in existing models like Flamingo and Kosmos-1. LENS utilizes vision modules such as CLIP for tag and attribute recognition, and BLIP for generating diverse image captions.

Critically, LENS capitalizes on the semantic reasoning capabilities inherent to LLMs. It processes detailed textual descriptions generated by vision modules to perform vision-related tasks, such as object recognition and question answering, without extra vision-and-language alignment.

Experimental Evaluation

The paper evaluates LENS comprehensively across both computer vision and vision-language benchmarks. The zero-shot object recognition task demonstrates that LENS configurations deliver competitive or superior performance compared to standalone models like CLIP. Notably, LENS, integrating ViT-H/14 as the visual backbone and Flan-T5\textsubscript{xxl} as the LLM, achieved enhancement over baseline CLIP models. Moreover, few-shot experiments highlight that increasing dataset shots and employing larger vision encoders further amplify performance.

In vision-language reasoning tasks, LENS shows robust results against state-of-the-art models that rely on extensive multimodal pretraining. Specifically, LENS with Flan-T5\textsubscript{XXL} surpassed several iterations of Flamingo on benchmarks like VQA 2.0, demonstrating its efficacy without the computational expense associated with training on large paired datasets. However, on certain datasets such as OK-VQA, LENS does lag behind, possibly due to less extensive LLM knowledge bases compared to models like Flamingo equipped with larger LLMs.

Implications and Future Directions

The paper illustrates significant theoretical and practical advancements, offering a streamlined method for incorporating vision capabilities into LLMs without additional training. It highlights the modular potential of LENS to enhance existing and future LLMs with minimal overhead, thereby pushing the boundaries of multimodal AI integration.

Future developments may involve extending this modular approach to other domains such as audio or video processing, enriching the cross-modal integration capabilities. Furthermore, optimizing the efficiency of integrating large LLMs with complex vision tasks remains an open area of exploration.

Conclusion

LENS stands as a promising shift in how vision and language domains are harmonized, showcasing competitive results in computer vision and reasoning tasks without the burdens of traditional multimodal pretraining. The modularity and applicability across diverse LLMs suggest a significant step toward more accessible and computationally efficient AI models, fostering further exploration and innovation in multimodal AI integration.