Language Models Can See: Plugging Visual Controls in Text Generation (2205.02655v2)

Abstract: Generative LLMs (LMs) such as GPT-2/3 can be prompted to generate text with remarkable quality. While they are designed for text-prompted generation, it remains an open question how the generation process could be guided by modalities beyond text such as images. In this work, we propose a training-free framework, called MAGIC (iMAge-Guided text generatIon with CLIP), for plugging in visual controls in the generation process and enabling LMs to perform multimodal tasks (e.g., image captioning) in a zero-shot manner. MAGIC is a simple yet efficient plug-and-play framework, which directly combines an off-the-shelf LM (i.e., GPT-2) and an image-text matching model (i.e., CLIP) for image-grounded text generation. During decoding, MAGIC influences the generation of the LM by introducing a CLIP-induced score, called magic score, which regularizes the generated result to be semantically related to a given image while being coherent to the previously generated context. Notably, the proposed decoding scheme does not involve any gradient update operation, therefore being computationally efficient. On the challenging task of zero-shot image captioning, MAGIC outperforms the state-of-the-art method by notable margins with a nearly 27 times decoding speedup. MAGIC is a flexible framework and is theoretically compatible with any text generation tasks that incorporate image grounding. In the experiments, we showcase that it is also capable of performing visually grounded story generation given both an image and a text prompt.

Overview of "LLMs Can See: Plugging Visual Controls in Text Generation"

The paper "LLMs Can See: Plugging Visual Controls in Text Generation" introduces an innovative framework known as MAGIC (Image-Guided text generation with CLIP). This approach leverages both text and image modalities to enhance the text generation capabilities of LLMs such as GPT-2. While LLMs excel at generating text, their ability to incorporate non-textual modalities like images remains limited. MAGIC addresses this gap by allowing LLMs to perform multimodal tasks, like zero-shot image captioning, without the need for additional training.

MAGIC employs a "plug-and-play" strategy that combines the strengths of the generative capabilities of GPT-2 and the image-text association of CLIP, a pre-trained image-text matching model. A key component of MAGIC is the introduction of a CLIP-induced scoring mechanism, termed the "magic score," which is integrated into the decoding process. This score aids the LLM in producing text that is semantically aligned with a given image while ensuring coherence with the preceding context. Notably, this scheme avoids gradient updates, significantly boosting computational efficiency.

The authors thoroughly evaluate MAGIC on two benchmarks: MS-COCO and Flickr30k. The results demonstrate remarkable performance improvements in zero-shot image captioning, with MAGIC surpassing state-of-the-art methods by substantial margins, notably achieving a 27-fold increase in decoding speed. Furthermore, its flexibility extends to visually grounded story generation, where it also exhibits strong performance. These results underscore the framework's capability to handle a variety of multimodal text generation tasks.

Implications and Future Directions

From a theoretical perspective, MAGIC sets a precedent for integrating visual information into LLMs without the need for retraining or fine-tuning, suggesting a shift towards more adaptable and resource-efficient approaches. The scalability and adaptability of such frameworks imply potential expansion to accommodate various control modalities beyond images, such as audio and video, broadening the applicability of generative LLMs in multimodal artificial intelligence.

Practically, the framework could be pivotal for applications in fields that require seamless integration of visual data into text, such as automated content creation, multimedia analysis, and interactive AI systems. The ability to guide language generation with visual contexts opens avenues for more sophisticated AI systems capable of understanding and generating text based on rich multimodal inputs.

Further work could aim to explore the integration of other sensory modalities, investigate the adaptability of MAGIC to newer LLMs, and assess its performance across a broader range of real-world applications. Emphasis on creating even more efficient methods for aligning diverse modalities will be essential in advancing AI's capability to process and generate multi-sensory information effectively.