Grounding Language Models to Images for Multimodal Inputs and Outputs (2301.13823v4)

Abstract: We propose an efficient method to ground pretrained text-only LLMs to the visual domain, enabling them to process arbitrarily interleaved image-and-text data, and generate text interleaved with retrieved images. Our method leverages the abilities of LLMs learnt from large scale text-only pretraining, such as in-context learning and free-form text generation. We keep the LLM frozen, and finetune input and output linear layers to enable cross-modality interactions. This allows our model to process arbitrarily interleaved image-and-text inputs, and generate free-form text interleaved with retrieved images. We achieve strong zero-shot performance on grounded tasks such as contextual image retrieval and multimodal dialogue, and showcase compelling interactive abilities. Our approach works with any off-the-shelf LLM and paves the way towards an effective, general solution for leveraging pretrained LLMs in visually grounded settings.

The paper "Grounding LLMs to Images for Multimodal Inputs and Outputs" (Koh et al., 2023 ) introduces Frozen Retrieval Over Multimodal Data for Autoregressive Generation (FROMAGe), a method for grounding pre-trained text-only LLMs to the visual domain. This enables the model to process arbitrarily interleaved image-and-text data and generate text interleaved with retrieved images. The key idea is to leverage the existing capabilities of LLMs, such as in-context learning and free-form text generation, while adapting them to handle visual information.

The approach involves keeping the LLM frozen and fine-tuning input and output linear layers to facilitate cross-modality interactions. The model is trained with a multi-task objective:

• Image captioning: learning to process interleaved multimodal inputs.
• Image-text retrieval: learning to produce interleaved multimodal outputs.

For image captioning, visual embeddings are extracted using a pre-trained visual encoder. A linear mapping, $\mathbf{W}_c \in \mathbb{R}^{m \times kd}$, is learned to map these embeddings into the input space of the LLM via a maximum-likelihood objective. $m$: dimension of visual embeddings $k$: number of vectors $d$: hidden dimensionality

For image-text retrieval, the LLM learns a new [RET] token representing an image. Another linear mapping, $\mathbf{W}_t \in \mathbb{R}^{p \times q}$, is trained using contrastive learning to map the [RET] embeddings for a caption to be close to the visual embeddings of its paired image. The visual embeddings $v_{\phi}(y_i)$ are mapped into the same retrieval space using the linear mapping $\mathbf{W}_i \in \mathbb{R}^{m \times q}$. $p$: hidden representation of the [RET] token from the last hidden layer of the LLM

$q$: retrieval dimension, where $q < p$

The normalized cosine similarity for the image and text embeddings is computed as:

$$\text{sim}(x, y) = \frac{(h_{\theta}(x)^T \mathbf{W}_t) (v_{\phi}(y)^T \mathbf{W}_i)^T}{ \lVert h_{\theta}(x)^T \mathbf{W}_t \rVert \lVert v_{\phi}(y)^T \mathbf{W}_i)^T \rVert }$$

Where: $x$: caption $y$: paired image $h_{\theta}(x)$: output of the last hidden layer of the LLM (LLM) for the [RET] token $v_{\phi}(y)$: output of the visual encoder for the image $\mathbf{W}_t$: linear mapping to map the hidden representation of [RET] from the last hidden layer of the LLM (LLM) $\mathbf{W}_i$: linear mapping to map the visual embeddings

The InfoNCE loss is minimized for text-to-image (t2i) and image-to-text (i2t) retrieval over a batch of $N$ text-image pairs $(x_i, y_i)$. The loss functions are:

$\mathcal{L}_{\text{t2i} = -\frac{1}{N} \sum_{i=1}^N \left( \log \frac{\exp(\text{sim}(x_i, y_i) / \tau)}{ \sum_{j=1}^N \exp(\text{sim}(x_i, y_j) / \tau )} \right)$

$\mathcal{L}_{\text{i2t} = -\frac{1}{N} \sum_{i=1}^N \left( \log \frac{\exp(\text{sim}(y_i, x_i) / \tau)}{ \sum_{j=1}^N \exp(\text{sim}(y_i, x_j) / \tau )} \right)$

Where: $\tau$: learnable temperature parameter.

The final training loss is a weighted sum of the captioning loss $\mathcal{L}_{\text{c}}$ and the retrieval losses:

$\mathcal{L} = \lambda_c \mathcal{L}_{\text{c} + \lambda_r (\mathcal{L}_{\text{t2i} + \mathcal{L}_{\text{i2t})$

Where: $\lambda_c$: captioning loss weight $\lambda_r$: retrieval loss weight

During training, only the linear mappings ($\mathbf{W}_c$, $\mathbf{W}_t$, and $\mathbf{W}_i$) and the [RET] embedding vector are updated.

The paper evaluates FROMAGe on tasks such as contextual image retrieval and visual dialogue, demonstrating strong zero-shot performance. Key findings include:

• Autoregressive LLMs can perform text-to-image retrieval with greater sensitivity to input text compared to existing models.
• The existing capabilities of pre-trained text-only LLMs can be leveraged for visually grounded tasks.

Experiments on the Visual Storytelling (VIST) dataset [huang2016visual] show that FROMAGe outperforms CLIP [radford2021learning] in contextual image retrieval, especially when provided with longer, temporally dependent sentences and interleaved image-and-text context. On Visual Dialog (VisDial) [das2017visual], FROMAGe achieves competitive results in zero-shot text answer selection and significantly outperforms prior work in text-to-image retrieval. Ablation studies validate the importance of freezing the LLM and using a dedicated retrieval token. The paper also presents results showing a positive correlation between model size and performance.