---
title: 'V.I.P.: Versatile Image Outpainting'
url: https://www.emergentmind.com/topics/v-i-p
type: topic
---

# V.I.P.: Versatile Image Outpainting

VIP, in the sense of “Versatile Image Outpainting Empowered by Multimodal Large Language Model,” is an image outpainting framework that extrapolates the surrounding parts of an image from a given center patch while supporting both unconditional generation and user-specified customization through text prompts. The framework combines automatic prompt generation by a Multimodal Large Language Model (MLLM), a Center-Total-Surrounding (CTS) cross-attention block, and light fine-tuning of a Stable Diffusion inpainting backbone rather than training from scratch. On the Scenery, Building, and WikiArt benchmarks, it is reported to surpass prior state-of-the-art methods in Fréchet Inception Distance (FID), while also exhibiting customized outpainting behavior under changes in the surrounding prompt specification [2406.01059].

## 1. Problem setting and design goals

Image outpainting aims to extrapolate the surrounding parts of an image given its center contents. In VIP, the task is formulated as enlarging an image from its central patch so that the generated borders are semantically coherent and visually seamless in all four surrounding directions [2406.01059].

The framework is organized around three stated objectives. The first is **versatility**, meaning support for both unconditional outpainting and conditional outpainting driven by free-form text prompts. The second is **customization**, meaning that users can dictate what kinds of objects or styles appear in newly generated regions. The third is **efficiency**, achieved by leveraging a pre-trained text-to-image diffusion model, specifically Stable Diffusion, through light fine-tuning rather than training a new outpainting model from scratch [2406.01059].

This combination places VIP in a distinct position relative to earlier outpainting methods summarized in the paper. The work explicitly identifies the lack of versatility and customization as a practical limitation of recent approaches, and proposes prompt-aware diffusion fine-tuning as the remedy. A plausible implication is that the framework is intended not merely as a reconstruction system but as a controllable generative system whose output can vary for the same center crop under different prompt specifications.

## 2. Automatic prompt extraction and conditioning interface

A central component of VIP is the automatic extraction of structured textual prompts from training images using GPT-4V. Each image is fed to the MLLM with the instruction:

> “Please use three English keywords to describe the center and the surrounding of the image, and output in the following format: Center:xxx,xxx,xxx; Surrounding:yyy,yyy,yyy.” [2406.01059]

GPT-4V returns prompts in exactly that template. The resulting prompt corpus is then partitioned into two subsets. In **conditional** examples, both the `Center` and `Surrounding` fields are non-empty. In **unconditional** examples, the keywords are simply zeroed out, yielding `Center:; Surrounding:`. The paper states that no human labeling is required for this stage [2406.01059].

This prompt format is structurally important because the model does not treat the prompt as an undifferentiated text string. Instead, VIP preserves the distinction between center-related and surrounding-related tokens and uses that distinction in its attention mechanism. The `Center:` field corresponds to already visible image content, whereas the `Surrounding:` field corresponds to the masked region that must be synthesized. This suggests that the prompt interface is designed not only for conditioning, but also for explicit region-to-text alignment.

The paper further reports that even in unconditional outpainting, the structured empty-format prompt is beneficial. On Scenery, a traditional empty string prompt gives FID \(=15.42\) and IS \(=4.13\), whereas the VIP prompt format `Center:; Surrounding:` gives FID \(=14.88\) and IS \(=4.12\) after the full method, indicating that the explicit prompt schema itself contributes to the model’s behavior [2406.01059].

## 3. Diffusion backbone and Center-Total-Surrounding cross-attention

VIP starts from the Stable Diffusion Inpainting model, whose UNet input has 9 channels: noisy latent, masked latent, and binary mask. During both training and inference, the ground-truth image \(x_0\) is encoded by a frozen VAE encoder into a latent \(z_0\). A timestep \(t\) is sampled, Gaussian noise \(\epsilon \sim N(0,I)\) is added, and the noisy latent is formed as

$$
z_t = \sqrt{\alpha_t}\, z_0 + \sqrt{1-\alpha_t}\,\epsilon.
$$

The concatenated input \((z_t,\text{masked\_}z_t,\text{mask})\) is then fed into the UNet, and text prompt embeddings are injected in each cross-attention layer [2406.01059].

The distinctive architectural addition is the **Center-Total-Surrounding (CTS)** cross-attention module. Standard cross-attention between image features \(F_I \in \mathbb{R}^{HW \times d}\) and text features \(F_T \in \mathbb{R}^{L \times d}\) is written as

$$
Q = W_Q F_I,\qquad
K = W_K F_T,\qquad
V = W_V F_T,
$$

$$
Y = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V.
$$

CTS splits the prompt embedding into center tokens \(F_T^C\) and surrounding tokens \(F_T^S\), and computes two parallel cross-attentions:

$$
Y^C = \mathrm{softmax}\!\left(\frac{Q(K^C)^\top}{\sqrt{d}}\right)V^C,\qquad
Y^S = \mathrm{softmax}\!\left(\frac{Q(K^S)^\top}{\sqrt{d}}\right)V^S.
$$

A global attention branch \(Y^G\) over the full prompt is retained as well. The region-specific outputs are then blended with the binary mask \(m \in \{0,1\}^{HW}\), where \(m=1\) in the outpaint region:

$$
\hat{Y}^C = Y^C \odot (1-m),\qquad
\hat{Y}^S = Y^S \odot m,
$$

$$
\hat{Y}_{\mathrm{region}} = \hat{Y}^C + \hat{Y}^S.
$$

The final attention output is

$$
Y_{\mathrm{final}} = (1-a)\,Y^G + a\,\hat{Y}_{\mathrm{region}},
$$

where \(a\) is a learnable scalar [2406.01059].

In functional terms, CTS makes the center prompt primarily influence the unmasked region and the surrounding prompt primarily influence the masked region, while preserving a global prompt-conditioned path. This suggests a hybrid mechanism: region-specific alignment for local control, combined with global attention for coherence across the full image.

## 4. Fine-tuning protocol and training configuration

VIP is trained with the standard denoising diffusion mean-squared-error loss,

$$
L = \mathbb{E}_{z_0,\epsilon,t,\text{text\_prompt}}
\bigl[\| \epsilon - \epsilon_\theta(z_t,t;\text{prompt}) \|^2\bigr].
$$

The paper states that no additional regularizers are introduced beyond the usual diffusion framework [2406.01059].

The trainable and frozen components are explicitly separated. The VAE encoder and decoder, the text encoder, and the mask generator are frozen. The entire UNet, including the newly introduced CTS cross-attention blocks, is trainable [2406.01059]. This is the basis for the paper’s efficiency claim: VIP is “just slightly fine-tuned” on top of an off-the-shelf Stable Diffusion model rather than trained from scratch.

Two training regimes are described.

| Regime | Data | Settings |
|---|---|---|
| Specific Fine-Tuning (SFT) | Scenery \(5\text{K}/1\text{K}\), Building \(16\text{K}/1.5\text{K}\), WikiArt \(45.5\text{K}/19.5\text{K}\) | \(192\times192\) image, central \(128\times128\) masked, 30,000 steps, batch size 4, Adam lr \(=5\mathrm{e}{-6}\) |
| Generalized Fine-Tuning (GLT) | 224 automatically annotated Laion2B samples | \(256\times256\) image, center masked to \(128\times128\), 10,000 steps, batch size 4, Adam lr \(=5\mathrm{e}{-6}\) |

DDIM with 50 inference steps is used as the sampler [2406.01059].

The distinction between SFT and GLT is methodologically significant. SFT specializes the model to each benchmark dataset, whereas GLT attempts to produce a more general model from automatically annotated Laion2B samples. The reported results show that dataset-specific adaptation remains important for peak benchmark performance.

## 5. Quantitative performance and ablation results

The principal quantitative comparison is reported using FID on Scenery, Building, and WikiArt. The paper compares VIP with “Neural ODE + copy” and “PQDiff + copy,” as well as several VIP variants [2406.01059].

| Method | Scenery FID↓ | Building FID↓ | WikiArt FID↓ |
|---|---:|---:|---:|
| Neural ODE + copy | 19.35 | 21.01 | 14.83 |
| PQDiff + copy | 20.10 | **19.13** | 7.97 |
| VIP (GLT) | 26.27 | 35.05 | 10.03 |
| VIP (GLT)+copy | 23.44 | 27.16 | 7.33 |
| VIP (SFT) | *14.88* | 27.36 | *3.81* |
| VIP (SFT)+copy | **10.72** | 17.78 | 3.01 |

The paper states that on all three benchmarks, VIP(SFT) achieves the best FID to date, and that Inception Scores likewise meet or exceed prior work [2406.01059]. The strongest reported numbers in the table are obtained by VIP(SFT)+copy on Scenery and WikiArt, and by VIP(SFT)+copy on Building relative to the other VIP variants, while PQDiff + copy remains the boldfaced Building baseline in the reproduced comparison table.

The ablation study on Scenery isolates two elements: the prompt format and the CTS module. The baseline without fine-tuning has FID \(=178.6\) and IS \(=4.82\). Adding `Center:; Surrounding:` prompts reduces FID to \(16.83\) with IS \(=4.03\). Adding CTS further improves FID to \(14.88\) with IS \(=4.12\) [2406.01059]. These results support two claims made in the paper: structured prompts are useful even in unconditional generation, and CTS provides an additional performance gain beyond prompt formatting alone.

A plausible interpretation is that VIP’s gains do not arise from a single modification. Rather, the results indicate a compound effect from prompt structure, region-aware cross-attention, and diffusion fine-tuning on an inpainting backbone.

## 6. Customized outpainting behavior and qualitative characteristics

Beyond benchmark scores, VIP is explicitly designed for customized outpainting. The paper’s qualitative examples use the same center patch while varying the `Surrounding` keywords, including “shirt,” “cap,” “sign,” and “box,” yielding different plausible extrapolations [2406.01059]. This demonstrates the intended form of user control: the visible center remains fixed, while the generated outer region changes according to prompt specification.

The paper also reports that VIP handles irregular masks, stating that with “ground-truth objects at arbitrary shape,” the model still extrapolates seamlessly beyond square borders [2406.01059]. Since the training setup is described with central square masking, this qualitative result suggests some flexibility beyond the exact training geometry.

Figure descriptions in the summary characterize VIP’s outputs as showing “crisper textures” and “better global coherence” than prior unconditional outpainting methods in side-by-side comparisons [2406.01059]. Because this is a qualitative description tied to the figure setup, it should be understood as an observational claim from the paper’s visual comparison rather than an independently quantified metric.

The framework therefore unifies two modes of operation within one system: unconditional outpainting, where plausible surrounding content is hallucinated, and conditional outpainting, where the surrounding content is steered by prompt tokens extracted automatically or modified by a user. This dual functionality is one of the main distinctions emphasized by the authors.

## 7. Position within diffusion-based outpainting research

VIP is presented as a diffusion-based outpainting method that differs from many prior systems in two ways. First, it introduces multimodal prompt extraction to support customization without human labeling. Second, it achieves its results by fine-tuning only the UNet of a Stable Diffusion Inpainting checkpoint while freezing the VAE and text encoder, rather than constructing a model from scratch [2406.01059].

Within the paper’s own comparison, both GAN-based and diffusion-based baselines are considered, including “Neural ODE + copy” and “PQDiff + copy” [2406.01059]. VIP’s reported results place it within the recent trend of adapting large pre-trained generative backbones to editing and completion tasks, but its specific contribution is the coupling of MLLM-derived `Center:/Surrounding:` prompts with mask-guided cross-attention.

More broadly, the work suggests a shift in outpainting from pure extrapolative completion toward controllable extrapolation. Instead of asking only whether the generated border is plausible, VIP also asks whether it can be specified. On the evidence reported in the paper, the answer is affirmative: the framework supports user-driven prompt changes, improves benchmark FID under dataset-specific fine-tuning, and does so with a resource-efficient adaptation of an existing Stable Diffusion inpainting model [2406.01059].

Source: https://www.emergentmind.com/topics/v-i-p