---
title: 'Make-A-Scene: Controllable Image Synthesis'
url: https://www.emergentmind.com/topics/make-a-scene-model
type: topic
---

# Make-A-Scene: Controllable Image Synthesis

The Make-A-Scene model is a scene-controllable text-to-image generation framework that integrates semantic scene representations, human-prior-driven tokenization, and classifier-free guidance within a large autoregressive transformer. This approach enables precise and user-controllable image synthesis, supporting advances in image fidelity, compositional flexibility, and robustness to diverse prompts and editing tasks. The model achieves state-of-the-art Frechet Inception Distance (FID) and strong human evaluation metrics on benchmark datasets, while introducing novel capabilities such as direct scene control and story illustration generation [2203.13131].

## 1. Scene Representation and Encoding

Make-A-Scene models scenes as multi-channel semantic segmentation maps that capture layered human priors about the visual world. Scene representations comprise:

- Panoptic semantic classes ($m_p=133$)
- Human parsing classes ($m_h=20$)
- Face parsing classes ($m_f=5$)
- An additional “edge” channel marking class/instance boundaries

The resulting input is a scene tensor $i_y \in \mathbb{R}^{h_y \times w_y \times m}$ with $m = m_p + m_h + m_f + 1$.

Scene tokenization is performed using a Vector-Quantized VAE (VQ-SEG), where the encoder $E_s$ generates latent codes, which are quantized using a codebook of $K_s$ vectors: for grid location $p$, the quantized code is
$$
t_y[p] = \arg\min_j \| E_s(i_y)_p - e_j \|_2, \quad t_y \in \{1, \ldots, K_s\}^{h' \times w'}
$$
Each $t_y[p]$ is then mapped to a $d$-dimensional embedding before being processed by the transformer.

## 2. Model Architecture and Multimodal Fusion

The generator is a 4B-parameter autoregressive GPT-style transformer with 48 layers, each with 48 self-attention heads and a hidden dimension $d=2560$. Three modalities—text, scene, and image—are tokenized independently and concatenated to form the transformer’s input sequence:
- $n_x = 256$ for text (tokenized via BPE)
- $n_y = 256$ for scene (VQ-SEG)
- $n_z = 1024$ for image (VQ-IMG)

The training pipeline is as follows:
1. Text inputs: $t_x = \operatorname{BPE}(i_x)$
2. Scene: $t_y = \operatorname{VQ-SEG}(i_y)$
3. Image: $t_z = \operatorname{VQ-IMG}(i_z)$
4. Concatenate to $t = [t_x; t_y; t_z]$
5. Input each token embedding $E_\text{tok}(t[p]) + E_\text{pos}(p)$ to the transformer
6. Autoregressively predict $t[p+1]$, optimizing cross-entropy loss.

Modal fusion is performed implicitly: the transformer self-attends over the complete concatenated sequence, learning to integrate information across modalities.

## 3. Human-Prior Tokenization and Specialized Objectives

Tokenization for both scene and image leverages human priors to preserve spatial and semantic detail:

- VQ-IMG (image tokenizer) extends VQGAN with dedicated perceptual losses:
    - **Face-aware VQ**: Perceptual loss over face crops, computed as
      $$
      \mathcal{L}_\text{Face} = \sum_{k=1}^{k_f} \sum_l \alpha_f^l \| \mathrm{FE}^l(\hat{S}(c_f^k)) - \mathrm{FE}^l(c_f^k) \|_2
      $$
      where $\mathrm{FE}^l$ denotes activations in VGGFace2, $c_f^k$ are up to $k_f$ face crops, and $\hat{S}$ denotes the reconstructed crop.

    - **Object-aware VQ**: Perceptual loss over object crops using VGG, with analogous formulation.

    - VQ-IMG loss combines reconstruction, codebook, adversarial, global perceptual, face, and object losses.

- VQ-SEG (scene tokenizer) adds a weighted BCE term emphasizing face-part labels:
  $$
  \mathcal{L}_\text{WBCE} = \alpha_\text{cat} \cdot \mathrm{BCE}(s, \hat{s})
  $$
  with $\alpha_\text{cat} = 20$ for facial parts and $1$ otherwise. Complete VQ-SEG objective is:
  $$
  \mathcal{L}_\text{SEG} = \mathcal{L}_\text{recon} + \mathcal{L}_\text{commit} + \mathcal{L}_\text{codebook} + \mathcal{L}_\text{WBCE}
  $$
  This approach ensures that minor semantic and facial details are preserved throughout tokenization and decoding.

## 4. Classifier-Free Guidance in Autoregressive Generation

The Make-A-Scene transformer is fine-tuned for classifier-free guidance via scheduled unconditional sampling (probability $p_{CF}=0.2$ during the final 30k training iterations, substituting the text stream with padding tokens).

At inference, each autoregressive decoding step computes two logit streams:

- $ \mathrm{logits_{cond}} = T(\text{scene, image so far} \mid \text{text}) $
- $ \mathrm{logits_{uncond}} = T(\text{scene, image so far} \mid \varnothing) $

Fused as:
$$
\mathrm{logits_{cf}} = \mathrm{logits_{uncond}} + \alpha_c \cdot (\mathrm{logits_{cond}} - \mathrm{logits_{uncond}})
$$
with guidance scale $\alpha_c = 5$ (or 3). Sampling proceeds using $\text{softmax}(\mathrm{logits_{cf}})$ and a truncated multinomial over the top 50% logits.

This method brings the controllability of classifier-free diffusion models to the autoregressive transformer paradigm, improving sampling sharpness and alignment with user prompts.

## 5. Empirical Performance and Ablative Analysis

Quantitative evaluation demonstrates improved generation fidelity and human preference:

- 256×256 model: FID $= 7.55$ (unfiltered), $=11.84$ (filtered)
- With scene-based guidance and $\alpha_c$: FID $=4.69$
- Ground-truth FID (COCO validation): $\sim2.47$

Human evaluation over 500 prompts (5 AMT workers per prompt) indicates $\approx92\%$ preference for Make-A-Scene over CogView$_{256}$ across image quality, photorealism, and text alignment dimensions.

Ablative studies reveal the contribution of each component:

| Method Variant                   | FID   | Human Preference (%) |
|----------------------------------|-------|---------------------|
| Base (no scene/no CF)            | 18.01 | --                  |
| +Scene tokens only               | 19.16 | >50                 |
| +Face-aware VQ                   | 14.45 | --                  |
| +Classifier-free guidance (CF)   | 7.55  | 76.8 (quality)      |
| +Object-aware @512               | 8.70  | --                  |
| +Scene at inference              | 4.69  | --                  |

*Values as reported in the source.*

These results establish new benchmarks for text-to-image models at the time of publication [2203.13131].

## 6. Capabilities and Applications

The Make-A-Scene architecture enables multiple generation and editing modalities:

- **Text-only synthesis:** Generates high-fidelity 512×512 images purely from textual descriptors.
- **Out-of-distribution prompt handling:** User-sketched scenes (e.g., “mouse hunting lion”) can direct the transformer to synthesize compositions with rare or previously unseen class combinations.
- **Scene editing:** Segmentation maps can be extracted from existing images, edited (e.g., relabeling sky to sea), and re-rendered into consistent new images.
- **Text editing with anchor scenes:** Scene layout can be fixed; varying the text prompt yields a controlled set of semantically consistent image variants.
- **Story illustration:** Sequentially generates consistent illustrations for narrative text using user-provided scene sketches.

This set of capabilities highlights the model’s explicit and implicit control mechanisms and generalization to creative and task-driven workflows [2203.13131].

## 7. Implementation Specifics

Key implementation parameters include:

- **VQ-SEG:** 600k training iterations, batch size 48, codebook size 1024, class weights for facial parts $\alpha_\text{cat}=20$.
- **VQ-IMG:** 256×256 (800k, batch 192), 512×512 (940k, batch 128), codebook size 8192, variable channel depth.
- **Transformer:** 48 layers, 48 heads, $d_{model}=2560$, $n_x=256$, $n_y=256$, $n_z=1024$, 170k iterations, batch 1024, Adam optimizer with $\beta_1=0.9,\, \beta_2=0.96$, weight decay $4.5\times10^{-4}$, loss ratio (image:text $=7:1$).
- **Classifier-free fine-tuning:** Last 30k iterations, $p_{CF}=0.2$, inference guidance scale $\alpha_c=5$ (or 3).
- **Sampling:** Top 50% logit truncation per step, multinomial sampling.
- **Output resolutions:** Available models for 256×256 and 512×512 outputs.

This architectural configuration supports Make-A-Scene’s state-of-the-art quantitative and qualitative performance, and its novel suite of editing and illustration features [2203.13131].

Source: https://www.emergentmind.com/topics/make-a-scene-model