---
title: 'Phenaki: Text-Conditioned Video Synthesis'
url: https://www.emergentmind.com/topics/phenaki
type: topic
---

# Phenaki: Text-Conditioned Video Synthesis

Phenaki is a generative modeling system for producing open-domain videos of variable length, directly conditioned on sequences of textual prompts. It introduces architectural and algorithmic innovations in video tokenization and text-to-video synthesis that enable spatio-temporally coherent generation over arbitrary durations and evolving prompts. Phenaki is distinguished by its causal, temporally-factorized video tokenizer and a bidirectional masked Transformer, integral for efficient and generalizable open-domain video generation from text [2210.02399].

## 1. Model Architecture and Generation Pipeline

Phenaki's generative process bifurcates into two principal components:

1. **Discrete Video Tokenizer (C):** An input video is encoded into a compact, variable-length 3D tensor of discrete codebook indices via a novel causal transformer-based tokenization scheme. The decoder reconstructs RGB video from these discrete tokens.

2. **Bidirectional Masked Transformer (MaskGIT):** A 24-layer Transformer predicts sequences of video tokens conditioned on text embeddings derived from a frozen T5X encoder. Generation proceeds by iteratively sampling masked tokens, using classifier-free guidance for improved diversity and control.

The generation workflow operates as follows. At inference, the MaskGIT model predicts a full grid of masked video tokens over tens of refinement steps. The C-decoder transforms these discrete tokens into video frames. For long or multi-scene generation, a segmental approach is used: after generating each segment, the last $K$ frames are re-encoded to provide temporal context for subsequent prompts, enabling seamless transitions and "visual stories" with evolving textual guidance.

## 2. Discrete Video Tokenizer: Architecture, Losses, and Temporal Factorization

The C tokenizer processes an input $\mathbf{x}\in\mathbb{R}^{(T_x+1)\times H_x\times W_x\times C_x}$, e.g., 11 frames at $128\times128\times 3$, via patchification and stacked transformer blocks:

- **Patchification:** The first frame is decomposed into $8\times8\times3$ 2D patches; subsequent frames are decomposed into $2\times8\times8\times3$ spatio-temporal patches.
- **Transformer Blocks:** An initial spatial transformer block (4 layers, all-to-all attention) operates at each time index independently, followed by a temporal block (4 layers, causal self-attention in time) applied per patch spatial location. Causality is enforced by masking future frames in the temporal attention matrix.
- **Vector Quantization:** Output vectors $z(\tau,j)\in\mathbb{R}^{d_z}$ are quantized to a codebook ($|E|=8192$ entries), yielding discrete indices for each patch and time position.

**Loss Function:** The tokenizer is trained with a composite objective:
\[
\mathcal{L}_{C} = \mathcal{L}_{VQ} + \lambda_2 \|x-\hat{x}\|_2^2 + \lambda_{IP}\mathcal{L}_{IP}(x,\hat{x}) + \lambda_{VP}\mathcal{L}_{VP}(x,\hat{x}) + \lambda_{Adv}\mathcal{L}_{Adv}(\hat{x})
\]
where $\mathcal{L}_{VQ}$ is the vector-quantization loss, $\mathcal{L}_{IP}$ and $\mathcal{L}_{VP}$ are perceptual losses via VGG and I3D, and $\mathcal{L}_{Adv}$ is an adversarial loss with StyleGAN-style discriminator. Default weighting is $(\lambda_2, \lambda_{IP}, \lambda_{VP}, \lambda_{Adv}) = (1.0, 0.1, 1.0, 0.1)$. The temporal factorization with causal self-attention ensures the tokenizer supports variable-length video encoding and decoding.

## 3. Text Conditioning and MaskGIT Modeling

Text prompts are tokenized using a frozen T5X encoder (vocabulary size ≈32,000, output embedding length $L \approx 64$, dimension 1024). The bidirectional masked Transformer (MaskGIT-style) receives these embeddings via learned cross-attention inserted after standard self-attention at each layer.

Masked visual token modeling is used for training: for each example, a random mask ratio $\gamma_i\in[0,1]$ is sampled, and $\lceil \gamma_i N \rceil$ tokens are masked (out of $N$ total). The transformer minimizes the negative log-likelihood of the true tokens at masked positions, conditioned on visible positions and text:
\[
\mathcal{L}_{mask} = -\sum_{i: m_i=1} \log p(a_i \mid a_{\bar{M}},\mathrm{text})
\]
where $a_i$ is the true code index. Approximately 10% of steps drop the text condition, facilitating classifier-free guidance at inference.

Inference proceeds via parallel denoising iterations: at each step, logits for masked tokens are predicted; the highest-confidence predictions are kept and the remainder are re-masked, iteratively refining the grid until all tokens are filled. This produces improved sample efficiency over autoregressive decoding and, when paired with causal tokenization, allows for efficient long-form or multi-scene generation.

## 4. Data, Joint Training, and Style Transfer

Phenaki is trained on approximately 15 million video-text clips at 8 fps and two large image–text datasets (WebVid+LAION-400M, collectively ≈450M pairs). Each batch contains about 80% video and 10% from each image pool (batch size 512). 

A dynamic masking strategy is applied: image data are treated as 1-frame “videos,” and only their spatial tokens are masked and predicted, skipping temporal positions. This joint training enables strong transfer of image-derived styles (e.g., “pencil sketch”) to video synthesis, enhancing visual diversity and controllability in open-domain prompts. Mixing ratios impact trade-offs between temporal dynamics (FVD) and appearance fidelity (FID_img).

## 5. Variable-Length Generation and Story Prompting

The causal time-factorized tokenizer enables encoding any number of context frames. To build long or multi-scene outputs, the pipeline alternates prompt-conditioned MaskGIT sampling and re-encoding of recent frames for continuity. The process:

1. Prompt $p_0$ → sample $L_0$ frames.
2. After $K$ frames, switch to prompt $p_1$.
3. Re-encode last $K$ frames with $C$, append new prompt, and generate $L_1$ frames.

This enables temporally coherent visual stories, including smooth morphing between distinct prompt semantics, with transitions such as "teddy bear → panda." Finite context (fixed $K$) may affect very long-range consistency.

## 6. Evaluation and Benchmarks

Quantitative evaluation is conducted using established metrics:

- **Video Reconstruction (Quantization) on Moments-In-Time:** C tokenizer achieves FVD=65.8 (vs. 166 for per-frame ViT-VQGAN) with 1536 tokens per clip, compared to 2560 tokens for baselines. There is a small penalty in FID but substantial gain in spatio-temporal consistency and token efficiency.
- **Text-to-Video (Kinetics-400, zero-shot):** Phenaki achieves FID_vid ≈ 3.8, outperforming NUWA (7.0), and FID_img ≈ 37 for open-domain, zero-shot generation.
- **Video Prediction (BAIR Robot Push, Kinetics-600):** Despite not being specialized for this task, Phenaki is competitive with state-of-the-art, as measured by FVD.
- **Text-to-Image:** CLIP score and FID on held-out LAION images demonstrate transferable style learning from the image pools.

Ablation studies reveal trade-offs: pure video training yields lowest FVD but highest FID_img; mixed batches (e.g., 80/20) improve appearance but slightly degrade temporal metrics.

## 7. Limitations and Prospective Directions

Phenaki's performance on visual fidelity and coherence currently trails the photorealistic capabilities of static image generators, with observable artifacts in complex or high-motion scenes. The approach is constrained by a finite context window $K$, leading to gradual degradation in very long temporal continuity. Higher resolutions and frame rates incur substantial computational costs. The model's reliance on large, unfiltered web-scale corpora introduces bias and content safety issues.

Potential enhancements include the development of more powerful long-range temporal models (e.g., hierarchical or memory-augmented transformers), improved dataset curation and safety mechanisms, and extensions to audio-video, interactive, or key-frame-controllable synthesis. These avenues may address both qualitative limitations and application scalability [2210.02399].

Source: https://www.emergentmind.com/topics/phenaki