Class-Conditional Video Generation

• Class-conditional video generation is the process of synthesizing coherent video sequences by conditioning on explicit class labels such as action types or object identities.
• It employs diverse models—ranging from VAE-based and GAN-based to diffusion and keypoint-driven approaches—to ensure that both static and dynamic video elements align with the target class.
• Evaluation using metrics like Fréchet Video Distance (FVD) confirms that these methods enhance visual fidelity and class consistency, enabling applications in video prediction and synthetic dataset creation.

Class-conditional video generation is the controlled synthesis of temporally coherent video sequences conditioned on explicit class information, such as action categories, object identities, or semantic attributes. This paradigm enables generative models to output videos aligned with designated class labels, supporting tasks such as image-to-video translation, video prediction, and zero-shot composition of unseen class combinations. Distinct from unconditional generative models, class-conditional frameworks explicitly integrate semantic control signals into the generative process, yielding outputs that respect both temporal structure and class specificity.

1. Foundational Principles and Problem Formulation

Class-conditional video generation extends the conditional generative modeling framework to the temporal domain, requiring a model to synthesize video sequences $X_{1:T} = \{x_1, x_2, ..., x_T\}$ given a class label $c$ or attribute vector $a$. The generative objective is to model $p(X_{1:T} | c)$ such that the synthesized video exhibits the visual and dynamic characteristics associated with the target class.

Two main structural properties are essential in this context:

• Temporally Invariant Attributes: Persistent content such as subject identity or category.
• Dynamic Content: The evolving motion or scene dynamics characteristic of the class.

Models are expected to factorize temporal consistency and class-conditional semantics, and to allow attribute control both at training and at test time (He et al., 2018).

2. Model Architectures and Conditioning Mechanisms

Class-conditional video generation models span a range of architectural paradigms:

(a) Latent Variable Models with Holistic Attribute Control

Probabilistic models such as conditional VAEs assign global attribute vectors (e.g., class labels) that guide the evolution of temporal latents $z_{1:T}$ and influence every frame decoder and prior module. Conditioning is achieved by embedding the class attribute $a$ and injecting it into (i) the encoder, (ii) the initial latent prior $p(z_1|a)$, and (iii) the temporal prior $p(z_t|z_{t-1}, a)$ (He et al., 2018).

(b) Disentangled GANs with Explicit Control Streams

Conditional MoCoGAN decomposes the video generation process into a static content stream ($z_c$, $y_c$) and a dynamic motion stream ($c$0, $c$1). Class labels for content and motion are concatenated directly ("hard-wired") to latent vectors for each stream, with GRUs generating temporally-evolving motion, and deep deconvolutional networks synthesizing individual frames (Kimura et al., 2021).

(c) Conditional Diffusion Models with Masking and Multi-tasking

Diffusion-based models (e.g., MCVD) treat class or context frames as conditioning information injected via concatenation or specialized normalization (e.g., SPATIN). Random masking of past/future frames during training enables the network to learn prediction, unconditional generation, and interpolation jointly. Conditioning signal maskings via Bernoulli variables result in a single network solving multiple video tasks under a unified formulation (Voleti, 2023). GenRec implements class conditioning through classifier guidance, where a recognition head is trained alongside the diffusion backbone, and the class label is enforced at sampling time using the gradient of the classifier’s log-probability (“classifier guidance”) (Weng et al., 2024).

(d) Structured Keypoint-Based Conditional Models

Keypoint-based models learn unsupervised spatial landmarks, and synthesize motion trajectories via class-conditional cVAE models. The detected keypoints serve as pseudo-labels for the motion generator, with class labels embedded and concatenated or FiLM-modulated into the temporal sequence generator. The final video frames are synthesized by translating the initial image through the predicted keypoint sequences (Kim et al., 2019).

(e) Hybrid Architectures (Latent Flow Diffusion Models)

LFDM decouples spatial content and temporal motion. A latent flow autoencoder learns low-dimensional video representations and flow predictors, while a conditional 3D U-Net diffusion model generates temporally-coherent flow sequences conditioned on the class label. These flows warp the input image in latent space, producing class-consistent videos (Ni et al., 2023).

3. Mathematical Formulations and Training Objectives

Core mathematical principles are shared across models, adjusted for paradigm and conditioning:

• VAE-based approaches maximize a class-conditional ELBO, incorporating class control into both the inference and generative processes:

$c$2

(He et al., 2018).

• GAN-based frameworks employ conditional adversarial losses:

$c$3

with explicit injection of content and motion class labels into both discriminators and generators (Kimura et al., 2021).

• Diffusion-based models are trained to minimize expected noise prediction error under class-conditional masking:

$c$4

For class conditioning, classifier guidance during sampling modifies the denoiser’s output by backpropagating gradients from a classification head (Voleti, 2023, Weng et al., 2024).

• Keypoint-based pipelines combine perceptual or adversarial reconstruction losses for the image translator and variational plus adversarial sequence losses for the motion generator (Kim et al., 2019).

4. Evaluation Protocols and Quantitative Performance

Standard evaluation leverages both perceptual quality and class-consistency metrics. Key quantitative results include:

Method	Dataset	Metric	Representative Result	Source
MCVD	Cityscapes 128×128	FVD (2p→28f)	141 vs. prior SOTA ~418	(Voleti, 2023)
LFDM	MUG 128×128	FVD	32 (vs 108–171 for baselines)	(Ni et al., 2023)
GenRec	SSV2, EK-100	FVD (class-conditioned)	46.5 (SSV2), 49.3 (EK-100)	(Weng et al., 2024)
Keypoint-VAE	PennAction	Recognition Accuracy	≈ 87% (vs. 75% next best)	(Kim et al., 2019)
Holistic-Attr	Moving MNIST	FID	42.3 vs 58.7 (uncond’l)	(He et al., 2018)

Fréchet Video Distance (FVD) is the primary metric, assessing visual fidelity and temporal consistency. Additional metrics include Inception Score, diversity scores, and, where appropriate, class-consistency or recognition accuracy on generated videos (He et al., 2018, Kim et al., 2019, Voleti, 2023, Ni et al., 2023, Weng et al., 2024). Human evaluations (AMT) and action-recognition probes further validate semantic alignment (Kim et al., 2019).

5. Specialized Conditioning Strategies and Zero-Shot Control

Distinctive conditioning approaches have been designed to maximize semantic control:

• Holistic Attribute Injection: Attributes are encoded and injected into all relevant latent, prior, and decoder modules, ensuring global class consistency (He et al., 2018).
• Disentangled Streams: Explicit segregation of content and motion streams, with class conditioning applied independently, enables zero-shot recombination—allowing the synthesis of unseen content-motion pairs at test time (Kimura et al., 2021).
• Masked-Frame Conditioning and SPATIN: MCVD employs random Bernoulli masking of past/future frame blocks, facilitating multi-tasking (prediction, interpolation, unconditional generation) and robust context-aware modeling (Voleti, 2023).
• Classifier Guidance: GenRec leverages classifier gradients to steer the reverse diffusion process toward the target class, without explicit label injection at training time (Weng et al., 2024).
• Class-Conditioned Keypoint Trajectories: Motion generators receive one-hot or embedded class labels, enabling action-correlated motion synthesis even when keypoint detectors remain unsupervised and agnostic to class (Kim et al., 2019).

6. Domain Adaptation, Zero-Shot, and Limitations

Models such as LFDM demonstrate domain adaptation by fine-tuning only the spatial decoder, enabling transfer to new image domains without retraining the full generative pipeline (Ni et al., 2023). Conditional MoCoGAN achieves zero-shot video generation by independently conditioning motion and content streams, supporting novel class compositions absent during training (Kimura et al., 2021).

Noted limitations encompass:

• Reliance on one-hot or non-semantic class encodings constraining scalability and expressivity (Kimura et al., 2021).
• Keypoint-based methods may struggle with multi-object or ambiguous scenarios, and are sensitive to fixed-camera assumptions (Kim et al., 2019).
• Conditioning mechanisms that do not fully disentangle class semantics may result in degraded FID or partial class leakage (Kimura et al., 2021).

Emergent directions include deploying semantic embeddings (e.g., word2vec), auxiliary class consistency losses, self-attention and 3D convolutional architectures for higher-resolution video, and integrating domain adaptation more deeply into training.

7. Impact and Outlook

Class-conditional video generation has yielded state-of-the-art results on numerous benchmarks, enabling controlled synthesis, improved temporal consistency, and flexible adaptation to diverse scenarios. Applications span video prediction, human action synthesis, facial expression translation, and synthetic dataset creation. Recent architectures unify generation and recognition, suggesting increasingly multi-functional models (Weng et al., 2024). Robustness to limited visual input and the ability to generalize compositionally to novel class configurations remain active areas of investigation.

Progress continues toward more efficient, scalable, and semantically expressive conditioning mechanisms, with an emphasis on modular architectures, representation disentanglement, and principled multi-task learning (Voleti, 2023, Ni et al., 2023, Weng et al., 2024).