VideoCrafter2: Text-to-Video Framework

• VideoCrafter2 is a text-to-video generation framework that overcomes limited video data by disentangling appearance and motion through a two-stage training process.
• It employs a factorized 3D U-Net architecture with spatial modules (inherited from Stable Diffusion) and lightweight temporal modules to ensure high frame fidelity and motion consistency.
• Experimental evaluations show competitive visual quality and temporal coherence, with human studies favoring its outputs over previous open-source models.

VideoCrafter2 is a text-to-video generation framework designed to overcome the limitations imposed by the lack of large, high-quality video datasets available for open-source research. Unlike commercial models such as Gen-2 or Pika Labs, which leverage massive corpora of high-resolution clips, VideoCrafter2 achieves high visual fidelity and temporal coherence by disentangling the learning of appearance and motion and strategically exploiting separate data sources—low-quality videos and synthesized high-resolution still images—within a two-stage training and finetuning scheme. This approach enables the model to bridge the gap to commercial video diffusion systems without direct access to HD video corpora (Chen et al., 2024).

1. Architectural Foundation

At its core, VideoCrafter2 adopts a “factorized 3D U-Net” architecture prevalent in contemporaneous video diffusion models such as MagicVideo, ModelScope T2V, and its predecessor, VideoCrafter1. The architecture comprises:

• Spatial modules $\theta_S$: 2D per-frame denoising blocks inherited from Stable Diffusion, implementing cross-attention, spatial convolutions, and various up-/down-sampling operations.
• Temporal modules $\theta_T$: Lightweight 1D convolutions, inserted between spatial blocks, propagate and smooth feature maps across the temporal axis (frames).

The inference procedure alternates between applying $\theta_S$ to each frame and $\theta_T$ across the temporal ($T$) axis on latent sequences $$x_t \in \mathbb{R}^{B \times T \times C \times H \times W}$$, where $B$ is batch size, $T$ time steps, $C$ channels, and $H \times W$ spatial dimensions. This parallel yet coupled structure enables joint modeling of appearance and motion through latent diffusion.

2. Training Paradigm and Distribution Shifts

VideoCrafter2’s training regimen addresses the distribution gap between low-quality video and high-quality imagery through a two-phase process:

• Stage A: Full video pretraining
  • Data: WebVid-10M (approximately 10 million low-resolution video clips) and LAION-COCO-600M (to mitigate concept forgetting).
  • Objective: Standard latent-diffusion denoising loss:

  $$L_\mathrm{diffusion}(\theta_T, \theta_S) = \mathbb{E}_{x_0, \epsilon, t}\left[\| \epsilon - \epsilon_\theta(x_t, t, c) \|^2\right]$$

  Both $\theta_S$ and $\theta_T$ are updated, ensuring that spatial modules also absorb the appearance statistics of video data, increasing the coupling between motion and appearance representations.

• Stage B: Spatial finetuning with high-resolution still images
  • Data: A synthetic, high-diversity set of four million high-resolution images (JDB, generated by Midjourney) with conceptually rich prompts.
  • Objective: Denoising loss applied only to $\theta_S$, with $\theta_T$ frozen:

  $$L_\mathrm{ft}(\theta_S) = \mathbb{E}_{x_0^I, \epsilon, t}\left[\| \epsilon - \epsilon_{(\theta_T\,\mathrm{fixed}, \theta_S)}(x_t^I, t, c) \|^2\right]$$

  This selectively shifts the spatial modules from replicating “WebVid look” to a “JDB look” while preserving temporal coherence.

Expressed formally, given a fully pretrained model $M_F(\theta_T, \theta_S)$, finetuning yields $$M^*(\theta_T, \theta_S + \Delta\theta_S) \leftarrow \arg\min_{\Delta\theta_S} L_\mathrm{ft}(\theta_S + \Delta\theta_S)$$, with $\theta_T$ fixed.

3. Finetuning Strategy: Effectiveness and Variants

Experimental analysis reveals that direct finetuning of the full $\theta_S$ (“F-Spa-DIR”) substantially enhances visual quality—including removal of visible artifacts, increased sharpness, and improved aesthetic evaluations—while inducing nearly zero degradation of temporal dynamics. In contrast, modifying $\theta_T$ or jointly adapting both $\theta_S$ and $\theta_T$ leads to suboptimal tradeoffs, typically reducing motion quality or introducing flicker. Low-rank adapters (LoRA) were evaluated but were found insufficiently expressive compared to direct finetuning of the spatial components. The algorithmic formulation proceeds as:

for each iteration: sample prompt p and image x0_I sample t ~ Uniform({1,..., T}), ε ~ N(0, I) x_t^I = sqrt(α_t) * x0_I + sqrt(1 – α_t) * ε ε_θ = ε_(θ_T fixed, θ_S)(x_t^I, t, p) θ_S ← θ_S – η * grad_θ_S [||ε – ε_θ||^2] end

4. Empirical Evaluation and Benchmarks

Performance evaluation uses EvalCrafter’s four principal scores—Visual Quality, Text-Video Alignment, Motion Quality, and Temporal Consistency—each calibrated to human preferences. Supplementary DOVER aesthetic/technical measures and a user preference study were also conducted.

Method	Vis. Q.	Text-Vid	Motion	Temp. Cons.
Gen-2 (comm.)	67.35	52.30	62.53	69.71
Pika Labs (comm.)	63.52	54.11	57.74	69.35
VideoCrafter1	61.64	66.76	56.06	60.36
Show-1	52.19	62.07	53.74	60.83
AnimateDiff	58.89	74.79	51.38	56.61
VideoCrafter2	63.28	64.67	53.95	62.02

• VideoCrafter2 achieves Visual Quality scores matching Pika Labs and surpasses VideoCrafter1, with Motion and Temporal Consistency competitive relative to Show-1 and AnimateDiff, despite not using any high-resolution training video.
• In human pairwise testing on 50 prompts, VideoCrafter2 was preferred over AnimateDiff for visual quality 69% (motion 64%), over Show-1 for visual quality 82% (motion 59%), and over VideoCrafter1 for visual/motion at 61%/63%.
• Finetuning using the JDB image set, as opposed to LAION-Aesthetics, yields superior compositional fidelity, enabling better conceptual integration of diverse prompt elements (“blue pig,” “scooter,” “sunny lakeside”).

5. Disentanglement of Appearance and Motion

VideoCrafter2 demonstrates an effective data-level disentanglement of appearance and motion representations. This is accomplished by leveraging high-quality 2D image data for spatial module finetuning and low-quality video to preserve temporal dynamics. The framework eliminates the necessity of high-resolution video datasets, harnesses large-scale image generation for frame-level fidelity, and ensures retention of natural motion derived from WebVid. This disentanglement closes much of the quality gap to leading commercial models.

6. Limitations and Prospective Directions

Notable limitations include a minor residual performance gap on the most challenging commercial motion benchmarks and dependence on the diversity coverage of synthetic image prompts—uncommon “fantastical” prompts may yield artifacts. A further limitation is the potential adverse effect of domain mismatch between optimized spatial and temporal modules if their training distributions diverge excessively.

Possible future extensions may involve multi-stage cascaded architectures, deployment of lightweight temporal adapters for long-form video coherence, and adversarial finetuning methods to enhance the congruence between high-resolution appearance and learned video motion dynamics (Chen et al., 2024).