VBench Video Generation Benchmark

• VBench is a benchmark suite that defines video generation quality using 16 distinct dimensions categorized into temporal and frame-wise aspects.
• It employs both automatic metrics (e.g., DINO, CLIP, RAFT) and rigorous human annotations to ensure high alignment with human perception.
• VBench facilitates detailed diagnostic analysis of text-to-video and image-to-video systems, revealing trade-offs between visual fidelity and dynamic content.

VBench is a comprehensive benchmark suite for evaluating video generative models, designed to decompose the multifaceted concept of "video generation quality" into a precise, hierarchical, and disentangled set of dimensions. It enables model developers and researchers to rigorously assess text-to-video (T2V) and image-to-video (I2V) systems with transparent, human-aligned metrics, supporting both technical development and meaningful progress in the video synthesis field (Huang et al., 2023, &&&1&&&).

1. Decomposition of Video Quality: Dimensions and Formulation

VBench operationalizes video generation quality along 16 distinct dimensions, grouped into two high-level categories: Video Quality and Video-Condition Consistency. Temporal and frame-wise aspects are explicitly disentangled.

Temporal Quality Dimensions:

• Subject Consistency: Assesses stability of subject appearance across frames, measured via DINO feature similarity.

$$S_{\text{subject}} = \frac{1}{T-1} \sum_{t=2}^{T} \frac{1}{2}(\langle d_1, d_t \rangle + \langle d_{t-1}, d_t \rangle)$$

where $d_i$ is the normalized DINO feature of the $i$-th frame.

• Background Consistency: Evaluated using CLIP features averaged across background regions.
• Temporal Flickering: Quantifies frame-to-frame pixel fluctuation using normalized mean absolute error.
• Motion Smoothness: Computed by measuring discrepancies between interpolated and actual frames using frame interpolation models.
• Dynamic Degree: Percentage of frames exhibiting nontrivial motion, quantified via RAFT optical flow.

Frame-wise Quality Dimensions:

• Aesthetic Quality: Scored by predictors (e.g., the LAION aesthetic model).
• Imaging Quality: Rates low-level artifacts using MUSIQ (trained on SPAQ).

Video-Condition Consistency:

• Semantics: Object Class (via GRiT), Multiple Objects, Human Action (via UMT classifier), Color, Spatial Relationship, Scene (via Tag2Text).
• Style: Appearance Style (CLIP feature similarity to style prompt), Temporal Style (ViCLIP for motion style).
• Overall Consistency: Integrative metric using ViCLIP to reflect joint semantic and stylistic alignment.

This decomposition reveals not only aggregate quality, but dimension-specific strengths, weaknesses, and trade-offs for any given generative model.

2. Human Alignment and Annotation Protocol

A foundational property of VBench is its systematic alignment with human perception. For each dimension, generated videos from multiple models (given the same prompt) are compared by human annotators through a pairwise judgment interface, focusing exclusively on one dimension at a time. These annotations are aggregated into win ratios that are directly compared against the automatic numerical metrics produced by VBench.

Correlations between human preference ratios and metric scores routinely surpass 90%, indicating that VBench metrics robustly capture aspects of quality that matter to human viewers. The annotation protocol involves rigorous pre-labeling trials, clear instruction sets with examples, and iterative rounds of consistency checking to ensure high-quality ground truth data.

3. Model Performance Insights and Diagnostic Utility

Evaluations across the 16 dimensions enable fine-grained diagnostic analysis:

• Models that maximize subject/background consistency (e.g., LaVie) may minimize dynamic degree, "cheating" by generating quasi-static content.
• Trade-offs emerge: increased dynamic content can lead to reduced temporal consistency, reflecting an intrinsic tension between visual fidelity and dynamism.
• Category-specific analysis (e.g., Animals, Humans, Vehicles) uncovers differential model strengths across content types.
• Comparative analysis with text-to-image systems shows that while frame-wise quality transfers, video generative models often lag in compositional and temporal aspects.

These insights inform both architectural improvements and training strategies for new generative models.

4. Open-Source Infrastructure and Community Collaboration

VBench provides a fully open-source evaluation framework comprising:

• Evaluation Dimension Suite: Metric definitions, grouping logic, and implementation details (including formulas).
• Prompt Suite: Stratified cases for each dimension across content categories.
• Generated Video Datasets: Comprehensive collections from varied models and prompts.
• Human Annotation Datasets: Raw pairwise comparisons, win ratios, and processing scripts.
• Evaluation Codebase: Modular routines, configuration guides, and labeling/inference pipelines.

This infrastructure underpins reproducibility, facilitates fair model comparison, and enables extension to new modalities (e.g., video editing, image-to-video). Community involvement is actively encouraged.

5. Extension: VBench++ and Trustworthiness Evaluation

VBench++ builds on VBench by supporting both T2V and I2V workflows. Notably, it introduces the "Image Suite" with an adaptive aspect ratio cropping scheme, enabling fair comparison across I2V models irrespective of default image shapes or resolutions. For I2V, high-resolution 4K+ images from Pexels and Pixabay are systematically processed to retain essential content.

Trustworthiness dimensions—such as gender and skin tone bias (via face detectors and Fitzpatrick scaling), and safety (via classifiers like NudeNet and Q16)—are additionally evaluated. For example, gender bias is measured by the formula:

$$\text{bias score} = 1 - \left|\left(\frac{n_{\text{male}}}{N}, \frac{n_{\text{female}}}{N}\right) - \left(\frac{1}{2}, \frac{1}{2}\right)\right|_1$$

where $n_{\text{male}}$/$n_{\text{female}}$ are the respective detected counts.

Safety checking aggregates signals from multiple detectors to flag unsafe or offensive content.

VBench++ is accompanied by a public leaderboard (e.g., on Hugging Face Spaces) that is continuously updated as new models are evaluated.

6. Future Directions and Impact

VBench is positioned as an evolving tool for steering future developments in video generation:

• Supports stacking compositionality, temporal smoothness, and dynamic content for next-generation model design.
• Its feedback loops facilitate research on compositionality, object synthesis, and spatial reasoning—identified as persistent challenges by VBench diagnostics.
• Widely adopted in recent model papers (e.g., ContentV, STIV, Owl-1, MotionAgent), VBench’s methodology is integrated into state-of-the-art evaluation pipelines.
• Extension towards emerging domains (video editing, multi-modal fusion) is anticipated, leveraging the modularity of its evaluation dimensions.

In summary, VBench offers a rigorously validated, multi-dimensional, and human-aligned framework for quantifying and diagnosing video generative model quality. Its analytic granularity, open-source transparency, and alignment with community standards position it as a foundational benchmark for both current research and future advancements in video synthesis (Huang et al., 2023, Huang et al., 20 Nov 2024).