EvalVerse: Pipeline-Aware and Expert-Calibrated Benchmarking for Professional Cinematic Video Generation

Abstract: The rapid evolution of generative video foundation models has propelled the field toward professional-grade cinematic synthesis. To achieve such demanding quality, the community transitions towards Reinforcement Learning (RL) and agentic workflows. However, reliable evaluation has emerged as a critical bottleneck. Existing benchmarks predominantly evaluate ''whether it is right'' (basic prompt-following) while fundamentally neglecting ''whether it is good'' (cinematic quality, acting, and aesthetics). Furthermore, current automated metrics lack the domain-specific rigor required to provide trustworthy signals, creating a severe credibility gap between human aesthetic perception and machine scoring. To bridge this gap, we introduce EvalVerse, a comprehensive, pipeline-aware, and expert-calibrated evaluation framework. We treat video generation assessment not merely as an engineering task, but as a core scientific problem: the systematic digitization of subjective cinematic expertise. First, we organize domain knowledge into an evaluation taxonomy aligned with the professional filmmaking workflow (pre-production, production, and post-production). Second, we distill human expert judgments into a curated dataset with large-scale human annotations. Third, we inject this knowledge into Vision-LLMs (VLMs) through an expert-calibrated fine-tuning strategy, enabling the VLM to perform explicit Chain-of-Thought reasoning. Compared to previous works, EvalVerse not only retains compatibility with foundational ''rightness'' metrics, but also significantly expands the criteria to ''goodness'' and broaden the task coverage to complex multi-shot sequencing and audio-visual integration. Consequently, by providing granular diagnostic signals, EvalVerse transcends a static leaderboard and establishes a fundamental infrastructure for future work, such as reward models and evaluator agent.

• The paper introduces a pipeline-aware, expert-calibrated evaluation framework that maps generative video synthesis to professional cinematic workflows.
• It employs a hierarchical taxonomy and automated real-to-gen sampling to ensure balanced, multidimensional assessments across narrative, visual, and multimodal aspects.
• Benchmark analysis and expert-machine calibration reveal enhanced interpretability and performance over conventional, holistic evaluation metrics.

EvalVerse: Pipeline-Aware and Expert-Calibrated Benchmarking for Professional Cinematic Video Generation

Motivation and Framework Design

The proliferation of generative video foundation models has created the need for rigorous evaluation protocols that transcend basic prompt alignment and address professional-grade cinematic synthesis. Contemporary benchmarks inadequately differentiate between "rightness" (prompt-following) and "goodness" (cinematic excellence, aesthetics, narrative, and multimodal synchronization). Further, generic VLM scores lack domain-specific credibility, failing to capture nuanced, subjective criteria embedded in professional filmmaking. EvalVerse introduces a pipeline-aware, expert-calibrated evaluation infrastructure that formalizes subjective cinematic expertise into computable metrics through a five-stage framework:

Figure 1: EvalVerse systematizes subjective cinematic expertise into a scalable expert-calibrated evaluation pipeline comprising taxonomy establishment, dataset curation, expert-machine calibration, and broad application.

Hierarchical Taxonomy for Cinematic Evaluation

The core innovation is the hierarchical, pipeline-aware taxonomy mirroring the traditional filmmaking workflow. Instead of treating AI-driven video generation as discrete subprocesses, EvalVerse maps the multimodal output onto professional pre-production, production, and post-production stages, encompassing 7 cinematic aspects, 18 main dimensions, 45 sub-dimensions, and 196 granular rationales.

Figure 2: EvalVerse’s evaluation taxonomy mirrors professional filmmaking pipelines, enabling structured multi-stage assessment.

• Pre-Production: Focuses on visual concept design, ensuring identity and logical consistency for characters and environments.
• Production: Assesses acting (asset consistency, physical/narrative validity), cinematography (composition, lensing, pacing), aesthetics (technical fidelity and artistic rendering), and affectivity (emotional arc and visual-emotional synergy).
• Post-Production: Evaluates the logic and rhythm of multi-shot sequences and audio-visual integration, including vocal and soundscape fidelity.

The taxonomy enables interpretable diagnostic probing, facilitating expert-level distinctions beyond holistic metrics.

Dataset Curation: Real-to-Gen Sampling Pipeline

EvalVerse’s data engine constructs realistic "Real-to-Gen" test pairs using structured annotation of million-scale cinematic databases. Industrial-grade metadata extraction (camera parameters, asset data, environments) is manually verified and proportionally sampled across cinematic dimensions, thereby ensuring industry-representative distribution and eliminating stochastic bias prevalent in prompt-based benchmarks.

Figure 3: Automated pipeline for dataset annotation and balanced test-pair construction, supporting full-modality evaluation.

• Automated curation and annotation pipelines yield structured JSON metadata verified by professional operators.
• Sampling protocols maintain coverage across narrative, visual, and multimodal aspects.
• Test pairs adapt to both reference-driven and text-driven generation, leveraging high-fidelity images and depth references.

Human and Machine Evaluator Calibration

EvalVerse bridges the methodological gap by aligning expert judgment with algorithmic scoring via a progressive human-machine calibration mechanism:

• Professional Operator Extraction: Specialized extractors (e.g., DINOv2, YOLO, SyncNet, Whisper) provide deterministic evidence (identity, semantic anchors, AV sync, emotion recognition).
• Expert-Guided CoT Reasoning: Fine-tuned VLMs generate explicit Chain-of-Thought (CoT) rationales; self-reflection and context-aware gating mechanisms mitigate hallucinations and handle abstract or narrative-dependent metrics.
• Two-Stage VLM Fine-Tuning: Preference alignment via pairwise comparisons (Bradley-Terry ranking loss) followed by score calibration on pointwise expert-annotated datasets installs CoT reasoning and absolute metric interpretation.

Benchmark Analysis: Inter-Model and Dimension Performance

Comprehensive benchmarking across closed-source and open-source models demonstrates clear hierarchical capability distributions:

Figure 4: Aggregate performance of major video generation models across EvalVerse’s main dimensions.

• Seedance 2.0 achieves superior scores across all facets (soundscape fidelity, identity, camera, and perceptual quality), leading both text-to-video (T2V) and reference-to-video (R2V) tasks.
• Kling-v3-Omni and Happy Horse 1.0 exhibit stable performance in aesthetics, cinematography, and multi-shot organization, but weaker in affectivity and sound.
• HoloCine and MultiShotMaster specialize in multi-shot narrative but lag in acting and rendering.

  Figure 5: Fine-grained analysis of model performance in the Text-to-Video (T2V) scenario, showing dimension-wise strengths.

  

  Figure 6: Fine-grained analysis in the Reference-to-Video (R2V) scenario, reflecting model strengths in multi-modal preservation and post-production dimensions.

Expert-Machine Consistency: Robust Alignment

EvalVerse demonstrates robust human-machine alignment across granular win-ratio and correlation analyses:

Figure 7: Scatter plots of expert versus machine win ratios per model and dimension, confirming strong alignment (linear fits, Pearson’s correlation) across all dimensions.

• Pixel-grounded criteria (lighting, chromaticity, asset identity) achieve high linear and rank correlation (SRCC/PLCC: 0.75–0.84).
• Abstract dimensions (multi-shot rhythm, affectivity progression, soundscape fidelity) see the highest agreement when parameter-level SFT calibration is employed.
• CoT enables interpretable reasoning for perceptually tractable metrics, while SFT covers subjective, temporally entangled dimensions.

Bold claim: EvalVerse conclusively digitizes subjective cinematic criteria into dense, expert-aligned reward vectors, outperforming prior benchmarks on interpretability, scope, and validity.

Implications and Future Directions

EvalVerse establishes the missing infrastructure for scalable cinematic quality assessment in generative video models. By providing diagnostic signals rather than static leaderboards, it unlocks expert-level reward modeling for RL fine-tuning and enables evaluation agents for autonomous workflows. Practically, this catalyzes the evolution of generative AI from passive image generators to professional-grade virtual directors.

Theoretical implications include:

• Formalization of subjective, multi-stage evaluation as a computable scientific problem.
• Demonstrated synergy between CoT reasoning and SFT parameter alignment.
• The pipeline-aware evaluation paradigm is extensible to other generative modalities (music, animation, interactive narratives) requiring hierarchical, domain-specific criteria and expert calibration.

Future work will address current limitations:

• Enhanced temporal modeling and continuous context tracking via advanced VLM architectures.
• Extension to long-form narrative evaluation (hour-scale stories, episodic content) with memory mechanisms ([Wu2025memorysurvey], [Zhang2024memorysurvey], [Du2025memorysurvey]).
• Coverage of avant-garde/nontraditional artistic styles with adaptive, open-ended expert guidance.

Conclusion

EvalVerse provides a pipeline-aware, expert-calibrated benchmark suite, rigorously quantifying both prompt-following and cinematic "goodness" for video generation. Its structural taxonomy, calibration protocols, and robust human-machine alignment establish a new standard for professional video model evaluation. As generative video synthesis transitions to agentic and RL-driven workflows, EvalVerse forms the foundational infrastructure needed for reward modeling, iterative fine-tuning, and explainable diagnostic feedback, facilitating the next advance in computable cinematography and creative AI systems (2605.23271).