Long-CODE: Isolating Pure Long-Context as an Orthogonal Dimension in Video Evaluation

Abstract: As video generation models achieve unprecedented capabilities, the demand for robust video evaluation metrics becomes increasingly critical. Traditional metrics are intrinsically tailored for short-video evaluation, predominantly assessing frame-level visual quality and localized temporal smoothness. However, as state-of-the-art video generation models scale to generate longer videos, these metrics fail to capture essential long-range characteristics, such as narrative richness and global causal consistency. Recognizing that short-term visual perception and long-context attributes are fundamentally orthogonal dimensions, we argue that long-video metrics should be disentangled from short-video assessments. In this paper, we focus on the rigorous justification and design of a dedicated framework for long-video evaluation. We first introduce a suite of long-video attribute corruption tests, exposing the critical limitations of existing hort-video metrics from their insensitivity to structural inconsistencies, such as shot-level perturbations and narrative shuffling. To bridge this gap, we design a novel long-video metric based on shot dynamics, which is highly sensitive to the long-range testing framework. Furthermore, we introduce Long-CODE (Long-Context as an Orthogonal Dimension for video Evaluation), a specialized dataset designed to benchmark long-video evaluation, with human annotations isolated specifically to genuine long-range characteristics. Extensive experiments show that our proposed metrics achieve state-of-the-art correlation with human judgments. Ultimately, our metric and benchmark seamlessly complement existing short-video standards, establishing a holistic and unbiased evaluation paradigm for video generation models.

• The paper introduces Long-CODE, a metric that isolates long-context evaluation using a dual-branch approach combining Dynamic Structure Alignment and MLLM-based reasoning.
• It demonstrates that conventional short-video metrics fail to capture narrative causality and temporal logic, highlighting the need for orthogonal evaluation.
• Empirical tests show Long-CODE achieving strong correlations with human ratings, surpassing traditional methods in capturing long-range dependencies.

Long-CODE: An Orthogonal Framework for Long-Context Video Evaluation

Problem Formulation: Orthogonality of Long-Context Video Attributes

Conventional metrics for video generation evaluation (e.g., VBench, VideoCLIP-based paradigms) are bound to short-video properties, predominantly measuring frame-level fidelity and localized temporal coherence. As generative models scale towards longer narratives with complex, multi-shot structure, these metrics prove fundamentally insufficient: shot-level visual quality is decoupled from critical long-range dependencies such as narrative causality, shot ordering, and identity consistency across scenes. The authors formalize this distinction by demonstrating that any aggregation-based short-video metric (using symmetric aggregators on per-shot qualities) is insensitive to sequence-level perturbations and, in information-theoretic terms, is orthogonal to long-context evaluation signals.

Corruption Test Suite and Empirical Validation

To empirically test the orthogonality hypothesis, the paper designs a suite of long-range attribute corruption operators:

• Shuffle: Random permutation of shot order while preserving intra-shot content.
• Replace: Substitution of shots with semantically plausible samples from an unrelated bank.
• Edition: Intra-shot editing (e.g., character attribute replacement) with high-level semantics disrupted but local motion preserved.
• Synthesis: Surreal re-synthesis of shots based on modified captions.

Short-video metrics display minimal variation in response to these corruptions, validating their lack of sensitivity to long-range structure (as demonstrated in Figure 1).

Figure 1: Correlation between metric scores and corruption strengths, illustrating that standard short-video metrics are insensitive to long-range corruption intensity.

The Long-CODE Metric: Dual-Branch Long-Video Evaluation

The core proposal is Long-CODE—a dedicated, orthogonally designed benchmark and metric suite for comprehensive long-video evaluation. The framework consists of two principal branches:

1. Dynamic Structure Alignment (DSA):

This module computes the structural consistency between the intended shot sequence (prompt-derived) and the realized video. L2-normalized embeddings for both text and video segments are mapped into corresponding similarity vectors, and their concordance is quantified via Spearman-rank correlation. DSA directly targets the preservation of temporal logic, causal chains, and narrative progression—dimensions invisible to intra-shot assessment.

2. MLLM-Based Long-Context Reasoning:

To address narrative fidelity and global causality, a Multimodal LLM is leveraged in a three-step process: (i) automated captioning and segmentation, (ii) “deep thinking” by comparing original prompts with inferred captions and generating an analytical summary, and (iii) final quality assessment via LLM-based scoring. This module is robust to local visual perturbations and comprehensively scores narrative alignment, causal plausibility, and identity consistency.

The final Long-CODE score is the weighted sum of DSA and MLLM branches, balanced by a tunable hyperparameter $\alpha$.

Figure 2: The Long-CODE evaluation architecture, combining DSA for temporal-structural fidelity and MLLM for deep semantic analysis.

The Long-CODE Benchmark Dataset

The paper introduces a novel dataset specifically designed to probe long-range attributes. Using MLLM-aided decomposition, each video is associated with a multi-shot narrative breakdown, and several state-of-the-art VGMs synthesize corresponding outputs. Critically, human annotators are guided to rate only long-context properties (e.g., cross-scene causality, consistent identity, and narrative flow) while ignoring per-shot visual details. This isolation protocol ensures that the dataset provides an unbiased ground-truth alignment for long-context evaluation.

Figure 3: Illustrations of the Long-CODE benchmark and comparative sensitivity of short-video and Long-CODE metrics to long-range corruption.

Quantitative and Qualitative Results

Extensive benchmarking across six representative video generation architectures (including modular VGoT, HoloCine, StoryMem, and closed-source commercial engines Veo3.1, Sora2) demonstrates the efficacy of Long-CODE:

• Short-video metrics yield overall Spearman/Pearson correlations near zero or negative when compared with human ratings of long-range attributes (Figure 1, Table of human alignment in main text).
• In contrast, Long-CODE achieves state-of-the-art correlation with human judgments across all models (minimum per-model Spearman $>$ 0.57), outstripping the runner-up by a relative margin exceeding $105\%$.
• Ablation studies confirm the orthogonality and complementarity of DSA (which dominates on temporal/structural fidelity) and MLLM (which excels in semantic/narrative consistency).

Qualitative case studies (Figure 4) further highlight failure modes (e.g., broken causal chains, identity drift) not captured by short-video metrics but robustly penalized by Long-CODE.

Figure 4: Case study visualizing generation results on the Long-CODE dataset, exposing narrative degradation across state-of-the-art models.

Implications and Future Directions

This work establishes that long-context attributes in video are fundamentally orthogonal to shot-level visual quality and must be directly and separately benchmarked for holistic evaluation of generative models. Long-CODE fills a crucial gap in existing methodologies, providing both an open-source dataset and a validated evaluation metric.

The practical implication is the ability to systematically track progress on long-video generation, where human perceptual alignment depends on subtle temporally-extended structure rather than immediate frame consistency. Theoretically, the orthogonality principle introduced has ramifications for future architectures: improving long-video generation quality will require advances not only in visual diffusion but also in explicit modeling (and evaluation) of cross-shot dependencies and causal logic.

Future research will likely expand Long-CODE with multilingual/cultural narratives, adversarial structure-focused corruptions, and potentially closed-loop training of VGMs with DSA/MLLM feedback as direct optimization signals.

Conclusion

Long-CODE isolates long-context video evaluation as an independent and essential criterion, providing a methodological foundation and empirical baseline for the next generation of video generative models. Its orthogonal dual-branch metric robustly tracks long-range narrative quality, structures a new benchmark dataset, and closes a critical coverage gap missed by all prior short-video-centric approaches.