CULTURESCORE: Evaluating Cultural Faithfulness in Video Generation Models

Abstract: As video generation models like Veo 3.1 and LTX-2 advance, their ability to accurately represent diverse global cultures remains a critical yet understudied frontier. Current metrics, such as VideoScore, only measure visual quality but offer no mechanism for assessing cultural faithfulness. Consequently, a model that replaces a Namaste with a handshake receives the same score as one that generates the gesture correctly. We propose CultureScore, a compositional evaluation framework that decomposes cultural faithfulness into three granular dimensions: Identity (who is represented), Context (culturally localized background), and Behavior (normative gestures and interactions). We operationalize this framework through an evaluation suite spanning 10 countries, yielding 6,180 generated videos across three state-of-the-art models. Our evaluation reveals that no current model achieves culturally faithful video generation: the best-performing model reaches only 56.8\% overall CultureScore, with Behavior the most challenging dimension, which remains below 52\% across all models. Furthermore, human preference rankings align directionally with CultureScore but are inverted relative to VideoScore; the highest-scoring model on visual quality was ranked last by annotators, underscoring that cultural faithfulness is an essential criterion for equitable video generation.

• The paper introduces CultureScore, a framework for compositional evaluation of cultural representation in video models.
• It decomposes cultural cues into Identity, Behavior, and Context to diagnose nuanced misrepresentations using targeted QA pairs and extended prompts.
• Results reveal an inverse relationship between perceptual quality and cultural accuracy, emphasizing the need for culturally-aware generative evaluation.

CultureScore: A Compositional Evaluation for Cultural Faithfulness in Video Generation Models

Motivation and Context

With the rapid proliferation of large-scale text-to-video generative models, the fidelity of cultural representation in their outputs has become a salient concern. Models such as Veo 3.1, LTX-2, and Wan 2.2 achieve high photorealism and temporal coherence, yet their ability to faithfully represent the nuanced diversity of global cultures is not assessed by standard metrics. Existing evaluation tools, e.g., VideoScore, focus on perceptual attributes (visual quality, temporal consistency), ignoring the semantic granularity of culture, and thus conflate cultural misrepresentation with mere visual error. This lack of targeted evaluation can lead to subtle but impactful failures—such as the replacement of specific gestures (e.g., Namaste) with culturally default alternatives (e.g., handshake)—that current deployments would not detect or penalize.

The "CultureScore" framework (2606.07311) addresses this foundational gap by introducing a compositional, diagnostic metric, specifically designed to probe cultural faithfulness across Identity, Behavior, and Context dimensions in video generation.

Figure 1: The CultureScore framework decomposes cultural faithfulness into Identity, Context, and Behavior for multi-dimensional, compositional evaluation.

Methodology

Compositional Decomposition: Identity, Behavior, Context

CultureScore decomposes each cultural prompt into three interpretable axes:

• Identity: Pertains to who is depicted, capturing visual markers such as ethnicity, attire, and demographic attributes.
• Behavior: Encodes what agents do, operationalizing temporally grounded, culture-specific actions (e.g., gestures, sequences, interactions).
• Context: Captures where and in what setting an event unfolds, reflecting decor, environment, and implicit social norms.

This decomposition recasts evaluation as a set of aspect-specific question-answer pairs, enabling fine-grained localization of cultural inaccuracies that aggregate scores cannot surface.

Figure 2: Diagram of the overall CultureScore evaluation pipeline. Prompts are decomposed into components, videos are generated, and VLMs evaluate aspect-based QA pairs for each dimension.

Dataset and Prompt Engineering

Adapting the CulturalFrames T2I benchmark, the authors curated 2,943 prompts validated by native informants, spanning ten countries and five key domains (dates of significance, religion, etiquette, family, greetings). Each prompt was expanded into:

• Base Prompt: Compact, geographically explicit description.
• Extended Prompt: Decomposed and lexically elaborated with Oxford Dictionary definitions for maximum cultural specificity.
• Geographical Constraint Removed: Eliminates nation/city tokens, testing for implicit, trigger-independent cultural generalization.

  Figure 3: Illustration of prompt variants—Base (explicit geography), Extended (fully decomposed), and Geographical Constraint Removed (tests implicit knowledge).

Video Generation and VLM-Based Scoring

Three SOTA models—Veo 3.1 Fast, LTX-2, and Wan 2.2—were prompted to generate videos for each variant. For each video, a set of corresponding dimension-specific QA pairs were authored (or automatically generated then human-verified), requiring VLMs (Qwen3-VL235B-A22B-Instruct) to act as judges. Evaluation proceeds as:

1. For each video, ask: Does the behavior/localization/identity in the video answer the prompt-specific question?
2. A VLM grades the fidelity of the output against the gold-standard answer set.
3. Scores across QA items are averaged within each dimension, and rolled up into an overall CultureScore.

Experimental Analysis

Dimension-Level Performance

All models exhibit a consistent pattern: Context is the highest-performing dimension, with Wan 2.2 achieving up to 69.3%, followed by LTX-2 (62.1%) and Veo 3.1 Fast (59.0%) when given extended prompts. Identity scores lag, with top (Wan 2.2, 51.3%), and Behavior remains the hardest, sub-52% under all conditions. These trends elucidate that cultural misrepresentation is most acute in temporally contingent, non-static aspects (gestures, interactions)—a regime static image metrics cannot probe.

Figure 4: Average CultureScore across Identity, Behavior, Context for three video models with extended prompts. Context dominates across models; Wan 2.2 is the highest overall.

Inverse Correlation with Perceptual Quality (VideoScore)

A critical finding is the strong inverse correlation between traditional VideoScore and CultureScore: LTX-2, the model preferred by perceptual metrics, consistently exhibits the lowest cultural faithfulness, whereas Wan 2.2, while lowest in "cinematic" quality, is highest in cultural adherence. This result highlights an intrinsic tradeoff, or at least divergence, in optimization between perceptual appeal and authentic cultural rendering.

Figure 5: Inverse relationship between VideoScore and CultureScore—models excelling in visual quality (e.g., LTX-2) underperform on cultural faithfulness, and vice versa.

Impact of Prompt Decomposition

Explicitly structured extended prompts dramatically improve culture-aligned generation. Gains are most pronounced in Context (+25.5% for LTX-2), moderate for Identity, and smallest for Behavior. While detailed prompting raises scores across the board, there remains a persistent behavioral error floor, indicating that current models cannot internalize and reproduce the kinematics of culture-specific actions with simple scale or prompting improvements.

Figure 6: Extended prompts confer consistent boosts to CultureScore compared to base prompts, across all major video generation models.

Implicit Cultural Representation: The Weakness of Generalization

When geographic tokens are excised to test implicit knowledge, model accuracy collapses, especially on Context and Behavior, by up to 33+ percentage points (LTX-2). This suggests that models largely "pattern-match" triggers rather than internalizing the semantics of culture, exposing the superficiality of their world knowledge. The effect is more pronounced for categories and nations underrepresented in the data.

Figure 7: Removal of explicit geographic cues severely degrades performance, especially for LTX-2 and for the Context dimension, revealing the lack of deep cultural generalization.

Human Evaluation and Metric Alignment

Native informant studies (N=45, across 9 nations) reveal that human preference is not aligned with VideoScore: the most visually appealing model is typically least preferred for cultural accuracy. By contrast, CultureScore shows strong directional agreement with native annotator ranks—waning perceptual quality is tolerated when cultural authenticity is preserved, confirming that faithful cultural rendering is a crucial but unmeasured requirement.

Figure 8: CultureScore aligns with human cultural preferences, whereas VideoScore is inversely correlated across regions and models.

Practical and Theoretical Implications

Practically, this work exposes the urgent need for compositional, culture-aware metrics in GAI system deployment, especially in global, intercultural applications (e.g., education, media, advertising). Evaluations that focus solely on fidelity or aesthetics will systematically encode and perpetuate cultural erasure and bias.

Theoretically, the framework demonstrates that decomposition into semantically interpretable dimensions not only enables better diagnosticity but is essential for exposing and quantifying real-world harms. It further indicates that generative models’ internalization of culture is more shallow than previously assumed, dependent on prompt engineering and explicit cues rather than latent world models.

Persistent failures in the Behavior axis suggest that video model architectures and training recipes must directly address temporal semantic alignment and should move beyond static, frame-centric objectives. Architectures or training regimens that explicitly encode action kinematics and their cultural dependencies may be necessary to close this gap.

Limitations

The evaluation pipeline is partially reliant on VLMs as surrogate judges, inheriting both their potential biases and limitations. Annotator coverage omits certain locales (e.g., Iran, for geopolitical reasons); Veo 3.1 Fast coverage is limited by cost, which may impact the generality of quantitative model comparisons.

Conclusion

CultureScore presents a robust, compositional framework for evaluating cultural faithfulness in video generation models, addressing a fundamental deficit in current generative AI benchmarking. Results systematically invalidate naïve assumptions that high visual quality entails high cultural accuracy, and demonstrate that aspect-oriented prompt engineering can yield significant improvements, albeit with persistent behavioral deficits. Going forward, the framework provides a reusable, extensible foundation for research and auditing in culturally contingent generative systems.