Intentionally Cultural Evaluation

• Intentionally Cultural Evaluation is a systematic framework that explicitly embeds cultural context in AI assessments, integrating cultural sensing, scoping, and fluency.
• It redefines traditional evaluation benchmarks by incorporating dynamic, community-informed metrics and contextualized protocols.
• This approach enhances model performance and fairness across cultural domains by addressing biases and promoting methodological rigor.

Intentionally cultural evaluation is a systematic approach to assessing AI systems that makes the cultural context of evaluative decisions explicit and deliberate, rather than leaving cultural influences implicit or accidental. In this formulation, culture is not confined to explicit “culture quiz” tasks or national trivia. It enters task choice, prompt format, interaction setting, reference standards, annotator positionality, and metric design, and it affects whether systems can detect culturally sensitive situations, scope the relevant cultural context, and respond with accurate, rich, and socially appropriate situated knowledge (Oh et al., 1 Sep 2025, Dev et al., 1 Mar 2026). Across recent work on LLMs, vision-LLMs, and text-to-image systems, intentionally cultural evaluation has therefore become a broad research program spanning factual knowledge, contextual behavior, explanation quality, representational harms, multilingual robustness, and community-grounded assessment (Bravansky et al., 13 Jan 2025, Nayak et al., 10 Jun 2025).

1. Conceptual foundations

A central formulation defines cultural intelligence as “the set of capabilities to detect and scope culturally sensitive interactions, and to generate competent responses grounded in the retrieval and application of accurate, rich, and comprehensive situated cultural knowledge” (Dev et al., 1 Mar 2026). That framework organizes culture into three domains: Cultural Production, Behavior and Practices, and Knowledge and Values. It then decomposes cultural intelligence into Cultural Sensing, Cultural Scoping, and Cultural Fluency, with cultural fluency further analyzed as Epistemic Fidelity, Representational Richness, and Pragmatic Proficiency (Dev et al., 1 Mar 2026).

This capability-oriented view differs from treating culture as a fixed inventory of facts. Several papers instead describe culture as dynamic, situated, contested, relational, and performed in context. “Rethinking AI Cultural Alignment” argues that cultural alignment should be reframed as a bidirectional process shaped by how humans structure interaction with an AI system, rather than as a one-directional imposition of standardized values (Bravansky et al., 13 Jan 2025). “Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lens” similarly argues that many benchmarks assume the wrong ontology of culture by reducing it to static facts, homogeneous values, or stereotype probes, whereas anthropology treats culture as enacted in language and interaction (AlKhamissi et al., 7 Oct 2025).

A related conceptual move is the separation of background construct from measurement. The unified framework based on measurement validity theory and the four-level measurement approach of Adcock and Collier distinguishes the background concept of cultural intelligence from its operationalization through indicators, probes, and metrics (Dev et al., 1 Mar 2026). This distinction is methodologically significant because it permits multiple measurement strategies for the same capability and avoids equating one benchmark with the full construct.

2. Critique of trivia-centered and one-directional evaluation

A recurring criticism is that prevailing cultural evaluation is trivia-centered. Position papers argue that current methods often reduce culture to static facts or values and test models through multiple-choice or short-answer formats that treat culture as isolated recall, even though culturally appropriate behavior is often plural, interactive, and situational (Oh et al., 1 Sep 2025). The critique is not limited to explicitly cultural tasks: the same paper cites work showing that 28% of MMLU requires culturally sensitive knowledge, indicating that supposedly neutral evaluation settings already contain cultural assumptions (Oh et al., 1 Sep 2025).

“Rethinking AI Cultural Alignment” makes this objection concrete through a GPT-4o case study using GlobalOpinionQA from Durmus et al. across the United States, China, Japan, and India. It compares a constrained classification condition, a chain-of-thought condition, and an unconstrained condition using ten realistic social scenarios such as a phone survey, public debate, radio show, blog post, and community forum. The paper argues that forced-choice evaluation is a poor proxy for cultural expression because it suppresses hedging, refusal, uncertainty, and nuanced answers not captured by options, and because unconstrained generations often become “unclassifiable” under an MCQ-oriented extractor (Bravansky et al., 13 Jan 2025).

An anthropological synthesis further organizes benchmark assumptions into four lenses:

Lens	Typical benchmark form	Recurrent concern
Culture-as-Knowledge	Facts, customs, symbolic references	Flattens culture into recall
Culture-as-Preference	Likert judgments, majority labels	Assumes coherence and consensus
Culture-as-Dynamics	Scenarios, dialogues, role-sensitive behavior	Still comparatively rare
Culture-as-Bias	Stereotype and fairness probes	Often uses simplified identity categories

This framework is paired with six recurring methodological issues: platform bias, conflating nation-states with culture, treating annotators as cultural representatives, reducing moral reasoning to survey scales, assuming cultural consensus, and stripping context from cultural scenarios (AlKhamissi et al., 7 Oct 2025). A plausible implication is that intentionally cultural evaluation is not merely the addition of more cultural items; it is a redesign of the ontology, context, and evidential basis of evaluation.

3. Language-centered benchmarks and protocols

Several benchmarks implement intentionally cultural evaluation for LLMs by changing both the task format and the evidential target. CDEval measures a model’s tendency to choose one side of a binary cultural orientation across six cultural dimensions—PDI, IDV, UAI, MAS, LTO, and IVR—and seven domains: Family, Education, Work, Wellness, Lifestyle, Arts, and Scientific. Its final dataset contains 2,953 questions, uses six prompt variants per item, and formalizes model culture as aggregated likelihood over prompt templates, with human-model similarity computed against Hofstede-style country profiles (Wang et al., 2023).

Other benchmarks replace direct value questions with contextualized inference. PerCul is a Persian benchmark of 592 multiple-choice question-answer pairs derived from Hall’s Triad of Culture, focused on the technical and formal levels and extending the technical level with Iconic Figures and Objects. Its stories imply the target concept without naming it directly, and its reported results show a 11.3% gap between the best closed-source model and the layperson baseline, increasing to 21.3% for the best open-weight model (Monazzah et al., 11 Feb 2025). Kalahi, a handcrafted Filipino suite, uses 150 prompts with structured fields—User, Context, Personal situation, and Instruction—to test whether a model produces what an average Filipino would say or do in culturally specific situations. It combines MC1, MC2, and open-ended generation scoring, and reports a best model accuracy of 46.0% against a native Filipino baseline of 89.10% (Montalan et al., 2024).

A parallel trend moves away from multiple-choice altogether. TARAZ converts Persian cultural benchmarking into short-answer evaluation and adds Persian-specific normalization—character normalization, digit normalization, stop-word removal with hazm, suffix stripping, conjunction-based segmentation, and a hybrid semantic scorer using maux-gte-persian-v3 with threshold $\text{cosine similarity} > 0.85$. The abstract reports that this hybrid evaluation improves scoring consistency by +10% compared to exact-match baselines (Iranmanesh et al., 26 Feb 2026). MAKIEval evaluates open-ended generation across 13 languages, 19 countries and regions, and 6 topics, extracting entities with GPT-4o-mini and grounding them in Wikidata. It reports 1,716 unique prompts, 500 generations per prompt per model, over 85.8 million generated texts, and 24,042 matched cultural entities, and proposes granularity, diversity, cultural specificity, and consensus across languages as metrics (Zhao et al., 27 May 2025).

Dynamic and causal protocols further extend the design space. MCEval constructs questions from source cultural corpora and then applies Counterfactual Rephrasing and Confounder Rephrasing to test whether models track causal cultural content rather than memorized surface forms. It spans 13 cultures and 13 languages, with 39,897 cultural awareness instances and 17,940 cultural bias instances (Huang et al., 13 Jul 2025). CRaFT shifts evaluation from answers to answer-explanation pairs, defining Cultural Fluency, Deviation, Consistency, and Linguistic Adaptation for multilingual explanation-based assessment (Hossain et al., 15 Oct 2025). A different extension uses Bloom’s Taxonomy with Retrieval-Augmented Generation over a Taiwanese Hakka archive, evaluating Remembering, Understanding, Applying, Analyzing, Evaluating, and Creating in closed-book and open-book conditions (Lee et al., 3 Nov 2025).

4. Visual, multimodal, and text-to-image evaluation

In multimodal settings, intentionally cultural evaluation is defined by a shift from generic perception to culturally grounded interpretation. CulturalVQA introduces 2,378 image-question pairs with 2,328 unique images and 7,206 total answers covering 11 countries across 5 continents and five cultural facets: clothing, food, drink, rituals, and traditions. Its results show strong performance disparities across regions, with stronger cultural understanding for North America and substantially lower performance for Africa (Nayak et al., 2024).

K-Viscuit applies a semi-automated human-VLM collaboration pipeline to Korean culture. It contains 657 multiple-choice QA samples over 237 unique images, split into Type 1 visual recognition and Type 2 visual reasoning. GPT-4-Turbo generates candidate questions using demonstrations, guidelines, and image-specific knowledge, and native Korean speakers verify quality and cultural relevance. On this benchmark, Gemini-1.5-Pro reaches 81.58 overall accuracy, GPT-4-Turbo reaches 80.82, and the best open-source model, Idefics2-8B, reaches 63.62, leaving a gap of about 18 points (Park et al., 2024).

Other multimodal benchmarks target higher-order interpretation. VULCA-Bench contains 7,410 matched image–critique pairs across 8 cultural traditions with Chinese-English bilingual critiques and 225 culture-specific dimensions. It formalizes cultural understanding through a five-layer hierarchy from L1 Visual Perception to L5 Philosophical Aesthetics, and reports that pilot models drop sharply from L1–L2 to L3–L5, with layer gaps of about 25–40 percentage points (Yu et al., 12 Jan 2026). CulturalFrames introduces a benchmark for text-to-image systems that distinguishes explicit cultural expectations, directly named in the prompt, from implicit expectations, culturally necessary but not directly named. It spans 10 countries, 5 socio-cultural domains, 983 prompts, 3,637 images, and 10,911 ratings, and uses culturally knowledgeable annotators to diagnose explicit, implicit, or joint failure modes (Nayak et al., 10 Jun 2025).

Prompt-based probing of text-to-image models predates these benchmarks. “Navigating Cultural Chasms” organizes culture into cultural dimensions, cultural domains, and cultural concepts, and tests multilingual text encoders using prompt templates such as English Reference, Fully Translated Prompt, Translated Concept, English with Nation, and Gibberish prompts. It introduces CulText2I and evaluates images using intrinsic CLIP-space measures, extrinsic VQA with BLIP2, and human assessment, showing that language form, nationality naming, and script fragments can alter the cultural content surfaced by a model (Ventura et al., 2023).

A different multimodal trajectory rejects single-score benchmarking altogether. “Back to the Communities” develops a mixed-methods community-based methodology for text-to-image evaluation through a literature review of 27 studies, co-creation workshops with 34 participants, and broader validation involving 59 individuals from 19 countries. Its six-step process—Positionality, Text prompts, Initial reflection, Image generation, Evaluation, and Final reflection—treats first-person experience and disagreement as constitutive evidence rather than annotation noise (Kiden et al., 31 Oct 2025).

5. Metrics, scoring logics, and recurrent empirical patterns

Intentionally cultural evaluation employs heterogeneous scoring logics because the target construct differs across tasks. Some protocols retain discrete correctness. Kalahi defines

$$\text{MC2}=\frac{\sum_{r \in R} p(r \mid x)}{\sum_{r \in R \cup I} p(r \mid x)},$$

where $R$ is the set of relevant responses and $I$ the irrelevant ones, and computes probabilities as byte-length-normalized log-probability completions (Montalan et al., 2024). CDEval estimates the likelihood that a model selects the target cultural orientation under repeated prompt variants,

$\hat{P}_{M}(g_i|s_t) = \frac{1}{R}\sum_{k=1}^{R} \mathbbm{1}[\hat{a}_{tk} = g_i],$

then aggregates over templates and compares the resulting cultural profile to human survey scores (Wang et al., 2023). VULCA-Bench uses Dimension Coverage Rate,

$$\text{DCR}(c, k) = \frac{|\mathcal{D}_k^c|}{|\mathcal{D}_k|},$$

as a coarse diagnostic of how many culture-specific dimensions appear in a generated critique (Yu et al., 12 Jan 2026).

Other papers explicitly score gradience, explanation quality, or multilingual stability. CulturalFrames operationalizes prompt-image alignment on the 3-point scale $\{0.0, 0.5, 1.0\}$, with any score below 1 triggering structured analysis of explicit, implicit, or joint failure modes (Nayak et al., 10 Jun 2025). CRaFT evaluates explanation-based reasoning with Cultural Fluency, Deviation, Consistency, and Linguistic Adaptation, including

$\text{Deviation}_{ijr} = 1 - \cos(e_{ijr}, q_{ij})$

and

$$\text{Linguistic Adaptation}_{i,r} = 1 - \cos(e_{i,\text{EN},r}, e_{i,\text{TL},r}),$$

to capture semantic drift and cross-lingual reorientation (Hossain et al., 15 Oct 2025). MAKIEval grounds open-ended generations in Wikidata and measures granularity, diversity, cultural specificity, and cross-language consensus via Jaccard similarity over entity sets (Zhao et al., 27 May 2025).

Empirically, the literature reports recurring shortfalls. In CulturalFrames, cultural expectations are missed 44% of the time on average; explicit expectations are missed at 68%, implicit expectations at 49%, and among ratings below 1, 50.3% are attributed to explicit issues, 31.2% to implicit issues, and 17.9% to both. Existing automatic metrics correlate poorly with human judgments: VIEScore is best, but only reaches Spearman 0.30 for prompt alignment versus human-human agreement of 0.38, and its explanation score is 2.19 out of 5 (Nayak et al., 10 Jun 2025). In the GPT-4o case study of open-ended cultural value expression, unconstrained generations yield high percentages of unclassifiable outputs—28.83% for the US, 40.26% for China, 31.14% for Japan, and 32.27% for India—showing that forced-choice protocols miss a substantial part of model behavior (Bravansky et al., 13 Jan 2025). MCEval finds that counterfactual rephrasing usually causes larger drops than confounder rephrasing and that methods appearing successful in English can produce performance degradation of up to 66.7% in other language settings (Huang et al., 13 Jul 2025). MAKIEval reports that models tend to exhibit stronger cultural awareness in English, suggesting that English prompts often activate culturally grounded knowledge more effectively (Zhao et al., 27 May 2025).

6. Disagreement, harms, and future directions

A consistent finding is that intentionally cultural evaluation cannot assume singular ground truth. The community-based text-to-image study reports that stereotypical representations appeared in nearly 75% of images, that about half of stereotypical images were also seen as demeaning, and that the average demeaning rate across all images was around 27%. Yet inter-annotator agreement varied widely, with some groups showing high agreement and others negative or near-zero ordinal Krippendorff’s alpha for some dimensions; Swiss French and Swiss German evaluators, Nigerian evaluators with different living locations, South Korean evaluators of different ages, and Irish evaluators at home versus abroad could disagree substantially (Kiden et al., 31 Oct 2025). This establishes disagreement as a central empirical feature rather than an anomaly.

These studies also identify multiple harm types. The same community-based framework maps findings to representational harms, quality-of-service harms, and social harms, linking misrepresentation to tourism impacts, economic harms, emotional distress, microaggressions, erasure of minority cultures, and additional user effort required to obtain culturally acceptable outputs (Kiden et al., 31 Oct 2025). The resource-scaling paper centered on the SCALE Repository reaches a complementary conclusion from a data perspective: underrepresentation is broad, the United States appears far more often than many other countries in underspecified prompts, and improving global applicability requires intentionally expanded cultural resources collected through Wikidata retrieval, LLM generation with human validation, community-based data collection, and context-aware localization (Stepanyan et al., 29 Oct 2025).

Future directions are increasingly explicit. Recommendations include using real-world narratives and scenarios, involving cultural communities and social scientists throughout design and validation, evaluating models in context rather than isolation, decoupling culture from nation-state proxies, using critical annotation that preserves ambiguity and annotator positionality, and recording plural ground truths and disagreement clusters (AlKhamissi et al., 7 Oct 2025). Position papers additionally stress that evaluator positionality shapes what problems become legible, which benchmarks count as legitimate, and which local needs remain invisible; they call for stakeholder-centered and participatory evaluation capable of surfacing “unknown unknowns” that current benchmarks do not yet capture (Oh et al., 1 Sep 2025). A plausible implication is that intentionally cultural evaluation is best understood not as a single benchmark genre but as a family of practices for making cultural assumptions, contestation, and consequences measurable without treating culture as a static label.