CultureBench: Cultural Competence Benchmarks

• CultureBench is a suite of benchmarks that systematically evaluates cultural knowledge, adaptation, and reasoning in large language and vision-language models.
• Its methodologies include human–AI red-teaming and expert-curated multiregional data to rigorously evaluate performance across diverse cultural contexts and modalities.
• Empirical findings reveal significant performance disparities and biases, underscoring the need for culturally diverse training data and adaptive model strategies.

CultureBench refers to a class of benchmarks and frameworks that systematically assess the cultural knowledge, awareness, adaptation, and reasoning capabilities of large language and vision-LLMs. These resources enable rigorous evaluation across cultural contexts, modalities, and levels of abstraction—from concrete regional facts to conversational adaptation and implicit value inference. The term has been explicitly used for curated benchmarks and analysis suites in large-scale research on both LLMs and multimodal AI systems, and spans both “CulturalBench” (factual/multichoice knowledge) and more advanced/abstract versions such as conversational or vision models. Multiple research groups have developed, analyzed, or extended “CultureBench”-style resources across different AI paradigms (Tan et al., 13 Oct 2025, Chiu et al., 3 Oct 2024, Havaldar et al., 13 Oct 2025, Shi et al., 21 Nov 2025).

1. Conceptual Scope and Theoretical Foundations

CultureBench, in its broadest sense, refers to benchmark suites that operationalize culture as a primary axis for evaluation, transcending simple language or demographic matching. Its structure and conceptual underpinnings borrow from sociocultural theories, such as Hofstede’s cultural dimensions, and from formal frameworks in cross-cultural psychology and communication studies (Havaldar et al., 13 Oct 2025, Wang et al., 2023).

Key constructs include:

• Explicit cultural knowledge: Factual knowledge, customs, and behavioral patterns anchored to geo-linguistic regions (e.g., “What is a common snack for preschoolers in West Java?”) (Myung et al., 14 Jun 2024).
• Conversational/Stylistic adaptation: Appropriateness of language style given situational, relational, and cultural context (e.g., politeness, directness) (Havaldar et al., 13 Oct 2025).
• Implicit values/metacognitive reasoning: Extraction and interpretation of unstated, underlying cultural beliefs and attitudes from narrative or conversational contexts (Liu et al., 1 Apr 2025).
• Multimodal grounding: Integration of visual and textual cues to identify, relate, or generate culture-bound knowledge, artifacts, or styles (Tan et al., 13 Oct 2025, Shi et al., 21 Nov 2025, Xu et al., 16 May 2025).
• Conflict resolution and pluralism: Model adjudication among conflicting, legitimate cultural value systems in open-ended reasoning (Rahman et al., 3 Oct 2025).

2. Dataset Construction Methodologies

CultureBench-style benchmarks emphasize robust, diverse, and high-quality data acquisition, integrating human expertise, adversarial and gamified annotation, and LLM assistance.

Major design pipelines include:

• Human–AI Red-Teaming: Annotators iteratively create or refine questions and scenarios to “fool” an AI verifier, often with LLM-generated suggestions for increased question difficulty, coverage, or distraction (Chiu et al., 3 Oct 2024, Chiu et al., 10 Apr 2024).
• Expert-Curated Multiregional Data: Recruiting native informants and domain experts for region-specific, nuanced content, with verification by majority vote or multi-stage review (Myung et al., 14 Jun 2024, Chiu et al., 3 Oct 2024).
• Conversational and Stylistic Generation: Systematic selection of scenarios by crossing interpersonal relationship, situational, and cultural context, generating multiple candidate stylistic responses (Havaldar et al., 13 Oct 2025).
• Multimodal Data Curation: Alignment of textual templates with synthetic or real images, human-aided filtering of tangibility and representation authenticity, and stratification over cultural facets or domains (Tan et al., 13 Oct 2025, Xu et al., 16 May 2025).
• Gamified and Assistive Annotation: Annotators receive immediate AI feedback (“success attack” rates, revision hints), fostering higher creativity and difficulty in questions (Chiu et al., 10 Apr 2024).

3. Benchmark Structure and Task Formulations

CultureBench benchmarks originate in a range of formats and task types, offering broad diagnostic coverage:

Benchmark	Format(s)	Cultural Axes/Regions	Modality
CulturalBench	MCQ, T/F, open-ended	45+ regions, topics	Text
BLEnD(-Vis)	MCQ, SAQ, VQA-style	16 regions, 13 languages	Text, Vision
CultureBench (CAC)	Dialogue response selection	8 countries, 6 stylistic axes	Text
TCC-Bench	Multilingual MCQ VQA	Traditional Chinese	Vision, Text
CultureVQA	Open VQA	11 countries, 5 facets	Vision, Text
CDEval	Binary-choice, profiling	6 Hofstede dimensions	Text
CQ-Bench	Value extraction, attitude detection	Global values (WVS)	Text
CCD-Bench	Dilemma, value-conflict	10 GLOBE clusters	Text
ArtELingo-28	Emotion captioning	28 languages	Vision, Text
C³B	Multichoice, struct. generation	77 cultures, 3 tasks	Vision, Text
CultureBench (T2I)	VQA, embedding probing	15 regions/languages	Vision

Typical task types:

• Factual recall or classification (region, entity, custom)
• Multiple-choice and true/false (single- or multi-answer)
• Open-answer and free-form generation
• Dialogue turn selection or ranking
• Culturally appropriate translation or emotional captioning

4. Metrics and Evaluation Protocols

CultureBench evaluations prioritize both global and fine-grained measurement of culture-dependent performance, relying on a mix of accuracy-type metrics, cultural consistency metrics, and human-model agreement.

Highlighted metrics:

• Accuracy per format and region: Zero-shot or few-shot top-1 accuracy, broken out by region/culture and topic/domain (Chiu et al., 3 Oct 2024, Tan et al., 13 Oct 2025).
• Cross-modal consistency: Joint correctness and agreement across text and image task variants (e.g., R-V Correct, R-V Agree in BLEnD-Vis);

$$\text{Consistency} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}\bigl(\hat y_i^\text{text} = \hat y_i^\text{image}\bigr)$$

(Tan et al., 13 Oct 2025)

• Stylistic sensitivity and subjective correctness: Proportion of responses within culture-specific “accepted style ranges,” often operationalized via standard deviations about human annotator mean ratings (Havaldar et al., 13 Oct 2025).
• Cultural dimension profiling: Average selection of a dimension pole over many binary-choice questions, producing a vector of dimension means across domains (Wang et al., 2023).
• Clustered or aggregated scores: Cross-model Kullback–Leibler divergence for preference clustering; Cramér’s $V$ for ordering effects (Rahman et al., 3 Oct 2025).
• Human agreement upper bound: Empirical measurement of annotator upper-limit performance (e.g., 92.4% in CulturalBench) (Chiu et al., 3 Oct 2024).
• Metric for culture-consistent generation: Matching to cultural reference via VQA classifiers or embedded similarity (Shi et al., 21 Nov 2025).

5. Empirical Findings and Model Gaps

CultureBench analyses consistently reveal significant, systematic disparities and technical limitations in state-of-the-art models:

• Performance varies by region/culture: Leading LLMs and VLMs exhibit 20–30+ percentage-point gaps between data-rich (e.g., US/UK/Netherlands) and low-resource or underrepresented regions (e.g., South America, North Africa, Middle East, Ethiopia) (Chiu et al., 3 Oct 2024, Tan et al., 13 Oct 2025).
• Brittleness to rephrasing and modality: Minor linguistic changes or shift from text to image can reduce accuracy or consistency, indicating pattern-matching over robust cultural understanding (Δ_Reph ≠ 0, low joint correctness) (Tan et al., 13 Oct 2025).
• Failure on multi-mode/ambiguous questions: LLMs collapse to single responses even with multiple correct answers, with human–model gaps exceeding 20% in such cases (Chiu et al., 3 Oct 2024).
• Bias toward Western/consensus values: Cultural dimension and dilemma-style benchmarks (CDEval, CCD-Bench) show that foundation models cluster toward Western, egalitarian, long-term-oriented, and low power-distance stances (Wang et al., 2023, Rahman et al., 3 Oct 2025).
• Stylistic adaptation limitations: LLMs struggle with situational/relational style adaptation, especially in contexts with indirectness or high power distance (e.g., Indian/Japanese family and day-to-day communication) (Havaldar et al., 13 Oct 2025).
• Visual grounding is incomplete: Multimodal systems only partially recover cultural facts from images, with cross-modal consistency and joint correctness rates well below textual or human upper bounds (Tan et al., 13 Oct 2025, Xu et al., 16 May 2025, Shi et al., 21 Nov 2025).
• Emotion and content transfer is language-dependent: Performance in multilingual cultural captioning/translation is significantly higher among linguistically/culturally related languages and when trained on native, not translated, data (Mohamed et al., 6 Nov 2024).

6. Design Recommendations and Future Directions

To address these gaps, CultureBench research advances several concrete directions:

• Increase and diversify cultural data coverage: Broader representation of low-resource, minority, and non-Western languages and cultures in pretraining and fine-tuning data is essential (Chiu et al., 3 Oct 2024, Myung et al., 14 Jun 2024, Tan et al., 13 Oct 2025).
• Explicit evaluation for multi-answer and ambiguous scenarios: Benchmarks should systematically include and measure model behavior on multi-mode and value-conflict items (Chiu et al., 3 Oct 2024, Rahman et al., 3 Oct 2025).
• Multimodal and multilingual expansion: Incorporate parallel, culture-grounded multimodal resources (audio, video, 3D) and native-language question-answering or generation (Xu et al., 16 May 2025, Mohamed et al., 6 Nov 2024).
• Model adaptation strategies: Use cross-modal or culture-aware fine-tuning, retrieval-augmented methods, and architecture- or adapter-level interventions (e.g., culture neuron activation) to boost culture-specific capability without sacrificing generalization (Shi et al., 21 Nov 2025).
• Gamified, interactive annotation: Human–AI collaborative red-teaming and gamification boost both dataset quality and annotator creativity and challenge, yielding harder benchmarks (Chiu et al., 10 Apr 2024).
• Multi-dimensional/continuous evaluation: Go beyond single-axis metrics to multidimensional culture profiles, open-ended reasoning, and stepwise decision-making under conflict (Wang et al., 2023, Rahman et al., 3 Oct 2025, Liu et al., 1 Apr 2025).

7. Significance and Impact

CultureBench and affiliated frameworks have catalyzed a shift in AI evaluation—from knowledge recall or translation accuracy toward multidimensional, adversarial, and context-sensitive assessment of cultural competence. Their rigorous construction and formalization have already informed the design of new multimodal models, inspired targeted adaptation strategies, and exposed both data-driven and alignment-induced biases present in foundation models. Ongoing CultureBench research defines the state of the art in evaluating, analyzing, and ultimately mitigating cultural performance gaps in globally deployed artificial intelligence systems (Tan et al., 13 Oct 2025, Chiu et al., 3 Oct 2024, Shi et al., 21 Nov 2025, Havaldar et al., 13 Oct 2025, Xu et al., 16 May 2025).