CulturalBench: Evaluating LLM Cultural Knowledge

• CulturalBench is a benchmark that evaluates LLMs’ factual cultural knowledge across 45 regions and 17 everyday topics using human-written and verified questions.
• It employs a three-step Human-AI CulturalTeaming process that includes interactive red-teaming, expert quality checks, and majority-vote filtering to refine questions.
• Empirical results reveal significant regional performance gaps and demonstrate that the Hard setup exposes LLMs’ over-reliance on multiple-choice heuristics.

CulturalBench is a human-written, human-verified benchmark for evaluating the cultural knowledge of LLMs across diverse regions and everyday cultural topics. It was introduced as a benchmark built through a Human-AI “CulturalTeaming” pipeline inspired by red-teaming, with the explicit aim of separating superficial multiple-choice heuristics from broader cultural understanding through paired Easy and Hard evaluation setups (Chiu et al., 2024). In the surrounding literature, the name also appears in two closely related senses: as an earlier prototype, CULTURALBENCH-v0.1, collected through interactive workshops (Chiu et al., 2024), and as a reference benchmark later reused to study multilingual reasoning, translation effects, and language–culture entanglement (Tam et al., 23 May 2025).

1. Origins, benchmark identity, and scope

CulturalBench emerged from the claim that robust, diverse, and challenging cultural knowledge benchmarks are necessary for measuring progress toward LLMs that are helpful across diverse cultures. The full benchmark is described as covering 45 countries or regions and 17 topic areas grounded in daily life, social etiquette, and wider society, with underrepresented areas such as Bangladesh, Zimbabwe, and Peru explicitly included (Chiu et al., 2024).

The benchmark’s reporting has two levels. The abstract describes CulturalBench as “a set of 1,696 human-written and human-verified questions,” whereas the detailed benchmark construction reports a final multiple-choice dataset of 1,227 questions retained after majority-vote filtering. The same paper then derives a Hard setup by converting each four-option multiple-choice question into four binary True/False judgments (Chiu et al., 2024). This suggests that the benchmark is best understood as a family of evaluation setups built from a single human-verified cultural question pool, rather than as a single flat item count.

An earlier antecedent, CULTURALBENCH-v0.1, was introduced through the CulturalTeaming system as a compact evaluation dataset of 252 carefully reviewed English MCQs spanning 34 distinct cultures. That precursor already targeted multicultural knowledge and nuanced social norms, and leading models on it ranged from 37.7% to 72.2% accuracy, indicating substantial headroom even before the larger benchmark was released (Chiu et al., 2024).

2. Human-AI CulturalTeaming and benchmark construction

The benchmark’s construction methodology is central to its identity. In the full 2024 release, CulturalBench is built via a three-step Human-AI CulturalTeaming pipeline. Step 1 is interactive AI-assisted red-teaming, in which human contributors brainstorm culturally relevant scenarios from personal experience, an AI helper transforms those scenarios into structured MCQs and suggests revision strategies, and an AI verifier is used iteratively so contributors can refine questions that are overly easy for models. Step 2 is a human quality check with five annotators per region per question. Step 3 is majority-vote filtering, keeping only questions with at least one majority label of at least four out of five votes (Chiu et al., 2024).

The topics are discovery-based rather than narrowly predefined. The benchmark groups them into three macro-categories. Daily Life includes Clothing, Food, Entertainment, Language/Communication, Schools, Workplace, Travel/Transport, Dating/Marriage, and Family. Social Etiquette includes Greeting, Dining, Gift, and Visiting and Punctuality. Wider Society includes Celebrations, Politics, Religion, and Others (Chiu et al., 2024).

The earlier CULTURALBENCH-v0.1 paper provides additional detail on the developmental workflow. Its one-hour workshop sessions used two variants: a Verifier-Only mode and an AI-Assisted mode. In the former, a verifier based on GPT-3.5-turbo-0125 returned a predicted option and confidence score; in the latter, the system also helped draft an MCQ from a user-provided scenario and generated context-aware revision hints, with GPT-4 used for hint generation. The system was explicitly gamified around a “beat-the-model” loop, and stronger AI assistance produced 15.7% more Hard questions per user, alongside a higher reported creativity score of 4.19 versus 3.58 (Chiu et al., 2024).

This construction strategy has two methodological consequences. First, it keeps human cultural judgment primary while using models for scaffolding and adversarial refinement. Second, it treats question difficulty as an empirical property of model failure, not merely a stylistic property of the prompt.

3. Dataset structure, regions, and scoring

CulturalBench is released in two paired evaluation setups. In CulturalBench-Easy, each item is a standard four-option MCQ. In CulturalBench-Hard, each original MCQ is decomposed into four binary True/False decisions, and a question counts as correct only when all four option-level judgments are correct (Chiu et al., 2024).

The scoring asymmetry is deliberate. Easy uses ordinary multiple-choice accuracy with a random baseline of 25%. Hard uses an all-correct criterion over four binary decisions, giving a random baseline of $0.5^4 = 6.25\%$ (Chiu et al., 2024). This makes Hard much more diagnostic of whether a model can validate all culturally appropriate and inappropriate options rather than merely identify a plausible-looking answer.

The paper reports the following structural details for the full benchmark:

Component	Reported value	Note
Final MCQ dataset	1,227 questions	After majority-vote filtering
Regions	45 countries/regions	Global coverage
Topics	17	Daily life, etiquette, wider society

The Hard setup yields 4,908 binary questions from the 1,227 retained MCQs. The paper also notes that one section reports “4,912,” but identifies that figure as a typographical error because $1{,}227 \times 4 = 4{,}908$ (Chiu et al., 2024).

The benchmark is English-only in its original form. That choice is explicit: it was intended to evaluate cultural knowledge independently of low-resource language proficiency. The paper does not report train, development, or test splits, because CulturalBench is positioned as an evaluation benchmark rather than a training resource (Chiu et al., 2024).

4. Empirical results and diagnostic findings

CulturalBench’s central empirical claim is that the Hard setup remains challenging even for strong frontier models. Human performance is reported at approximately 92.4% on Easy and 92.6% on Hard. On Easy, the best model, GPT-4o, reaches 88.8%, with lower-performing models around 61.7%. On Hard, GPT-4o reaches 61.5%, while Llama3-8B is reported at 21.4% (Chiu et al., 2024).

The benchmark is explicitly designed to expose heuristic exploitation in Easy. An embedding-based guesser that simply picks the option whose text is most cosine-similar to the culture name, using text-embedding-3-small and without reading the question, achieves 40.42% on Easy. The reported similarity gap between correct option and culture-name similarity, 0.166 versus 0.145, is significant under a Kruskal–Wallis test with $p \le 0.01$ (Chiu et al., 2024). This is the main empirical rationale for the Hard setup.

The benchmark also diagnoses failure on cultural plurality. For single-mode items, the average Hard accuracy across models is 43.4%; for multi-mode items, it drops to 25.2%, a decline of 18.2 percentage points. GPT-4o shows a 19.8-point drop, whereas humans show only a 2.2-point drop (Chiu et al., 2024). The authors interpret this as evidence of mode-seeking behavior: models over-converge on a single prototypical answer and miss multiple culturally valid alternatives.

Region-level Hard results further show uneven cultural coverage. Better-performing regions include North America at about 58.0%, South Asia at about 52.3%, West Europe at about 47.1%, and South Europe at about 45.4%. Worse-performing regions include South America at about 38.2%, East Europe at about 37.6%, and Middle East/West Asia at about 33.6%. The abstract separately emphasizes underperformance on North Africa, South America, and the Middle East (Chiu et al., 2024). This suggests that the benchmark is not merely testing generic knowledge difficulty, but is exposing structured geographic asymmetries.

5. Reuse in multilingual and reasoning studies

Later work has used CulturalBench as an instrument for studying the interaction between language, reasoning, and culture. In one study of multilingual large reasoning models, CulturalBench-Hard is treated as a non-reasoning cultural task. The authors keep the questions in English, but control the model’s internal “thinking language” by country-specific prefill tokens. Under this intervention, native-language thinking improves CulturalBench performance in South Europe by +1.0% on average and in Oceania by +2.9%, while South Asia shows the highest mean score at 57.3%. The same study reports that QwQ-32B is strongest in West Africa and that Chinese-developed model families underperform by 2.6 percentage points in East Asia (Tam et al., 23 May 2025).

Another study translates a subset of CulturalBench into Hindi, Chinese, Brazilian Portuguese, Swahili, and Hebrew to analyze language–culture entanglement in LLMs. That paper describes the full benchmark as containing 1,696 factual knowledge questions spanning 45 geographic regions, and evaluates a translated subset of more than 750 questions from 29 regions. For Qwen3-14B, cross-language differences are significant, with a Kruskal–Wallis statistic of $H = 45.5158$ and $p = 1.1395 \times 10^{-8}$. A placebo experiment using random strings yields $H = 1.0228$ and $p = 7.9574 \times 10^{-1}$, indicating that language behaves as a structured conditioning signal rather than as generic perturbation (Jain et al., 20 Jan 2026).

These downstream results change the interpretation of the original English-only design. They suggest that CulturalBench is not only a benchmark for factual cultural knowledge, but also a probe for how models couple culture to input language, internal reasoning language, and translation artifacts.

6. Interpretation, limitations, and later critiques

CulturalBench is frequently positioned in later literature as a “culture-as-knowledge” benchmark: a benchmark for cultural factual knowledge or trivia-style cultural knowledge, rather than for situated action, conversational style, or implicit value inference (Jain et al., 20 Jan 2026). A later conversational benchmark explicitly distinguishes itself from CulturalBench on that basis, arguing that factual cultural tests do not measure stylistic adaptation in realistic dialogue (Havaldar et al., 13 Oct 2025). This does not invalidate CulturalBench; it locates its construct more precisely.

Several limitations are explicit in the original benchmark. It is English-only. It relies on five annotators per question, with platform constraints that limited finer-grained subcultural recruitment except in some cases. It operates mostly at country or region granularity rather than at more detailed subcultural scales. It also shows that Easy is vulnerable to option-level heuristics, which is why Hard is indispensable for interpretation (Chiu et al., 2024).

Later reuse introduces additional reporting variation. In the multilingual reasoning study, CulturalBench-Hard is variously described as containing 1,200 questions, $N=4{,}907$, or $N=4{,}709$, with no train/dev/test splits reported for that study’s usage (Tam et al., 23 May 2025). This suggests that CulturalBench, especially in its Hard form, has been reused through multiple aggregation conventions. A plausible implication is that careful versioning and item-count documentation are necessary when comparing results across papers.

Within cultural evaluation more broadly, CulturalBench occupies the role of a robust, human-verified, difficulty-oriented benchmark for factual and pragmatic cultural knowledge. Its main contribution is methodological as much as empirical: it established Human-AI CulturalTeaming as a way to build adversarial cultural evaluation sets, demonstrated that multiple-choice cultural evaluation is easily gamed, and provided a harder option-level formulation that exposes structured blind spots in region coverage and multi-answer cultural reasoning (Chiu et al., 2024).