Toward Culturally Grounded Natural Language Processing

Abstract: Recent progress in multilingual NLP is often taken as evidence of broader global inclusivity, but a growing literature shows that multilingual capability and cultural competence come apart. This paper synthesizes over 50 papers from 2020--2026 spanning multilingual performance inequality, cross-lingual transfer, culture-aware evaluation, cultural alignment, multimodal local-knowledge modeling, benchmark design critiques, and community-grounded data practices. Across this literature, training data coverage remains a strong determinant of performance, yet it is not sufficient: tokenization, prompt language, translated benchmark design, culturally specific supervision, and multimodal context all materially affect outcomes. Recent work on Global-MMLU, CDEval, WorldValuesBench, CulturalBench, CULEMO, CulturalVQA, GIMMICK, DRISHTIKON, WorldCuisines, CARE, CLCA, and newer critiques of benchmark design and community-grounded evaluation shows that strong multilingual models can still flatten local norms, misread culturally grounded cues, and underperform in lower-resource or community-specific settings. We argue that the field should move from treating languages as isolated rows in a benchmark spreadsheet toward modeling communicative ecologies: the institutions, scripts, translation pipelines, domains, modalities, and communities through which language is used. On that basis, we propose a research agenda for culturally grounded NLP centered on richer contextual metadata, culturally stratified evaluation, participatory alignment, within-language variation, and multimodal community-aware design.

• The paper finds that multilingual capability and cultural competence are empirically distinct for language models;
• Multilingual evaluations often conflate the two, highlighting the need for culturally sensitive metrics, rich contextual metadata, and community-grounded data for accurate assessments;
• Key success factors for achieving cultural competence include native raters, culturally tailored supervision, and multimodal benchmarks

Overview and motivation

This paper is an overview of more than 50 works published between 2020 and 2026, spanning multilingual performance inequality, cross-lingual transfer, culture-aware evaluation, cultural alignment, multimodal benchmarking, benchmark-design critique, and community-grounded data practices. Its central claim is that multilingual capability and cultural competence are empirically distinct properties of LLMs, and that the field's dominant evaluation practices—language-labeled leaderboards over translated benchmarks—systematically conflate the two. The author proposes reframing languages not as isolated rows in a benchmark spreadsheet but as elements of "communicative ecologies": the institutions, scripts, translation pipelines, domains, modalities, and communities through which language is actually used.

The paper makes three contributions. First, it connects the multilingual transfer literature to culture-oriented evaluation work that has largely been discussed separately. Second, it identifies recurring empirical patterns across these literatures—most notably that scale matters but does not suffice, while tokenizer behavior, prompt language, benchmark design, local supervision, and modality all materially affect cultural performance. Third, it articulates a research agenda built around contextual metadata, culturally stratified evaluation, participatory alignment, within-language variation, and multimodal community-aware design.

From multilinguality to cultural competence

The synthesis rests on two robust findings from the multilingual literature. First, language technology remains unequally distributed: resource concentration, benchmark availability, and deployment are heavily skewed toward a small subset of the world's roughly 7,000 languages (Collaboration, 2020, Hadden et al., 2022). Second, widening coverage alone does not remove disparity. When evaluation extends to 205 languages and dialects in SIB-200, large gaps persist between high- and low-resource languages, with unseen languages and underrepresented families (e.g., Nilotic, Atlantic-Congo) performing worst (Jorgsson, 2023). The NLLB effort similarly shows persistent quality disparities even at massive scale (Team et al., 2022).

Work on cross-lingual transfer explains part of this pattern through pretraining data size, lexical overlap, script, tokenization, and typology (Yang et al., 2023). Decision-tree analysis over six multilingual models on SIB-200 finds pretraining data size most influential for seen languages, with script type and language family becoming more crucial for unseen ones (Liu et al., 2024); regression analysis with SHAP values adds token similarity and country similarity as pivotal factors beyond raw data quantity (Garcia-Saenz et al., 2 Jan 2025). Subword token alignability further refines token-overlap-based predictions when scripts differ (Oca et al., 7 Apr 2025).

The paper's key argumentative move is that these predictors answer a language-centric question—"when does one language help another?"—but say little about whether a model understands local norms, values, or region-specific knowledge once transfer succeeds technically. A model can reuse familiar subwords and achieve respectable accuracy yet fail on culturally loaded metaphors, local emotional expressions, or region-specific entities. Multilinguality is therefore necessary but not sufficient for cultural competence.

A second conceptual lesson concerns operationalization. Surveys of more than 90 papers find that none explicitly define "culture"; instead, studies probe partial proxies organized along demographic and semantic dimensions (Yi et al., 2024). Related taxonomies emphasize that culture spans within-human, between-human, and extra-human dimensions (Norden et al., 6 Feb 2025), and position work argues that static, nationalized proxies reflect a theoretical gap best addressed through sociocultural theory and localization framing (Kankanamge et al., 29 May 2025).

What recent benchmarks show

Text and value-oriented evaluation

Global-MMLU demonstrates that machine-translated benchmarks carry over source-culture assumptions and that model rankings on full MMLU change when restricted to culturally sensitive subsets; state-of-the-art models excel primarily at Western-centric concepts (Gadde et al., 25 Feb 2025). CDEval probes six cultural dimensions across seven domains (Dias et al., 2024), and WorldValuesBench—built from 94,728 World Values Survey participants into over 20 million examples—shows the task is difficult: GPT-3.5 Turbo achieves under 0.2 Wasserstein distance from human answer distributions on only 75.0% of questions, and Alpaca-7B on just 11.1% (Hozzová et al., 2024). CulturalBench, constructed via human-AI red-teaming with 1,696 verified questions across 45 regions, reports a stark gap: humans reach 92.4% accuracy while frontier models range from 28.7% to 61.5% on its hard version, with systematic weakness on North Africa, South America, and the Middle East (Cheung et al., 27 May 2025). Nunchi-Bench shows models recognize Korean cultural facts but fail to apply them situationally (Ballesta-Yagüe, 21 May 2025).

Cultural bias work reinforces the point. In Arabic, multilingual and monolingual models prefer Western-associated entities and generate culturally inappropriate associations (Makarov et al., 2024); follow-up analysis traces part of this bias to frequency-based tokenization interacting with polysemous entities and script-sharing languages, worsening with larger Arabic vocabularies (Tassopoulou et al., 10 Apr 2025). Most directly, a linear mixed-effects comparison against World Values Survey data across four languages finds no consistent relationship between language capability and cultural alignment across Gemma, OLMo, and OpenAI model families (Huang et al., 26 Sep 2025)—a strong empirical basis for evaluating the two axes separately.

Alignment results also show movement is possible but uneven. Prompt language and persona framing shift cultural outputs (Macula et al., 2024, Kulveit et al., 28 Jan 2025), and self-pluralising or culture-joint tuning improves alignment to multiple cultures without collapsing general ability (Ermakov, 27 Apr 2025).

Multimodal, interactional, and local evaluation

Multimodal benchmarks consistently expose failures invisible in text-only evaluation. CulturalVQA (2,378 image-question pairs across 11 countries) reveals strong VLM understanding of North America but significantly weaker coverage of Africa (Watanabe et al., 2024). GIMMICK, covering 144 countries and 728 cultural facets across 20 LVLMs and 11 LLMs, finds strong Western bias, better knowledge of tangible than intangible aspects, and benefit from external geographic cues (Qi et al., 29 May 2025). WorldCuisines—the largest multicultural VQA benchmark to date, with over 1 million data points across 30 languages—shows VLMs struggle with adversarial location contexts and specific regional cuisines (Cui et al., 16 Mar 2025). DRISHTIKON (64,000+ aligned text-image pairs across 15 Indian languages) exposes weaknesses in reasoning over low-resource languages and less-documented traditions (Kong et al., 10 May 2025), and CaMMT shows visual context generally improves translation of culturally-specific items (Afshar et al., 23 Oct 2025).

Interactional settings add further failure modes. CULEMO shows cross-cultural emotion understanding varies substantially across six languages, and notably that English prompting with explicit country context often outperforms in-language prompts (Gómez et al., 19 May 2025). A conversational framework grounded in sociocultural theory argues existing benchmarks misalign with real interactional challenges and shows top LLMs struggle with stylistic sensitivity and subjective correctness (Xie et al., 30 Oct 2025). Qualitative assessment of seven LLMs on code-mixed Swahili/English/Sheng WhatsApp sentiment finds high F1 scores can coexist with poor contextual nuance and opaque explanations (Liu et al., 18 Jul 2025). Within-region variation matters too: LORAXBENCH across 20 Indonesian languages shows register changes—especially high-politeness Krama Javanese—degrade performance, and no clear advantage for region-specific over general multilingual models (Kadelka, 30 Oct 2025). Culturally nuanced story generation for Javanese and Sundanese shows LLM-generated training data outperforming machine-translated alternatives while still lagging native writing in coherence (Sun et al., 30 Oct 2025).

Adaptation methods and their dependence on grounded supervision

Targeted intervention demonstrably helps. CARE shows that a modest amount of native-speaker preference data (3,490 questions, 31.7k judged responses) outperforms larger generic preference sets, though models with stronger initial cultural performance benefit more—implying regional gaps may widen (Hromadka et al., 29 May 2025). CLCA uses simulated social interactions toward target cultural values (Norden et al., 6 Feb 2025); CulFiT synthesizes multilingual critique data with fine-grained rewards to achieve state-of-the-art open-source cultural alignment (Nasibova et al., 29 May 2025); CultureSPA aligns pluralistically via self-generated supervision (Ermakov, 27 Apr 2025); and CulturalGround, 22 million VQA pairs across 42 countries and 39 languages, yields CulturalPangea with an average +5.0% gain on culture-focused multimodal benchmarks without degrading mainstream vision-language performance (Zhang et al., 28 May 2025).

The shared lesson is that cultural competence does not emerge automatically from multilingual scale; it requires native raters, culture-specific task framing, or carefully designed supervision. The paper treats this data burden as part of the science rather than an implementation detail.

Evaluation paradigms and ecological validity

A growing critique strand argues the bottleneck is partly what evaluations treat as culture. Position work describes a "trivia-centered paradigm" reducing culture to static facts (Podolak et al., 28 May 2025), and empirically, open-ended assessment yields different conclusions from closed multiple-choice formats, with even choice reordering producing inconsistent outputs (Ahmed et al., 27 May 2025). A survey of 105 cultural-alignment benchmarks finds severe imbalance: Region (37.9%) and Language (28.9%) dominate group representation, and 97.1% of datasets adopt majority-focused awareness scopes (Thomazini et al., 26 May 2025). An anthropological audit of 20 benchmarks identifies recurring issues including treating countries as cultures and overlooking within-culture diversity (Do et al., 27 Jan 2026).

Ecological validity emerges as part of the evaluation construct itself. LLM web agents achieve less than 10% awareness coverage and over 40% violation rates in live browsing settings, performing markedly worse than in non-agent setups (Danev et al., 26 Apr 2025). Video benchmarks show cultural competence requires joint interpretation of scene, gesture, speech, and event structure across 14 languages (Barkataki et al., 29 May 2025). Native-speaker validation sharpens the contrast: DaKultur finds automatically translated Danish data inadequate and that training on native-speaker data more than doubles response acceptance rates (Liu et al., 29 May 2025). Community-engaged resources such as HESEIA (46,499 sentences co-designed by 370 teachers and 5,370 students across 189 Latin American schools) surface stereotypes unrecognized by current LLMs (Tan et al., 29 May 2025), and SAFARI contributes 3,534 English and 3,206 native-language stereotypes from four sub-Saharan African countries via community-engaged telephonic surveys (Deng et al., 27 Jan 2026). Scaling efforts such as CultureInstruct (430K instructions) improve cultural benchmarks substantially—outperforming GPT-4o Mini and GPT-4o by 18.47% and 13.07% average relative improvement respectively—but raise provenance and normalization concerns about whether scaling erases local detail (Papicchio et al., 21 Apr 2025, Zhou et al., 27 Jan 2026).

Synthesis: recurring patterns

Four patterns recur across the literature. Data quantity remains necessary but insufficient: coverage predicts performance strongly, but the path is mediated by tokenizer allocation, token alignability, prompt language, and pretraining composition. Benchmark design imports external assumptions: translation flattens curricula, answer spaces, and common knowledge; survey instruments narrow culture to encoded dimensions; even locally authored benchmarks face question-writing and disagreement-handling choices. Local supervision and participation change what is measured: native raters and co-design improve both performance and diagnostic validity, so participation is epistemic infrastructure rather than an ethical add-on. Culture is distributed across modalities and within-language variation: it appears in images, food, rituals, emotion repertoires, politeness registers, dialects, and code-mixing, defeating any one-language-one-culture mapping.

Toward layered evaluation protocols

The paper distills these findings into a five-layer protocol separating questions current practice collapses:

Layer	Common shortcut	Stronger practice
Representation audit	Country/language labels only	Report authorship, variety, translation pipeline, minority/subcultural coverage
Elicitation diversity	Multiple-choice/Likert only	Mix closed items, open generation, pairwise judgments, qualitative coding
Ecological validity	Static text-only QA	Add conversation, agents, image/video, region-specific slices
Community validation	Expert-only or automatic scoring	Native-speaker review, disagreement analysis, participatory co-design
Adaptation reporting	"Culture-tuned" as black box	Publish supervision provenance, target population, trade-offs

Two implications deserve emphasis. First, claims about cultural competence should be scoped to settings actually tested, since performance degrades sharply when models must integrate local cues in realistic contexts. Second, cultural resources require temporal maintenance—versioning, refresh cycles, and revalidation—because local salience shifts faster than generic linguistic competence, and static releases risk freezing one historical snapshot.

Research agenda

The proposed agenda comprises eight commitments: distinguish multilingual coverage from cultural competence in reporting; publish richer contextual metadata and auditable benchmark slices rather than a scalar "culture score"; use mixed elicitation protocols distinguishing norm violation from plausible variation; prefer native-authored, locally validated data as core infrastructure; evaluate multimodality and within-language variation; treat alignment as ongoing localization rather than one-time correction; make participation and governance part of the experimental setup; and maintain cultural resources as living infrastructure. Each is anchored in specific empirical results reviewed above rather than asserted as principle.

Limitations

The paper is candid that it is an overview, not a new experimental study, so its claims inherit the unevenness of the underlying literature across regions, modalities, and tasks; many cited benchmarks target few countries or languages despite broad conceptual claims. It also follows the literature in relying on partial proxies—countries, prompt languages, surveys, artifacts—and explicitly disclaims that any single proxy captures culture exhaustively; "communicative ecologies" is offered precisely to resist such over-identification. Finally, given the pace of the subfield, the agenda is presented as a living program rather than a closed taxonomy. The ethical discussion adds that participatory alignment is not a panacea: communities are internally diverse, disagreements are substantive, and norms change, so whose judgments are represented must be reported alongside any alignment gains.

Conclusion

The paper's central, well-supported conclusion is that multilingual capability does not automatically yield multicultural competence: across text, values, emotion, conversation, multimodality, and low-resource settings, models transfer technically while failing culturally. The immediate implication for the field is methodological—culture-sensitive evaluation should become a standard reporting axis, with papers specifying whether items were translated or native-authored, standard or non-standard varieties, static or interactive settings, and which communities validated labels. Absent that shift, evaluation will continue to reward globally legible systems that remain locally brittle.