Progressing beyond Art Masterpieces or Touristic Clichés: how to assess your LLMs for cultural alignment?

Abstract: Although the cultural (mis)alignment of LLMs has attracted increasing attention -- often framed in terms of cultural bias -- until recently there has been limited work on the design and development of datasets for cultural assessment. Here, we review existing approaches to such datasets and identify their main limitations. To address these issues, we propose design guidelines for annotators and report on the construction of a dataset built according to these principles. We further present a series of contrastive experiments conducted with this dataset. The results demonstrate that our design yields test sets with greater discriminative power, effectively distinguishing between models specialized for a given culture and those that are not, ceteris paribus.

• The paper demonstrates that translation-based assessments can produce stereotypes and misaligned cultural outputs in LLM evaluations.
• It introduces the Tuguesice-PT dataset, a rigorously designed Q/A set that distinguishes culturally specialized LLMs from generic ones.
• The study reveals that explicit prompt engineering inflates accuracy scores, emphasizing the need for endogenous evaluation methods.

Assessing Cultural Alignment in LLMs: Dataset Design and Evaluation Paradigms

Motivation and Critical Analysis of Existing Approaches

The problem of cultural (mis)alignment in LLMs has become increasingly salient as these models are deployed in contexts where fine-grained cultural sensitivity is required. Existing cultural assessment methods predominantly rely on datasets either constructed through translation of generic or stereotypical content (e.g., BLEnD [Myung:2024:blend]), or through narrowly scoped factoid/question formats, typically covering encyclopedic or "high culture" knowledge. The paper identifies fundamental limitations with these approaches:

• Translational Equivalence and Stereotype Replication: Approaches such as BLEnD generate culturally “relevant” questions by translating templates and swapping entity references. This leads to questions (e.g., “What is the most popular indoor sport in Spain?”) that, when transposed, often yield ill-posed or culturally inappropriate instances (e.g., asking about US Thanksgiving traditions in China).
• Reliance on Generics and Stereotypes: Many questions either solicit generic, stereotype-prone answers (e.g., “What do young people drink at the nightclub?”), or reinforce “touristic clichés” that have minimal endogenous relevance.
• Encyclopedic and High-Culture Focus: Most datasets emphasize well-documented domains (historical events, major artworks, notable individuals) that are ubiquitous in the web-scale training corpora. Such content does not serve as a robust discriminator between culturally specialized and generic LLMs.
• Scope Leakage via Prompt Design: Explicitly including references to the target culture/country in both the question (e.g., “in China”) and the prompt introduces exogenous information that obfuscates the model’s endogenous cultural competence.
• Low Discriminative Power: Empirical results show that state-of-the-art LLMs rapidly saturate on such datasets, and the marginal difference between models fine-tuned for a specific culture and generic LLMs is minimal.

The review extends to other paradigms—such as personality/cultural survey simulation [alkhamissi-etal-2024-investigating, sukiennik2025evaluationculturalvaluealignment], culture-specific politeness protocols (e.g., TaarofBench [Sadr:2025:politelyinsist]), or proverb/name entity lists—which are shown to be either exceedingly narrow or methodologically misaligned with the objective of measuring substantive cultural alignment.

Dataset Design Principles for Cultural Alignment

The authors introduce a principled and operational set of guidelines for constructing cultural alignment test sets. The focus is explicitly on:

• Linguistic and Pragmatic Clarity: Questions must have bounded linguistic complexity, avoid subordinate clauses and rare vocabulary, elicit short factual answers, avoid yes/no formats, and be unambiguous.
• Factual and Endogenous Content: All Q/A pairs must concern information genuinely salient to members of the target culture, eschewing both global high-culture and local trivia unlikely to be widely known. The answer should be obvious to anyone educated within the culture, not specialist knowledge.
• Point-of-View Constraints: Annotators are instructed to avoid stereotypes, tourist tropes, external perspectives, or explicit cueing (proper names, direct mentions of the country), instead capturing naturally occurring, context-independent knowledge.
• Temporal Robustness and Generalization: Questions must concern information at least three years old, ensuring stability over time and resistance to fluctuation with political or social events.
• Discriminative Power: The dataset should highlight differences between models genuinely specialized for a culture and those that are merely competent at surface cues. Questions are constructed to not be easily answerable by Big Tech LLMs (e.g., ChatGPT) and to systematically distinguish similar but different national cultures (e.g., Portuguese versus Brazilian Portuguese).

This process led to the creation of the Tuguesice-PT dataset—a manually constructed, 327-entry Portuguese Q/A set, developed by native speakers via a rigorous adjudication pipeline.

Empirical Study: Comparative Model Assessment

The authors conduct a rigorous evaluation of Tuguesice-PT against a BLEnD-derived Portuguese set (BLEnD-PT), assessing several LLMs covering a spectrum of pretraining, fine-tuning regimens, and model sizes (e.g., Gervásio series fine-tuned for Portuguese, Llama baselines, Gemini 2.5 Flash, Mistral, and Sabiá 7B fine-tuned for Brazilian Portuguese).

A critical variable in the evaluation is the “oracle” scope prompt—an explicit instruction framing the context as Portuguese, versus a plain prompt. The results elucidate several key observations:

• Enhanced Discriminative Power: The Tuguesice-PT set yields substantially larger accuracy gaps between specialized and generic models in the plain prompt setting (e.g., delta of 14 percentage points between Gervásio 70B and Llama 70B, compared to <2 points on BLEnD-PT). This verifies that the dataset indeed measures what it purports to—a model’s genuine cultural competence.
• Reduced Ceiling Effects: Top plain prompt scores on Tuguesice-PT (e.g., Gervásio 70B: 39.76%) are well below those on BLEnD-PT, avoiding rapid saturation and maintaining utility for future benchmarking as models improve.
• Oracle Prompt Bias: The inclusion of explicit context instruction leads to artificial score inflation (up to 42 points), demonstrating how existing evaluation paradigms can produce misleadingly high accuracy even when the model’s intrinsic knowledge is limited.
• Orthogonality to Model Size: Larger models retain an performance advantage, but size alone does not guarantee success on the endogenous dataset, underscoring the value of cultural specialization.

Implications for Theoretical and Practical Development

From a theoretical perspective, this work illustrates the challenge of capturing "cultural alignment" in a manner robust to both annotation artifacts and transfer from web-scale corpora. It exposes the inadequacy of both translation-based and encyclopedic approaches, and the necessity of endogenous, context-independent Q/A construction. This has major implications for both the development of benchmarks and for claims regarding “culture-aware” LLMs.

Practically, the guidelines—particularly those enforcing implicitness and specificity—are generalizable across languages and cultures, providing a methodology for constructing similarly discriminative sets for other underrepresented or low-resource cultures. The findings also suggest urgent caution when interpreting LLM “cultural competence” as presented by existing benchmarks: artificial prompt engineering or reliance on global stereotypes can mask significant deficiencies.

Furthermore, the results call for refined evaluation, robust to prompt leakages and annotator bias, and suggest that future surveys of cultural alignment [Khan2025] must foreground dataset construction paradigms as central to any substantive claims. This also has potential to inform regulatory efforts and "digital sovereignty" debates [radu2021, Mugge02082024], supplying empirical grounding to assessments of national or community-specific alignment in foundational models.

Conclusion

The study delivers a comprehensive critique of current practices for evaluating LLM cultural alignment and demonstrates, via systematic dataset development and model evaluation, how existing paradigms fail to provide robust measurement. The proposed guidelines operationalize an endogenous, culturally situated, and discriminative approach to test set construction. Empirical evaluation with contemporary LLMs verifies that this paradigm can reveal fine-grained distinctions in cultural competence, unobservable in current benchmarks. Both immediate practical applications and future regulatory or theoretical studies of LLMs would benefit from adopting these principled methodologies.