Culture-Related Open Questions (CROQ)

• CROQ is a framework that defines open-ended prompts to evaluate AI's ability to interpret and generate culturally grounded responses covering beliefs, norms, and practices.
• It employs methodologies like human-AI red-teaming, curriculum-driven extraction, and clustering to create structured taxonomies and representative questions.
• Evaluation relies on advanced metrics such as information-theoretic measures, alignment distance, and pragmatic context sensitivity to uncover performance gaps and biases.

Culture-Related Open Questions (CROQ) refer to the class of open-ended, contextually grounded inquiries that probe a model’s or system’s ability to interpret, reason over, or generate content concerning the beliefs, norms, practices, values, and artifacts characteristic of human societies. In computational terms, CROQs operationalize cultural knowledge and reasoning requirements in AI benchmarks, highlighting both explicit and implicit phenomena such as tradition, etiquette, symbolic meanings, and subcultural nuance. The formulation, evaluation, and impact of CROQs have become central to measuring, improving, and auditing cultural alignment in LLMs, vision–LLMs, and interactive AI systems.

1. Taxonomies and Representative CROQs

Recent works converge on structured taxonomies for CROQ to ensure comprehensive coverage and systematic evaluation. The “Why are all LLMs Obsessed with Japanese Culture?” study defines an 11-domain, 66-subtopic taxonomy spanning beliefs, values, identity, social structure, knowledge, arts, food, geography, politics, health, media, history, and economy, with each subtopic supported by exemplar open-ended prompts (e.g., “What values shape family life {in region/place}?”, “What is the role of neighbors {in region/place}?”) (Landa et al., 23 Apr 2026). Other resources follow variants on this schema, such as MyCulture’s six-pillar coverage (arts, attire, customs, entertainment, food, religion) for Malaysia (Hew et al., 7 Aug 2025), the 17-topic, 45-region grid of CulturalBench (Chiu et al., 2024), and the 9-category, 400-topic GlobalCultureQA (Feng et al., 26 May 2025).

Multiple construction methodologies are used:

• Human-AI red-teaming: domain-expert scenario writing, AI draft generation, adversarial revision, and iterative review to surface authentic and nuanced cultural knowledge gaps (Chiu et al., 2024).
• Curriculum-driven extraction: multi-agent parsing of national learning objectives to generate aligned questions and answers across proficiency levels (Yoo et al., 8 Jan 2026).
• Open-domain clustering: large-scale clustering and classification (e.g., using BERT-derived models) to automatically source and select society/culture-labeled questions from open user submissions (Xu et al., 2021).

A central tenet is that generic, trivia-style, closed-format questions inadequately probe cultural reasoning, necessitating open, often generative prompts and multi-faceted answer formats (Oh et al., 1 Sep 2025, Hew et al., 7 Aug 2025).

2. Metrics, Evaluation Protocols, and Information-Theoretic Justifications

Evaluation of CROQ task performance in AI requires metrics that capture both factual accuracy and the expressivity and nuance of open-ended outputs.

• Information-theoretic discriminative power: MyCulture demonstrates that open-ended MCQ formats with $k$ correct out of $n$ (without options) yield randomly correct rates $P_{open} = 1 / \binom{n}{k}$, inducing much higher information value (e.g., for $n=8$, $k=4$, $I_{open}\approx 6.13$ bits vs. $I_{MCQ}=2$ bits) (Hew et al., 7 Aug 2025).
• Alignment distance: Wasserstein distance between model and human cultural distributions in binary-choice settings, as in the GPT-4o case study, enables quantification of (mis)alignment but is not sufficient for unconstrained, generative scenarios (Bravansky et al., 13 Jan 2025).
• Knowledge-unit F1 (CulFiT): Precision and recall over atomic, fact-level knowledge decomposed from generated answers produce per-language, per-topic scores for both interpretable and aggregate cultural reasoning evaluation (Feng et al., 26 May 2025).
• Pragmatic context sensitivity (PCS): The fraction of explicit style shift in pragmatic adaptation captured under implicit cueing reflects LLMs’ ability to infer cultural norms in the absence of direct instructions (Nasim et al., 20 Apr 2026).

Human annotation pipelines, multi-annotator agreement statistics (e.g., Cohen’s $\kappa$), minority-vote consensus, and LLM-as-a-judge paradigms (with fine-tuned evaluation rubrics) are common. Evaluation setups distinguish between easy (multiple-choice, high baseline) and hard (binary, open, multi-answer) settings to surface model overfitting or shortcut exploitation (Chiu et al., 2024).

3. Methodologies for Dataset Construction and Bias Analysis

CROQ benchmark creation has seen the deployment of scalable, semi-automated pipelines:

• Multi-agent parsing: Extraction of learning outcomes and achievement criteria from national curricula, followed by iterative LLM-driven question and answer generation, paraphrasing, multilingual translation, and stratified answer production (basic to advanced) with human review—illustrated in the CuCu/KCaQA pipeline (Yoo et al., 8 Jan 2026).
• Multilingual, fine-grained synthesis: Pipeline approaches (e.g., CulFiT) leverage corpora of culture-rich statements as seeds, LLM-generated knowledge paragraphs, auto-sampled question generation, translation/backtranslation for hallucination filtering, knowledge-unit decomposition, and meta-critique annotation in multiple languages (Feng et al., 26 May 2025).
• Red-teaming and annotation: Staged validation with region/topic-specific human annotators filters scenario/question pools, ensures majority label agreement, and admits ambiguity (multi-answer or “unclassifiable” rates) as a first-class evaluation signal (Chiu et al., 2024, Bravansky et al., 13 Jan 2025).
• Programmatic classification: BERT-family topic classifiers, trained on large-topic datasets (e.g., Yahoo! Topics), are directly transferred for society/culture labeling of raw user queries, supporting scalable analysis of the societal distribution of real open questions (e.g., ~12% of science center visitor questions labeled as culture) (Xu et al., 2021).

Bias audits systematically examine structural (format) bias (freeform vs. schema-constrained outputs), language bias (cross-lingual prompt and answer effects), and region/topic bias (over/underrepresentation in web or instruction datasets). Example findings include dramatic drops in open-ended scores (≥17% collapse across models versus closed form), cross-lingual disparities in model performance tied to pretraining resource imbalances, and emergence of regional “obsessions” after SFT phase (e.g., LLMs defaulting to Japanese or U.S. cultural references regardless of question context) (Hew et al., 7 Aug 2025, Jain et al., 20 Jan 2026, Landa et al., 23 Apr 2026).

4. Critical Findings on Model Performance and Cultural Reasoning Failures

Frontier evaluations of CROQ resources expose persistent gaps and pathologies:

• Open-ended and multi-hop questions reveal a stark overestimation of LLM cultural competence in close-form settings; open-ended benchmarks like MyCulture, GlobalCultureQA, and ID-MoCQA show performance drops of 17–43 percentage points compared to closed-format, with regional models often losing their local advantage under complex reasoning (Hew et al., 7 Aug 2025, Permadi et al., 3 Feb 2026, Feng et al., 26 May 2025).
• Cross-lingual and context entanglement: LLMs not only provide lower quality answers in low-resource languages, but also shift the very cultural worldview expressed in responses depending on the input language; even after translation, >70% of responses remain indexically tied to the query language’s associated culture (Jain et al., 20 Jan 2026).
• Mode-seeking and shortcut heuristics: Multi-answer questions consistently expose a mode-seeking bias, with LLMs underperforming by 18–20% versus single-answer questions. Simple string-matching heuristics (e.g., choosing the candidate closest in embedding space to the country name) can reach high accuracy without genuine cultural understanding (Chiu et al., 2024).
• Implicit adaptation and pragmatic style: LLMs largely fail to deploy culturally appropriate pragmatic features in response to implicit situational cues, recovering only ≈20% of their maximum explicit adaptation capabilities—particularly for collectivism and uncertainty avoidance dimensions (Nasim et al., 20 Apr 2026).

Tables summarizing performance gaps and bias effects:

Setting	Closed-Form (%)	Open-Ended (%)	Drop (pp)
GPT-4o MyCulture	67.9	38.4	–29.5
Llama3-2.5B	61.4	18.7	–42.7
Qwen3-1.7B	36.7	19.4	–17.3

Language	Bench Accuracy (CulturalBench, Qwen3-14B)
English	~82%
Swahili	~58%
Hindi	~65%

Bias Phase	Entropy (H)	Top-2 Country Share (%)
Pretraining (Base)	~0.84	US+JP < 30
SFT	~0.63	US+JP > 60
RLHF/Instr.	~0.64	US+JP > 60

5. Implications and Open Research Questions

CROQ research systematically uncovers the following agenda:

• Dynamic and adaptive value representation: How can models be aligned to continually evolve with changing cultures and community stances, moving beyond static survey-derived value distributions? (Bravansky et al., 13 Jan 2025)
• Interpretability and holistic metrics: How can refusals, ambiguity, uncertainty, and argument richness in open-ended outputs be incorporated into alignment metrics, replacing one-dimensional accuracy with multidimensional behavioral rubrics? (Bravansky et al., 13 Jan 2025, Feng et al., 26 May 2025)
• Implicit value inference and stylization: What architectural choices, training objectives, and data augmentations will enable robust pragmatic and metacognitive adaptation to contextually signaled—but not spelled-out—cultural cues? (Nasim et al., 20 Apr 2026, Liu et al., 1 Apr 2025)
• Cross-lingual equity and language–culture decoupling: How can data curation, fine-tuning, and evaluation guard against the entanglement of language choice with cultural representation, preventing systemic disadvantage to non-dominant language users? (Jain et al., 20 Jan 2026, Hew et al., 7 Aug 2025)
• Participatory evaluation and stakeholder involvement: How can benchmarks, data collections, and interpretation pipelines be democratized, surfacing “unknown unknowns” and pluralistic, community-grounded notions of cultural acceptability and offense? (Oh et al., 1 Sep 2025)

The field lacks consensus on universal best practices for open question elicitation, scenario realism, and culturally situated answer annotation. The need for participatory methods, thick evaluation (qualitative and quantitative blend), and transparency in researcher positionality is increasingly emphasized to avoid self-reinforcing bias and epistemic injustice (Oh et al., 1 Sep 2025).

6. Future Directions and Methodological Recommendations

Best practices emerging from CROQ work emphasize:

• Open-ended, multi-modal probe formats (including vision–language) leveraging both parametric memory and in-context adaptation. Benchmarks like CROPE provide structured multilingual settings to distinguish model knowledge from context-driven learning (Nikandrou et al., 2024).
• Algorithmic frameworks for fine-grained, interpretable evaluation: decomposing outputs into atomic knowledge units, supporting precision/recall scoring, and explicit critique synthesis (Feng et al., 26 May 2025).
• Discovery-driven taxonomy building and automated data expansion across low-resource, multilingual, and underrepresented cultural intersections, with continual iteration and quality filtering (Landa et al., 23 Apr 2026, Yoo et al., 8 Jan 2026).
• Cross-sectional and longitudinal audits of post-training (SFT, RLHF) distributional shift in cultural coverage, with entropy/diversity metrics to assess outcome sensitivity to pretraining and alignment objectives (Landa et al., 23 Apr 2026).
• Integration of cross-cultural, use-case-specific stakeholder engagement into both the design of task scenarios and the interpretation of ambiguous outputs—critical in high-stakes AI deployment domains (Bravansky et al., 13 Jan 2025, Oh et al., 1 Sep 2025).

In sum, CROQ research defines an evolving, technically rigorous domain at the intersection of cultural studies and computational evaluation, driving methodological and empirical advances in constructing, deploying, and critiquing culturally aware AI systems. The landscape remains highly dynamic, with explicit calls for more dynamic, adaptive, and participatory approaches to both the creation and auditing of cultural competence benchmarks.