CultureSynth-7 Synthetic Benchmark

• CultureSynth-7 is a multilingual benchmark designed to evaluate LLMs' cultural competence through a systematic taxonomy and retrieval-augmented generation process.
• It uses a hierarchical taxonomy derived from global classification systems to cover broad cultural domains and fine-grained nuances across seven languages.
• The benchmark offers both a manually validated mini set and an extensive max set, revealing performance gaps and biases among leading language models.

CultureSynth-7 Synthetic Benchmark is a multilingual evaluation suite designed to assess the cultural competence of LLMs through a systematically synthesized set of culturally relevant question-answer pairs. Developed within the CultureSynth framework (Zhang et al., 13 Sep 2025), CultureSynth-7 leverages a hierarchical taxonomy-guided process combined with retrieval-augmented generation (RAG) to create over 19,000 benchmark entries across seven languages. Its methodology draws on recent research in synthetic data generation, data-centric AI, and benchmarking practices to produce data that is both diverse and rigorously validated for cultural specificity, clarity, and answer quality.

1. Hierarchical Multilingual Cultural Taxonomy

CultureSynth-7 employs a hierarchical taxonomy, which serves as the backbone for topic selection and cultural content coverage. The universal tier is derived from five major library classification systems: DDC, UDC, LCC, CLC, and NDC. This yields 12 primary domains including Social Sciences, Philosophy and Psychology, Religion and Theology, Political Science, Law, Education, Language, Literature, Medicine, Applied Sciences and Technology, Arts, and Recreation/Sports/Entertainment. These are further refined into 130 secondary topics to ensure granularity.

A second tier is constructed using LLM-driven role-playing generation, which expands each primary topic into over 300 fine-grained tertiary topics and aggregates more than 1,000 culture-specific keywords per language or country context. This taxonomy enables branching from universal topics to local nuances, underpinning both broad and deep cultural question synthesis and supporting multi-lingual representation.

2. Retrieval-Augmented Generation Methodology

The creation of CultureSynth-7 instances relies on a RAG pipeline that consists of four primary steps:

1. Keyword Translation: Each keyword from the taxonomy is translated into the target language.
2. Document Retrieval: Cultural context-specific web content (typically Wikipedia pages) is scraped in both English and the target language.
3. Content Filtering: Special prompts (e.g., "Prompt for determining whether the retrieved page is related to the keyword") are used to ensure strict relevance and the presence of cultural content.
4. Knowledge Extraction & Synthesis: LLMs extract factual knowledge into structured JSON, then generate questions and answers using separate, rigorously designed prompts.

This pipeline ensures the resulting data is grounded in authentic, verifiable cultural contexts. Questions generation prompts push for depth (e.g., instructing LLMs to answer “in depth from a local’s perspective”), maximizing both relevance and linguistic precision.

3. Benchmark Structure, Scale, and Quality Control

CultureSynth-7 contains a total of 19,360 QA pairs, spanning Arabic, Spanish, French, Japanese, Korean, Portuguese, and Chinese. For development and evaluation flexibility, two disjoint sets are defined:

• CultureSynth-7-mini: 4,149 entries, each manually validated for question clarity (average 95.8%), cultural relevance (83.9%), and answer quality (98.8%).
• CultureSynth-7-max: 15,211 additional entries held privately to maintain integrity for future evaluation.

Native speaker verification and balanced topic distributions confirm the reliability and systematic design of the benchmark, supporting diverse model evaluation, training, and finetuning in multicultural contexts.

4. Evaluation of LLMs and Benchmarking Metrics

CultureSynth-7 has been used to evaluate 14 major LLMs. The results demonstrate:

• Performance Stratification: ChatGPT-4o-Latest leads across most domains, followed by Qwen2.5-72B-Instruct. No model below 3B parameters achieved satisfactory cultural competence, establishing a threshold for model capacity.
• Architectural Biases: Mixture-of-experts architectures excel in cultural knowledge retrieval; dense transformer models perform better with long-context processing.
• Geographic Disparities: Models diverge in performance across cultures. For instance, ChatGPT-4o displays lower efficacy on East Asian (Japanese, Korean) cultural tasks, while Claude-3.5-Sonnet struggles with Arabic and Korean.

Model comparison incorporates both human evaluation and automated pairwise LLM-judged scoring. Relative strengths are quantified by the net win rate formula:

$$\text{net win rate} = \frac{N_{\text{target wins}} - N_{\text{baseline wins}}}{N_{\text{total}}}$$

where $N_{\text{target wins}}$ and $N_{\text{baseline wins}}$ denote pairwise wins in LLM-judged comparisons.

5. Integration of Data-Centric and Synthetic Data Benchmarking Practices

The design and evaluation methodology align with recent advances in synthetic benchmark generation (Hansen et al., 2023, Maheshwari et al., 18 Sep 2024). Data-centric AI profiling (e.g., using Cleanlab, Data Maps, Data-IQ) is used for segmentation of dataset samples into profiles reflecting “easy,” “hard,” and “ambiguous” instances. These profiles guide generative model training and postprocessing, producing synthetic datasets with representative sample quality distributions.

Evaluation integrates not just statistical fidelity (inverse KL-divergence, MMD, Wasserstein distance) but also tasks measuring classification, feature selection, and model ranking consistency. For NLP benchmarks, absolute performance (Mean Squared Performance Difference, MSPD) and relative ranking fidelity (Spearman’s Rank Correlation, SRCC) are combined with bias factor analysis—quantifying whether a model favors tasks on its own generated data.

Table: Representative Benchmark Design Features

Feature	CultureSynth-7 Benchmark	Related Synthetic Tabular/NLP Benchmarks
Taxonomy-based coverage	12 primary, 130 secondary	Rare or absent
Multilingual support	7 languages	Single/dual language common
Retrieval-augmented generation	Yes	Traditional generation or prompt-based LLM
Manual validation (mini set)	Yes	Variable
Bias factor evaluation	Recommended for NLP	Emerging technique

A plausible implication is that direct integration of these profiling and evaluation methods into cultural Q&A synthesis will identify nuanced model biases and strengths in ways not captured by traditional divergence metrics alone.

6. Implications for Culturally Competent AI Systems

CultureSynth-7 establishes a scalable blueprint for assessing and improving the cultural competence of AI models:

• Systematic Cultural Coverage: Its hierarchical taxonomy and RAG synthesis produce broad yet precise cultural representation, facilitating global deployment and adaptation.
• Reduced Manual Annotation: The pipeline leverages factual retrieval and prompt engineering to automatically generate high-quality, diverse data, minimizing annotation bottlenecks.
• Policy and Social Impact: By highlighting geographic disparities and parameter thresholds, CultureSynth-7 enables targeted model improvement and more responsible AI designs for international contexts.

This suggests future benchmarks may adopt similar taxonomic structuring and retrieval augmentation strategies to ensure validity, scalability, and fairness in culturally oriented AI evaluation.

7. Limitations and Future Research Directions

While the CultureSynth-7 Synthetic Benchmark succeeds in combining taxonomy-guided coverage with RAG-driven synthesis, several limitations remain:

• Fine-grained Cultural Nuances: Automated synthesis may miss subtle local or minority group perspectives that exceed available retrieval sources or LLM generalization capacity.
• Model Bias and Generalization: Bias factor analysis has yet to be extended systematically to cultural competency tasks, and the interplay of model size, architecture, and cultural knowledge retention warrants further paper.
• Continuous Expansion: The semi-automated pipeline requires ongoing refinement to incorporate newly emerging cultures, languages, and topics without drift or loss in annotation quality.

A plausible implication is that combining human-in-the-loop and advanced multimodal retrieval could further strengthen the benchmark’s representativeness and coverage in future iterations.

CultureSynth-7’s synthesis of multilingual, taxonomy-driven, and RAG-assisted generation methodologies marks a comprehensive paradigm for benchmarking cultural competence in LLMs, with broad implications for socially responsible, globally deployable AI systems.