MyCulture: Malaysian Cultural Benchmark

• MyCulture is a Malaysian cultural benchmark featuring 2,652 expert-validated questions in Bahasa Malaysia across six domains such as arts, customs, and food.
• It pioneers an open-ended multiple-choice format that challenges models to generate exact answers, reducing guessing and format bias.
• Empirical findings reveal substantial performance drops in open-ended tasks, highlighting significant cultural and language biases in current LLMs.

Searching arXiv for the primary paper and a closely related survey to ground the article. MyCulture is a Malaysia-focused cultural benchmark for evaluating how well LLMs understand a culturally diverse, multilingual, and comparatively low-resource national context. It is introduced as the first expert-validated Malaysian cultural benchmark in Bahasa Malaysia, with 2,652 questions spanning arts, attire, customs, entertainment, food, and religion across East and West Malaysia, and it is designed to test cultural comprehension under low-resource language constraints rather than generic multilingual fluency alone (Hew et al., 7 Aug 2025). In the broader landscape of culturally aware NLP, MyCulture sits within a shift from treating culture as a vague proxy for language or nationality toward evaluating culture as grounded knowledge, norms, and context-sensitive reasoning (Liu et al., 2024).

1. Scope and cultural rationale

MyCulture is motivated by two linked claims. First, LLMs often exhibit cultural biases because the web-scale corpora used to train them are dominated by high-resource languages, especially English and Chinese. Second, Bahasa Malaysia is relatively low-resource in LLM training, so linguistic performance and cultural grounding are weaker in Malay than in high-resource languages (Hew et al., 7 Aug 2025).

Malaysia is used as a hard case for cultural alignment rather than as a narrow national exemplar. The benchmark is explicitly framed around a constitutionally and socially salient mix of ethnicities, languages, regions, and religions, including Malays, Chinese, Indians, indigenous communities, Islam, Buddhism, Hinduism, and strong East/West Malaysia regional differences. This suggests that MyCulture is intended not merely as a national knowledge probe but as an evaluation of whether a model can represent a multi-ethnic, multi-religious, and regionally differentiated society without collapsing it into a single dominant cultural template (Hew et al., 7 Aug 2025).

The benchmark also positions itself relative to prior culture-oriented resources such as Kulture, IndoCulture, and SaudiCulture. Those benchmarks are treated as useful, but MyCulture argues that they are often culturally narrower at the national level and do not capture the complexity of a multi-ethnic, multi-religious state like Malaysia. In this respect, MyCulture aligns with a broader research trend that treats culture in NLP as a composite of concepts, knowledge, values, norms, and sociocultural context rather than as a simple language label (Liu et al., 2024).

2. Dataset design and coverage

The benchmark covers six cultural pillars: arts, attire, customs, entertainment, food, and religion. It is intended to span both East and West Malaysia and include diverse races, religions, ethnicities, and local contexts. In total, the dataset contains 2,652 questions, split evenly into 1,326 closed-form MCQs and 1,326 open-ended MCQs. Both versions are expert-validated by local experts, and the benchmark is written in Malay/Bahasa Malaysia, which the authors stress is the national language and a low-resource language (Hew et al., 7 Aug 2025).

The questions are described as carefully crafted, locally contextualized, and validated by human experts, and the paper states that quality filtering affected the regional distribution. Figure 1 is described as showing the distribution across East and West Malaysia and the six pillars, intended to highlight regional representation differences. Although the paper does not provide a long procedural appendix, the emphasis on human and expert validation is central because the benchmark is explicitly designed for a low-resource-language setting in which automated or weakly localized question generation would be unreliable (Hew et al., 7 Aug 2025).

MyCulture also reflects a methodological preference for open-ended cultural evaluation over purely recognition-style evaluation. That preference parallels later work on Indonesian multi-hop cultural QA, which argues that cultural understanding requires reasoning across context and implicit social knowledge rather than only recalling isolated facts (Permadi et al., 3 Feb 2026). A plausible implication is that MyCulture’s value lies not only in Malaysian topical coverage but also in its attempt to reduce benchmark shortcuts.

3. Open-ended multiple-choice as an evaluation format

MyCulture’s central methodological innovation is the “open-ended multiple-choice” format without predefined options. The task preserves MCQ-style structure and objective answerability, but the model must generate the answer itself rather than choose from listed candidates. The paper instantiates three open-ended task types: multi-answer, ordering, and matching, ranked from easiest to hardest in that order (Hew et al., 7 Aug 2025).

The paper’s examples clarify the design. In a food question, the closed-form version offers alternatives such as “A. I, III” or “B. I, II, III,” whereas the open-ended version presents the statements directly and requires the model to output the exact subset, such as “A, C.” In matching, the closed-form version provides complete candidate correspondences, while the open version requires the exact mapping, such as “A-i, B-ii, C-iii, D-iv.” In ordering, the model must generate the exact sequence itself, such as “A, B, C, D” (Hew et al., 7 Aug 2025).

The benchmark’s theoretical argument is that removing predefined options reduces guessing, format-induced bias, and template exploitation. The formal setup defines a finite answer alphabet $\mathcal{A}$, a gold answer set $S^\star \subset \mathcal{A}$, model output $\hat{S} = g(f_\theta(x))$, and exact-set accuracy: $$\text{Acc}(f_\theta; q) = \begin{cases} 1, & \text{if } \hat{S} = S^\star, \ 0, & \text{otherwise}. \end{cases}$$ The scoring rule is intentionally strict: partial overlap does not count (Hew et al., 7 Aug 2025).

For ordinary MCQs, the intended random-guess baseline is $1/m$, and for four options this is $1/4$. For open-ended multi-answer generation, the paper claims the exact-match probability under uniform random guessing without replacement is: $$\Pr\big(\hat{S} = S^\star\big) = \frac{1}{\binom{n}{k}},$$ where $n = |\mathcal{A}|$. It also gives an information-theoretic comparison: $$I_{\text{open}} = \log_2 \binom{n}{k}, \qquad I_{\text{MCQ}} = \log_2 m.$$ For $n=8$ and $S^\star \subset \mathcal{A}$0, this yields $S^\star \subset \mathcal{A}$1, $S^\star \subset \mathcal{A}$2 bits, and $S^\star \subset \mathcal{A}$3 bits, leading the paper to conclude that each open-ended MCQ provides over 3 times more evaluative signal than a standard four-option MCQ (Hew et al., 7 Aug 2025).

The paper also defines separate random-guessing formulas for the three open-ended variants: $S^\star \subset \mathcal{A}$4

$S^\star \subset \mathcal{A}$5

$S^\star \subset \mathcal{A}$6

It summarizes the intended ordering as: $S^\star \subset \mathcal{A}$7 The paper itself notes no formal caveat here, but the accompanying description indicates that some formulas are imperfectly typeset and, in places, mathematically inconsistent. This suggests that the benchmark contribution is more compelling than the exact derivations, while the central intuition remains clear: predefined options drastically reduce search complexity and can inflate apparent competence (Hew et al., 7 Aug 2025).

4. Experimental protocol and scoring

The benchmark is evaluated in zero-shot settings across 14 models: GPT-4o, GPT-4o mini, DeepSeek-V3, Qwen3 (3.3B/30B MoE), Qwen3 8B, Qwen3 1.7B, Llama 4 Scout, Llama-3.2 3B, SeaLLMs-v3 7B, SeaLLMs-v3 1.5B, SEA-LION-V3 70B, SEA-LION-V3 8B, MaLLaM Small 2.5, and MaLLaM Tiny 2.5. The model set mixes international systems with Southeast Asian regional models, allowing the benchmark to test whether regional specialization improves Malaysian cultural understanding (Hew et al., 7 Aug 2025).

All core benchmark questions are in Malay. The same cultural content is evaluated in both closed-form and open-ended versions. For few-shot ablations, the authors sample examples from the dataset and include them in the system prompt, while removing prompt-included items from the test set for fairness. For structured-output experiments, they request predefined answer schemas; a JSON schema is specifically mentioned in relation to DeepSeek-V3’s failure. For language-bias experiments, they vary the instruction or system prompt language among Malay, English, and Chinese while keeping the question content fixed (Hew et al., 7 Aug 2025).

Scoring uses deterministic post-processing and exact answer matching. The formal setup includes a deterministic extraction function $S^\star \subset \mathcal{A}$8 that converts the model output string into an answer set $S^\star \subset \mathcal{A}$9, and only exact correspondence with the gold answer receives credit. The paper does not describe any fuzzy semantic matcher. This strictness is integral to the benchmark’s claim that it measures “full coverage” of the cultural answer space rather than partial plausibility or option elimination (Hew et al., 7 Aug 2025).

5. Empirical findings

The headline empirical result is the large drop from closed-form to open-ended evaluation. All models degrade substantially, supporting the paper’s claim that standard MCQs systematically overestimate LLM cultural competence (Hew et al., 7 Aug 2025).

Model	Closed-form	Open-ended
GPT-4o	67.87	38.39
DeepSeek-V3	62.89	37.17
SEA-LION-V3 70B	64.17	37.85
MaLLaM Small 2.5	61.53	19.45
Llama-3.2 3B	29.41	2.71

The full reported drops are substantial: GPT-4o falls by $\hat{S} = g(f_\theta(x))$0, GPT-4o mini by $\hat{S} = g(f_\theta(x))$1, DeepSeek-V3 by $\hat{S} = g(f_\theta(x))$2, Qwen3 MoE by $\hat{S} = g(f_\theta(x))$3, Llama 4 Scout by $\hat{S} = g(f_\theta(x))$4, SeaLLMs-v3 1.5B by $\hat{S} = g(f_\theta(x))$5, MaLLaM Small 2.5 by $\hat{S} = g(f_\theta(x))$6, and MaLLaM Tiny 2.5 by $\hat{S} = g(f_\theta(x))$7 (Hew et al., 7 Aug 2025).

Several comparative patterns are explicit. GPT-4o is best overall in both formats. Among open-source models, SEA-LION-V3 70B is strongest and is also the best-performing Southeast Asian regional model. MaLLaM is the clearest example of closed-form inflation: it reaches about 61% closed-form accuracy but falls to about 19% in open-ended evaluation. The paper uses this pattern to argue that some regionally tuned or Malaysia-oriented models appear strong when the benchmark permits option selection, yet do not sustain that performance when exact cultural answer generation is required (Hew et al., 7 Aug 2025).

The few-shot ablation finds little benefit from adding examples for GPT-4o and DeepSeek-V3, with no clear improvement trend. This is interpreted as consistent with prior work suggesting that few-shot prompting offers limited gains for cultural understanding tasks (Hew et al., 7 Aug 2025).

6. Structural bias, language bias, and broader significance

MyCulture includes two additional analyses beyond format choice: structural bias and language bias. The structural-bias experiment compares unstructured free-form generation with structured output. Structured output often helps by constraining the model to produce the expected answer form rather than verbose explanations, hallucinated content, or omitted answers. For example, GPT-4o improves from 38.39 to 41.03, GPT-4o mini from 33.86 to 37.40, SEA-LION-V3 70B from 37.85 to 43.06, and MaLLaM Small 2.5 from 19.45 to 38.31. DeepSeek-V3 is the outlier, dropping from 37.17 to 2.87 because it failed to follow the predefined JSON schema (Hew et al., 7 Aug 2025).

The language-bias experiment varies prompt language across Malay, English, and Chinese while keeping the same Malay cultural question content. Malay prompts do not consistently yield the best results. DeepSeek-V3 improves from 37.17 in Malay to 40.79 in English; SEA-LION-V3 70B is slightly better in Chinese at 39.51 than in Malay at 37.85; MaLLaM Small 2.5 improves from 19.45 in Malay to 23.53 in English and 25.56 in Chinese (Hew et al., 7 Aug 2025).

This result is especially notable because it cuts against the simple expectation that local-language prompting always best supports local-culture tasks. The paper interprets it as evidence of language bias rooted in pretraining imbalance: models may encode Malaysian culture more strongly through high-resource-language representations than through Bahasa Malaysia. That diagnosis resonates with later work showing that LLMs can often infer cultural background yet fail to apply it appropriately in final responses unless reasoning is explicitly structured (Miao et al., 16 Jun 2026).

The paper’s broader normative position is that culturally detached model behavior can worsen social exclusion and racial insensitivity, particularly in plural societies. MyCulture therefore functions simultaneously as a dataset, an evaluation proposal, and an argument about benchmark design. As a dataset, it provides 2,652 expert-validated Malaysian cultural questions in Bahasa Malaysia. As an evaluation proposal, it replaces answer-option selection with open-ended answer generation in multi-answer, ordering, and matching formats. As an empirical result, it shows that current LLMs, including strong global systems and regionally tuned models, still have major gaps in Malaysian cultural understanding, and that conventional closed-form benchmarks can substantially overstate competence (Hew et al., 7 Aug 2025).