Mind the Gap: Pitfalls of LLM Alignment with Asian Public Opinion

Abstract: LLMs are increasingly being deployed in multilingual, multicultural settings, yet their reliance on predominantly English-centric training data risks misalignment with the diverse cultural values of different societies. In this paper, we present a comprehensive, multilingual audit of the cultural alignment of contemporary LLMs including GPT-4o-Mini, Gemini-2.5-Flash, Llama 3.2, Mistral and Gemma 3 across India, East Asia and Southeast Asia. Our study specifically focuses on the sensitive domain of religion as the prism for broader alignment. To facilitate this, we conduct a multi-faceted analysis of every LLM's internal representations, using log-probs/logits, to compare the model's opinion distributions against ground-truth public attitudes. We find that while the popular models generally align with public opinion on broad social issues, they consistently fail to accurately represent religious viewpoints, especially those of minority groups, often amplifying negative stereotypes. Lightweight interventions, such as demographic priming and native language prompting, partially mitigate but do not eliminate these cultural gaps. We further show that downstream evaluations on bias benchmarks (such as CrowS-Pairs, IndiBias, ThaiCLI, KoBBQ) reveal persistent harms and under-representation in sensitive contexts. Our findings underscore the urgent need for systematic, regionally grounded audits to ensure equitable global deployment of LLMs.

• The paper audits five LLMs against weighted Pew survey data from 12 Asian countries and territories, finding frontier-model representativeness near 95% on general topics but roughly 90% on religion-related questions.
• The paper shows that minority religious identities receive disproportionate negative stereotyping, including IndiBias gaps for Shia (+28.9), Sunni (+23.3), Jain (+16.8), and Parsi (+16.5) groups.
• The paper finds that demographic priming, local-language prompts, and disambiguation improve alignment—Sinhala reduced Gemma-3’s JSD by about 31%, while KoBBQ accuracy rose from 0.611 to 0.961—but these interventions do not replace training and evaluation changes.

This paper presents a multilingual audit of how contemporary LLMs represent public opinion across twelve Asian countries and territories, using religion as the primary analytical lens. The authors evaluate GPT-4o-Mini, Gemini-2.5-Flash, and open-weight models (Gemma-3-12B, Llama-3.2-1B, Mistral-7B) against nationally representative Pew Research Center survey data from India, East Asia, and South/Southeast Asia. Their central finding is a consistent gap: models achieve high representativeness on general social topics but systematically misrepresent religious viewpoints—particularly those of minority groups—often amplifying negative stereotypes (2603.06264).

Ground truth and methodology

The study builds on three Pew-Templeton Global Religious Futures surveys covering India (2021), East Asia (2024), and South/Southeast Asia (2024), with multi-stage stratified random sampling and Pew's statistical weights applied to correct for design and non-response bias. Because the publicly available survey metadata is English-only despite the original in-language administration, the authors commissioned crowd-sourced manual translations into local languages (Hindi, Japanese, Korean, Mandarin, Vietnamese, Sinhala, Tamil, Thai, Khmer, Malay, Indonesian), paying translators roughly USD 125 in total.

The evaluation adapts the "whose opinions" methodology of Santurkar et al.: each multiple-choice survey question is posed to the model, and the token-level probabilities over answer options form a Model Opinion Distribution ($\mathcal{D}_\mathcal{M}$), compared against the weighted Human Opinion Distribution ($\mathcal{D}_\mathcal{O}$). Three bounded metrics are used: Jensen-Shannon Divergence (JSD) and Hellinger Distance (HD) as distributional dissimilarity measures treating options as unordered categories, plus the Wasserstein-based representativeness score $\mathcal{R}_\mathcal{M}$ that exploits the ordinal structure of answer scales. All decoding is deterministic (temperature zero).

Distributional alignment results

On non-religious items, both frontier models perform well: Gemini-2.5-Flash reaches 94.6% representativeness and GPT-4o-Mini 95.2%, dipping to approximately 89.9% and 90.2% respectively on religion-related prompts. A keyword taxonomy shows representativeness is highest on governance/politics items (95.2% / 94.6%), followed by other non-religion items and demographics, with religion lowest. Notably, the divergence shifts on the religion subset are small in absolute terms—GPT-4o-Mini's JSD is essentially flat ($A_{JSD} = -0.004$)—so the headline gap is modest at the aggregate level; the more consequential failures emerge in the subgroup and benchmark analyses below.

Demographic priming ("You are a citizen of ...") shifts outputs toward target distributions and reduces measured divergence on religion-related queries, indicating that latent cultural knowledge exists but is not activated by default prompting.

Language effects

Switching from English to local-language prompting consistently reduces JSD across all tested open-weight models, though HD remains largely unchanged. The largest effect is Gemma-3 in Sri Lanka, where Sinhala prompts yield a ~31% reduction in $A_{JSD}$ (0.47 → 0.32). A summary of representative results:

Model	Region (language)	$\mathcal{R}_\mathcal{M}$	$A_{JSD}$ (Eng → Local)	$A_{HD}$ (Eng → Local)
Gemma-3 12B-IT	Sri Lanka (Sinhala)	0.96	0.47 → 0.32	0.49 → 0.47
Llama-3.2 1B-Instruct	Taiwan (Chinese)	0.95	0.88 → 0.81	0.86 → 0.86
Mistral-7B Instruct-v0.3	Korea (Korean)	0.91	0.53 → 0.48	0.48 → 0.48

Two implications follow. First, native-language cueing helps concentrate probability mass on locally correct responses, so lightweight prompt-level interventions have partial value. Second—and more critically—the paper emphasizes that multilinguality does not guarantee cultural representativeness: Llama-3.2's $A_{JSD}$ in Taiwan exceeds 0.8 regardless of prompt language, indicating an outright failure to represent that population's opinions even when responding fluently in Chinese.

Downstream bias benchmarks

To connect distributional gaps to concrete harms, the authors evaluate four culturally grounded benchmarks.

CrowS-Pairs (religion subset): GPT-4o-Mini is robust, selecting anti-stereotype options in ~92% of cases (bias rate ~8%) consistently across six languages with zero invalid responses. Gemini-2.5-Flash is markedly weaker: ~16% bias rate, ~68% anti-stereotype accuracy, and 15–19 malformed responses per 105 items, worst in Vietnamese. This cross-model disparity raises concerns about Gemini's cross-lingual consistency and safety filtering behavior.

IndiBias: Using $\Delta$ELO between negative and positive plausibility splits for Indian religious identities, GPT-4o-Mini shows strong misrepresentation of minority groups—Shia (+28.9), Sunni (+23.3), Jain (+16.8), Parsi (+16.5)—while Hindu (−13.0), Sufi (−10.1), and Sikh (−9.3) receive comparatively favorable treatment. This asymmetry indicates that negative framings are disproportionately normalized for specific minority identities rather than uniformly degraded, consistent with stereotype prevalence in online discourse. Gemini-2.5-Flash shows convergent trends for Sunni.

ThaiCLI: Both models align well with Thai cultural norms under LLM-as-a-Judge scoring (GPT-4o as judge): GPT-4o-Mini scores above 8.3/10 across factoid and instruction formats; Gemini-2.5-Flash scores 7.52/10. This suggests cultural sensitivity in Thai contexts is comparatively strong relative to the religion-opinion gaps elsewhere.

KoBBQ: Disambiguation produces large calibration gains for GPT-4o-Mini—overall accuracy rises from 0.611 to 0.961, religion-related accuracy from 0.625 to 0.950, and differential religion bias falls from 0.275 to 0.100. Prompt specificity therefore substantially mitigates group-level calibration failures, implying that some measured bias reflects ambiguity resolution rather than entrenched preference.

Drivers of misalignment

The paper attributes persistent misalignment to structural factors: underrepresentation and stereotyping of minorities in internet-sourced training corpora; post-training pipelines (instruction tuning, RLHF) whose feedback reflects majority preferences and can create demographic hierarchies in refusal behavior; and architectural/tokenization choices favoring high-resource languages. The authors argue these mechanisms explain why representational gaps persist or amplify under local-language prompting—a direct challenge to the assumption that multilingual capability suffices for equitable performance. They also note that deeper interventions (activation steering, representation engineering, DPO-style fine-tuning) are largely inaccessible for black-box API models, making prompt-level steering the only practical lever for most deployments, which bounds what mitigation can achieve without changes to training data or access paradigms.

Limitations and open questions

The paper concedes several constraints. Religion serves as the sole lens on cultural values; political ideology, regional identity, and social hierarchy remain unexamined. The keyword-based taxonomy for classifying questions as "religious" is coarse, and the small absolute divergence shifts on religion subsets suggest the aggregate metric may understate subgroup-level harm—an issue the IndiBias results partially address but do not fully resolve. The ThaiCLI evaluation depends on GPT-4o as judge, inheriting known biases of LLM judges toward style over substance. Whether white-box interventions such as activation steering or fine-tuning on culturally specific corpora can close the residual gaps—and whether current binary-trait steering techniques can handle multi-dimensional cultural diversity—are left as open empirical questions.

Conclusion

This audit demonstrates that leading LLMs represent broad Asian public opinion reasonably well but fail specifically on religion, with minority faiths bearing disproportionate negative stereotyping across languages and benchmarks. Local-language prompting and demographic priming offer partial, measurable improvements but leave Hellinger-level probability shifts uncorrected, and one tested model fails outright regardless of language. The results support the paper's central claim that systematic, regionally grounded audits using native survey data are necessary before equitable global deployment can be claimed.