SEA-NLI: Natural Language Inference as a Lens into Southeast Asian Cultural Understanding

Abstract: Frontier LLMs perform well in Western contexts, but remain poorly tested on underrepresented cultures such as those in Southeast Asia (SEA). Existing NLI benchmarks are largely Western-centric, translation-derived, or monolingual, limiting their ability to measure culturally grounded reasoning. We introduce SEA-NLI, a native, culturally grounded NLI benchmark covering eight SEA countries in English and native regional languages, verified by native speakers. Across 17 encoder and decoder models, we observe a low performance from all models, especially for knowledge-intensive categories such as Languages and Science and Technology. Our analysis shows that failure cases mainly stem from missing SEA cultural knowledge: SEA-adapted models and culture-aware prompting improve performance, while CoT prompting offers limited gains.

• The paper introduces a novel NLI dataset tailored for Southeast Asia, addressing Western bias by incorporating culturally specific examples.
• It leverages advanced prompt engineering, iterative filtering, and human validation to ensure cultural relevance and robust quality control.
• Empirical evaluation shows that existing models struggle with culturally nuanced inference, underscoring the need for SEA-specific model adaptations.

SEA-NLI: A Resource for Diagnosing Southeast Asian Cultural Understanding through Natural Language Inference

Motivation and Background

The SEA-NLI dataset is developed to address a persistent gap in linguistic and cultural representation within NLI benchmarks. Existing datasets such as SNLI, MNLI, and XNLI are largely Western-centric or translation-derived, which means they inherit cultural assumptions and linguistic artifacts from English source texts. This propagates Western cultural bias and overlooks idiosyncratic, locally grounded knowledge found in Southeast Asia (SEA). SEA comprises over 700 million people and is notably underrepresented in evaluation resources. The creators of SEA-NLI explicitly target culturally situated reasoning by curating NLI examples rooted in SEA contexts and languages, rather than relying on translation or generic monolingual data. Also, SEA-NLI considers both language understanding and cultural knowledge gaps, enabling diagnostic evaluation at a finer granularity.

Figure 1: Comparison of SEA-NLI with the existing NLI datasets, highlighting native multilingual cultural coverage.

Dataset Construction and Quality Control

SEA-NLI encompasses eight SEA countries and languages, including Cambodian (KHM), Myanmar (MYA), Malay (ZSM), Thai (THA), Singaporean Tamil (TAM), Filipino (FIL), Indonesian (IND), and Vietnamese (VIE), with parallel English annotation. The construction process integrates LLM (GPT-5.2) generation, stringent prompt engineering, iterative filtering, and human validation.

The dataset creation pipeline includes:

• Concept gathering from Wikipedia, filtered by annotators to ensure regional relevance and specificity.
• Multi-stage generation of premises and hypotheses, leveraging role prompts for cultural accuracy, and strong constraints to mitigate annotation artifacts (e.g., lexical overlap, length parity, anti-negation, prohibiting generic noun phrase substitution).
• Lexical and semantic filtering using Bag-of-Words logistic regression and DeBERTa-v3, rejecting shortcut-solvable samples via both features and deep NLI model predictions.
• Iterative regeneration of "hard" examples that survive both filter layers, ensuring models cannot rely on superficial cues.

Human annotators from each region verified label correctness, cultural relevance, and linguistic naturalness, with inter-annotator agreement scores reaching 0.96 (ICC), and a cultural relevance weighted average of 4.49/5, confirming robust regional grounding.

Figure 2: The complete process for creating SEA-NLI, from concept gathering to iterative filtering and human validation.

Figure 3: Data statistics of SEA-NLI, showing the distribution across categories and nations.

Model Evaluation and Analysis

SEA-NLI is used to evaluate 17 models across encoder (XLM-R, mmBERT, mDeBERTa) and decoder (Qwen, Llama, Gemma, SEA-LION) architectures. Models are assessed in both zero-shot and fine-tuned settings, with metrics computed as weighted F1 over SEA-language and English translations.

Difficulty of Culturally Grounded NLI

Performance drops significantly from the normal to the hard set—average F1 scores decrease by 11.6% (SEA) and 13.17% (English). This degradation is primarily attributed to the removal of shortcut-solvable cases and the increased demand for cultural knowledge, with the hard set requiring inference beyond explicit lexical cues.

Figure 4: The overall performance of 17 models on SEA-NLI, showing marked drops on hard sets and SEA languages.

Category-Level Insights

Models demonstrate category-dependent weaknesses. Cultural concepts tied to visually salient or commonly encountered features (Musical Instrument, Cuisine, Landmark) yield higher scores ($\sim$72–78%), whereas Languages and Science & Technology—requiring fine-grained or deeply contextualized knowledge—are consistently challenging ($\sim$58–62% F1).

Figure 5: Weighted F1 performance across cultural concept categories on SEA languages (Hard set), showing diminished results for knowledge-intensive domains.

Failure Mode Taxonomy

A rigorous error taxonomy compares parallel English and SEA-language predictions:

• Cultural knowledge deficit (failure in both): Dominates, accounting for up to 25.56% of errors in some models.
• SEA language misunderstanding (ENG correct / SEA wrong): Up to 30.56% error in small models.
• Cross-lingual misalignment (ENG wrong / SEA correct): Rare ($<$8%).

  Figure 6: Error taxonomy mapping English and SEA prediction outcomes to distinct model deficiencies.

Prompt Engineering and Cultural Adaptation

Targeted prompt engineering (persona adoption, concept summaries, topic metadata) consistently improves SEA-adapted models, yielding up to 2.69 (normal set) and 4.86 (hard set) F1 gain compared to base variants. In contrast, chain-of-thought prompting and in-context learning tend to reinforce lexical matching, often leading to over-prediction of "neutral" and failing to activate deep SEA cultural representations.

Figure 7: Evolution of prompt templates for culturally-aware reasoning.

Notably, culture-aware prompting provides more substantial improvements than reasoning elicitations such as CoT, underscoring that SEA-NLI's challenge is primarily in the retrieval and application of cultural knowledge, not abstract logical reasoning.

Dataset Robustness and Bias Mitigation

SEA-NLI employs aggressive annotation artifact mitigation—length parity constraints, lexical overlap filtering, and hypothesis-only prediction experiments show F1 scores well below the random baseline, confirming models cannot exploit statistical shortcuts. Iterative regeneration boosts average hypothesis and premise length, aligning naturalness and relevance. Word overlap and hypothesis length heuristics are demonstrably less predictive post-filtering/regeneration, particularly in the hard subset.

Figure 8: Evolution of the SEA-NLI word frequency distribution, demonstrating mitigation of label-specific lexical spikes.

Practical and Theoretical Implications

The results indicate existing frontier LLMs, including large-scale SEA-adapted models, remain insufficient for robust NLI in SEA contexts—performance degradation is not explained by language understanding alone, but by substantive cultural knowledge gaps. Diagnostic error analysis exposes latent deficiencies in both language modeling and cross-lingual alignment, suggesting that domain adaptation must encompass local cultural competencies, not merely linguistic fluency.

SEA-specific adaptation (exposure to regional corpora, cultural taxonomy-informed training, targeted prompt engineering) demonstrates tangible improvements, establishing a basis for further research into culturally aware LLMs. SEA-NLI enables more nuanced benchmarking and interpretability of culture-dependent failures.

Figure 9: Inference speed vs. F1 score on SEA-NLI, illustrating the trade-off between model scale, speed, and cultural robustness.

Conclusion

SEA-NLI offers a comprehensive, rigorously curated benchmark for culturally grounded NLI in Southeast Asia, with dual-language annotation and stringent artifact mitigation. Evaluation across 17 models reveals that superficial improvements in reasoning and English performance do not transfer to knowledge-intensive SEA scenarios—only SEA-adapted models with explicit cultural awareness manage moderate gains. The dataset facilitates precise error analysis, underscores the importance of regional knowledge in NLI, and provides actionable guidance for further adaptation and prompt design. SEA-NLI will serve as a foundational resource for advancing equitable, robust, and interpretable LLM development in SEA and other culturally complex regions (2606.03284).