---
title: 'SinhalaMMLU: Native Multitask Benchmark'
url: https://www.emergentmind.com/topics/sinhalammlu
type: topic
---

# SinhalaMMLU: Native Multitask Benchmark

Searching arXiv for the benchmark paper and closely related benchmark references.
arxiv_search(query="2509.03162", max_results=5)
SinhalaMMLU is a multiple-choice question answering benchmark for evaluating multitask language understanding in Sinhala, a low-resource language, with explicit emphasis on native-language curation, curriculum alignment, and culturally grounded content [2509.03162]. It adopts MMLU’s multi-domain, multiple-choice design and prompt format, but it is built natively in Sinhala and aligned with the Sri Lankan national curriculum rather than derived from automatic translation. The benchmark contains 7,044 questions spanning secondary and collegiate education levels, covers six domains and 30 subjects, and is intended to expose model weaknesses that are often obscured in anglocentric or translation-based evaluation settings, particularly in Humanities and other culturally dense subject areas [2509.03162].

## 1. Origins and design rationale

SinhalaMMLU was introduced to address a specific deficiency in LLM evaluation: existing benchmarks are described as predominantly English-centric and often rely on machine-translated content, which can carry over Western cultural assumptions and introduce translation errors, especially in non-Latin-script, low-resource languages such as Sinhala [2509.03162]. The benchmark therefore uses native Sinhala questions curated from Sri Lankan exams, avoiding translation artifacts and culturally mismatched content.

The benchmark’s design philosophy is explicitly local rather than merely multilingual. It is aligned with the Ministry of Education Sri Lanka curriculum and sourced from real exam materials rather than translated benchmark items. Its coverage includes Sinhala-specific domains such as Sri Lankan history, drama and theatre, indigenous dancing, oriental music, and Sinhala language and literature. A human evaluation reported that the naturalness of native Sinhala STEM questions scored 97.30%, compared with 71.07% for machine-translated GlobalMMLU-si STEM questions, which the paper uses as empirical evidence that translation underperforms linguistically in this setting [2509.03162].

Within the broader benchmark landscape, SinhalaMMLU is positioned as complementary to language-specific MMLU variants such as ArabicMMLU and CMMLU, and to multilingual efforts such as GlobalMMLU [2412.03304] and INCLUDE [2411.19799]. Its distinguishing property is that it is native and curriculum-aligned, with an explicit cultural subset intended to reveal model vulnerabilities on culturally grounded knowledge rather than only on decontextualized cross-lingual transfer [2509.03162].

## 2. Corpus composition and curricular scope

The dataset contains 7,044 questions in a multiple-choice format, primarily with 4 or 5 options and a single correct answer; no multi-correct or “none of the above” cases are reported [2509.03162]. Educational coverage is restricted to Secondary (Grades 6–11) and Collegiate (A-Level, Grades 12–13), while Primary is excluded due to unsuitability for MCQs. Difficulty is mapped to grade bands: Easy corresponds to Grades 6–8, Medium to Grades 9–11, and Hard to Grades 12–13. Difficulty totals are Easy 1,893, Medium 2,585, and Hard 2,566 [2509.03162].

A small set of Grade 6–8 geography questions originally had 3 options, comprising 44 items. For format consistency, a fourth distractor was later added using Claude, but these synthetic distractors were not used in model evaluation [2509.03162].

The benchmark spans six domains and 30 subjects. Humanities is the largest domain, followed by Social Science and Other. Domain-level question counts and average character lengths are as follows [2509.03162]:

| Domain | Questions | Avg. character lengths (question; answer) |
|---|---:|---:|
| Humanities | 3,419 | 93.9; 22.2 |
| Social Science | 1,084 | 141.8; 22.3 |
| STEM | 629 | 157.8; 27.4 |
| Language | 397 | 74.2; 25.7 |
| Business Studies | 477 | 173.4; 33.0 |
| Other | 1,038 | 108.6; 28.2 |

Subject coverage is granular and curriculum-specific. Humanities includes History, Drama and Theatre, Dancing, Eastern Music, Arts, Buddhism, Catholicism, Christianity, Islam, Buddhist Civilization, Oriental Music, History of Sri Lanka, and Dancing Indigenous. Social Science includes Citizenship Education, Health and Physical Science, Geography, and Political Science. STEM includes Physics, Chemistry, Biology, and Science. Language is represented by Sinhala Language and Literature. Business Studies includes Business and Accounting Studies, Entrepreneurship Studies, and Economics. The Other domain includes Home Economics, Communication and Media Studies, Agriculture and Food Technology, Design and Construction Technology, and Biosystems Technology [2509.03162].

Examples identified in the paper figures include Sri Lankan history timelines, traditional arts and theatre, indigenous dance forms, Buddhist concepts, and general academic topics such as science, geography, health, citizenship, and business basics. This suggests that SinhalaMMLU is intended to stress both standard scholastic competence and knowledge embedded in local pedagogical and cultural practice.

## 3. Data creation, annotation, and quality control

The primary source for the benchmark is e-Thaksalawa, the Ministry of Education’s official e-learning platform, including past papers, marking schemes, and curated MCQs [2509.03162]. Additional materials came from government-released past papers and marking schemes from educational sites. Four annotators, all undergraduate or higher, performed manual extraction from PDFs after OCR and scraping, over a curation period of two months.

The curation criteria enforced a clear multiple-choice format with defined options and a single correct answer. Multimodal questions involving images, audio, or video were excluded, as was mathematics, because national papers in mathematics are not MCQ-based [2509.03162]. Questions were uniquely curated, with exact duplicates removed and near-duplicates filtered via cosine similarity; items with similarity greater than 95% were flagged and removed.

Metadata is retained at the item level. Each item includes subject, domain, difficulty level mapped to grade, original source and year, and province for provincial papers. Chemical and mathematical expressions are represented in a mix of LaTeX and plain text where unambiguous [2509.03162].

Translation avoidance is a central methodological commitment. The dataset is entirely native Sinhala and does not automatically translate English MMLU content. The human evaluation of linguistic naturalness used two Sinhala-speaking annotators on a 5-point scale linearly transformed to a 100-point scale, reporting 97.30% for native SinhalaMMLU STEM versus 71.07% for the translated GlobalMMLU-si STEM subset [2509.03162]. No Cohen’s $\kappa$ is reported for MCQ annotation, because the human evaluation focused on linguistic naturalness rather than label adjudication.

The benchmark is publicly accessible through GitHub and Hugging Face and is intended for research and education under Sri Lankan fair-use provisions, with release under an appropriate open license for research and educational use. No additional usage restrictions beyond fair use are noted [2509.03162].

## 4. Evaluation configuration and statistical framework

The evaluation setup is subject-structured. For each subject, the benchmark provides at least 104 test questions and a few-shot set of 3 questions for optional in-context prompting; the few-shot examples are not part of the test set [2509.03162]. Evaluation is closed-book MCQ, and no chain-of-thought is reported.

Two prompt templates are used: one with the subject name in the instruction and one without. Instructions are in Sinhala, matching the language of the question. Subject-inclusive prompts generally improve accuracy. For closed models, an additional system prompt is used to enforce the answer format, for example by requiring output of a single letter [2509.03162].

Answer extraction differs by model class. For open-source models, the selected option is the one with highest probability among the candidate choices. For closed-source models, the first generated token, typically a letter from A to E, is parsed via regex [2509.03162]. The recommended practice is to use the subject-included Sinhala prompt template, maintain a closed-book setup, enforce single-letter outputs for closed models, and avoid extraneous context or hints beyond the subject name.

The benchmark adopts standard MCQ evaluation metrics. Accuracy is defined as
$$
\mathrm{Acc} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[\hat{y}_i = y_i].
$$
Macro-averaged accuracy across subjects is defined as
$$
\mathrm{Acc}_{\text{macro} = \frac{1}{S} \sum_{s=1}^{S} \frac{1}{N_s} \sum_{i \in s} \mathbf{1}[\hat{y}_i = y_i]\).
$$
The random-guess baseline for $k$-option MCQs is
$$
p_{\text{rand} = \frac{1}{k}
$$
with $p_{\text{rand} = 0.25}$ for 4 options and $p_{\text{rand} = 0.20}$ for 5 options. Standard error is given by
$$
\mathrm{SE} = \sqrt{\frac{p(1-p)}{N}}
$$
and the 95% confidence interval by
$$
p \pm 1.96\,\mathrm{SE}.
$$
For paired model comparison, the benchmark recommends McNemar’s test with continuity correction:
$$
\chi^2 = \frac{(|b - c| - 1)^2}{b + c},
$$
where $b$ and $c$ are the counts for instances in which one model is correct and the other is wrong, and vice versa [2509.03162].

Reproducibility guidance is also explicit. The paper reports temperature in the 0–0.5 range, with GPT-4o evaluated at temperature=0.5 and top_p=0; deterministic decoding is preferred for closed models. Open models were run with 4-bit NF4 quantization, double quantization, and bfloat16 compute to fit memory constraints [2509.03162]. The test split is to be treated as held out, and benchmark items from public Sri Lankan past papers should not be used for fine-tuning.

## 5. Model suite and empirical findings

SinhalaMMLU evaluates 26 LLMs, including three closed-source systems—Claude-3-5-Sonnet, Claude-3-Haiku, and GPT-4o—and open-source families from LLaMA, Qwen2.5, Mistral, and Aya Expanse [2509.03162]. The open models include Llama-3.2-1B, 1B-chat; Llama-3.2-3B, 3B-chat; Llama-3-8B, 8B-chat; Llama-3-70B, 70B-chat; Llama-3.1-8B, 8B-chat; Llama-3.1-70B, 70B-chat; Llama-3.3-70B-chat; Qwen2.5 7B, 7B-chat, 32B, 32B-chat, 72B, 72B-chat; Mistral-7B and Mistral-7B-Instruct; and aya-expanse-8b and aya-expanse-32b.

In zero-shot evaluation with subject-included prompts, Claude 3.5 Sonnet achieved the highest average accuracy at 67.65%, followed by GPT-4o at 62.95% [2509.03162]. Among open models, Qwen2.5-72B-chat reached 41.18% and Qwen2.5-32B-chat 36.47%. Smaller open models such as LLaMA-3.2-1B and Mistral-7B clustered near approximately 22%, close to the 4-option random baseline.

Selected per-domain results show substantial heterogeneity. Claude 3.5 Sonnet attained 66.15% on Humanities, 62.37% on Language, 77.55% on Social Science, 61.40% on STEM, 73.12% on Business, and 65.58% on Other [2509.03162]. GPT-4o attained 62.02%, 51.29%, 71.32%, 60.59%, 67.53%, and 61.44% on those same domains. Average accuracy across models was 27.37 for Humanities, 26.10 for Language, 31.79 for Social Science, 27.16 for STEM, 27.78 for Business, and 28.07 for Other, indicating broad room for improvement.

The paper reports a consistent subject-level pattern: models perform better on fact-heavy, descriptive subjects such as Citizenship Education, Health and Physical Science, and some religions than on culturally dense areas such as Sinhala Language and Literature, History, and Drama and Theatre [2509.03162]. This suggests that high aggregate performance can coexist with major deficits in locally grounded reasoning and domain-specific language competence.

A hand-labeled cultural subset comprises 1,608 questions, about 22% of the dataset. Performance drops markedly on this subset: Claude Sonnet falls by 28.22 points, from 67.65 to 39.43; GPT-4o falls by 23.83 points, from 62.95 to 39.12; and Qwen2.5-72B-chat falls by 11.15 points, from 41.18 to 30.03 [2509.03162]. The paper’s subject-wise radar plots further indicate complementary strengths between Claude Sonnet, which is better in Buddhist concepts and language, and GPT-4o, which is better in music and history.

Difficulty also matters strongly. The Hard level, corresponding to A-Level items with typically 5 options, is notably challenging, and accuracy drops across all models relative to Easy and Medium [2509.03162]. Few-shot prompting with 3-shot examples yields only small or inconsistent gains; some instruction-tuned chat models degrade slightly, which the paper notes is consistent with prior findings on few-shot limits in instruction-tuned LLMs.

## 6. Failure modes, limitations, and research significance

The reported failure modes are concentrated in culturally specific and structurally difficult question types. Models struggle with cultural references and localized knowledge, including traditional arts, theatre, indigenous dance, and historical events and figures [2509.03162]. In Sinhala Language and Literature, they also struggle with idioms, morphology, and grammar-specific constructs, and performance sensitivity to subject naming in prompts is taken as evidence of weak priors.

Negation is a prominent challenge. Accuracy decreases on questions containing explicit negation markers. For Claude Sonnet, the reported accuracy is 58.36% on questions with negation versus 67.66% without; for GPT-4o, it is 59.59% versus 62.91% [2509.03162]. The paper describes typical errors on prompts such as “find the incorrect statement” or “which is not compatible,” where the model selects a distractor that is true but irrelevant rather than the intended negated answer.

Suboption questions, which comprise about 9.8% of the dataset, also reduce performance consistently across models. These require multi-step reasoning over matching, sequence correctness, or explanation relationships [2509.03162]. Illustrative failures include sequencing tasks whose correct answer depends on Sri Lankan historical chronology or causality in drama and theatre narratives. Hard 5-option questions further increase error rates because A-Level distractors are subtle. In a direct option-count analysis, reducing hard questions from 5 options to 4 improves accuracy from 60.11% to 66.13% for Claude Sonnet and from 56.71% to 62.17% for GPT-4o, without eliminating the underlying cultural and terminology challenges [2509.03162].

The paper also reports confidence–accuracy analysis for selected models. Correlation is weak or inconsistent for most models, but Qwen2.5-72B-chat shows moderate positive signals, with corr=0.7588 for 5-option questions and 0.6835 for 4-option questions [2509.03162]. A plausible implication is that calibration behavior on Sinhala exam-style MCQs is highly model-dependent and should not be inferred from aggregate accuracy alone.

SinhalaMMLU has several stated limitations. It is text-only, excluding images, audio, video, and tables; mathematics is excluded; Humanities and Social Science are overrepresented because of source availability; no human baseline scores are provided; and prompting strategies such as chain-of-thought and VLMs are not explored [2509.03162]. Planned or desired expansions include multimodal items, primary-level content, professional-level subjects, generative tasks, more balanced domain coverage, and refined difficulty calibration.

The benchmark’s broader significance lies in its demonstration that state-of-the-art proprietary models still struggle in cultural domains and that translation-based evaluation is insufficient for high-quality assessment in low-resource languages [2509.03162]. The paper argues for culturally aware adaptation strategies, robust negation and suboption reasoning, and proper terminology grounding. A plausible implication is that improvements for Sinhala will depend less on generic multilingual scaling than on targeted pretraining, fine-tuning, and data augmentation with native curricular materials, especially in Humanities and Language.

Source: https://www.emergentmind.com/topics/sinhalammlu