---
title: 'TR-MMLU: Turkish Multitask Language Benchmark'
url: https://www.emergentmind.com/topics/turkish-massive-multitask-language-understanding-tr-mmlu
type: topic
---

# TR-MMLU: Turkish Multitask Language Benchmark

The Turkish Massive Multitask Language Understanding (TR-MMLU) benchmark is a comprehensive, natively crafted evaluation suite designed to measure the linguistic and conceptual capabilities of large language models (LLMs) for Turkish. TR-MMLU addresses the specific challenges of morphologically rich and culturally distinct languages, providing a standard for assessing LLM performance with respect to both factual knowledge and deeper linguistic phenomena. The benchmark incorporates high-quality, curriculum-aligned content spanning a wide array of academic and professional domains rooted in the Turkish education system, and it emphasizes the importance of tokenization strategies tailored for Turkish morphological characteristics [2501.00593][2502.07057][2502.11020][2407.12402][2508.13044].

## 1. Benchmark Construction and Dataset Characteristics

TR-MMLU distinguishes itself by its scale, native authorship, and granularity of topic coverage. The principal dataset comprises 6,200 multiple-choice questions allocated across 62 sections, themselves derived from a candidate pool of 280,000 items originating from standardized Turkish exams such as the University Entrance Exam, AUZEF, KPSS, TUS, and the national Driver’s License exam. Disciplines span law, medicine, social sciences, history, natural sciences, and professional licensing, covering over 800 unique topics. Approximately 100 questions are distributed per section, and questions are evenly split among three difficulty levels—Easy, Medium, Hard—empirically determined by historical student performance [2501.00593][2508.13044]. 

Key dataset construction features include:
- All questions are originally authored by Turkish subject-matter experts, not translated, ensuring direct alignment with Turkish morphosyntax, agglutination, and cultural content.
- Curriculum specialists and linguists perform rigorous filtering to eliminate duplicates, outdated material, and items with potential overlap with LLM pretraining data, thereby preventing data leakage.
- Each question undergoes validation for linguistic correctness (orthography, diacritics, grammar) and contextual appropriateness (legal and historical terms, idiomatic expressions specific to Turkish society) [2501.00593][2502.11020].

## 2. Evaluation Protocol and Metrics

TR-MMLU employs a suite of standardized evaluation protocols:
- **Zero-shot evaluation** is predominant: models are presented with questions and options without in-context demonstrations or fine-tuning on the evaluation set.
- **Prompting** is adapted through five Turkish-centric prompt templates. The dominant template requests only the correct answer option:  
  `"Sana soru ve seçenekleri veriyorum. Sadece hangi seçeneğin sorunun doğru cevabı olduğunu yaz."`  
- **Scoring**: The primary metric is accuracy, defined per model or per section as
  $$
  \text{Accuracy} = \frac{\text{Number of correct answers}}{\text{Total number of questions}} \times 100\%
  $$
  Both overall and macro-averaged accuracies across sections are reported, alongside response normalization using semantic similarity checks to handle answer formatting variation via the "paraphrase-multilingual-mpnet-base-v2" model [2501.00593][2508.13044].  
- **Section-level analysis** enables fine-grained diagnostics, for example on TUS (medical specialization), KPSS (public personnel exams), and domain-specific categories.

Controlled randomness (seed=42) and consistent hardware are maintained for all model evaluations, with detailed logs of processing time and accuracy [2508.13044].

## 3. Model Benchmarks and Performance Analysis

A diverse array of 39 LLMs—both open-source and proprietary—have been systematically benchmarked. Key leaderboard results are summarized:

| Model         | Family | Param Size | Correct | Accuracy (%) | Time (s) |
|---------------|--------|------------|---------|--------------|----------|
| GPT-4o        | GPT    | -          | 5260    | 84.84        | 5021     |
| Claude-3.5    | Sonnet | -          | 5233    | 84.40        | 7379     |
| Llama3.3      | Llama  | 70.6B      | 4924    | 79.42        | 13355    |
| Gemini-1.5    | Gemini | -          | 4758    | 76.74        | 4985     |
| Gemma2-27B    | Gemma2 | 27.2B      | 4470    | 72.10        | 5506     |

Domain-level analysis shows that factual and legal questions (e.g., Driver’s License) yield the highest accuracy (≈97%), while reasoning-intensive sections (e.g., KPSS) show lower scores (as low as 66%). Medical specialization domains approach or surpass 90% for models with relevant fine-tuning. Catastrophic forgetting is observed in models fine-tuned on partial Turkish domains, highlighting the importance of continual learning strategies [2501.00593][2508.13044].

## 4. Tokenization Methodologies and Linguistic Challenges

Tokenization is central to Turkish language modeling performance. TR-MMLU explicitly evaluates tokenizer quality using metrics that reflect both computational and linguistic qualities:
- **Vocabulary Size (|V|)**: Larger vocabularies capture rare Turkish forms but can inflate model capacity unnecessarily.
- **Token Count (N)**: Efficient tokenization minimizes sequence length, reducing context window waste.
- **Turkish Token Percentage (%TR)**: Fraction of unique tokens aligning to valid Turkish words, as validated by ITU Web Service and Kalbur.
- **Pure Token Percentage (%Pure)**: Fraction of unique tokens corresponding to atomic Turkish morphemes.
- **Processing Time (T)**: Speed of full corpus tokenization [2502.07057].

Empirical results:
- %TR strongly correlates with downstream MMLU accuracy (r = +0.90, p < 0.05), outperforming %Pure (r = +0.68, p ≈ 0.10).
- Each point increase in %TR yields ~+1.26% accuracy, confirming that high linguistic alignment, not mere model scale, determines reasoning performance.
- Gemma-2 achieves a %TR of 48.63% and top open-source accuracy; morphological-informed tokenizers (as in Aya-Expanse, %TR=50.67%) are competitive [2502.07057].

Key best practices:
- Aim for Turkish token percentages approaching 50%, requiring vocabularies of 200,000–300,000 subwords.
- Incorporate morphological analyzers (e.g., Kalbur) in vocabulary construction.
- Validate all output tokens against a native Turkish lexicon to maximize semantic fidelity [2502.07057].

## 5. Comparative Evaluation and Related Benchmarks

TR-MMLU links closely with broader Turkic and Turkish evaluation initiatives:
- TUMLU extends the MMLU paradigm to Turkic languages using questions natively authored in Turkish, thus avoiding translation artifacts and maintaining morphosyntactic fidelity [2502.11020].
- TurkishMMLU provides a larger (10,032 questions) high-school–level benchmark, drawn from the EBA platform of the Turkish Ministry of Education, with explicit empirical difficulty calibration based on student correctness ratios [2407.12402].

Performance on these related benchmarks reaffirms that top proprietary LLMs (GPT-4o, Claude-3.5) attain ≈85% accuracy with few-shot or Chain-of-Thought prompting, while open-source LLMs lag (~70%). Chain-of-Thought yields only modest improvements except for select subjects (notably Turkish Literature), and mathematical reasoning remains a clear deficit.

## 6. Error Analysis, Limitations, and Recommendations

Several error modes hamper model performance:
- Mis-segmentation of Turkish agglutinated or morphologically complex tokens leads to 5–10% drops in accuracy.
- Culturally nuanced distractors and reasoning-heavy questions (multi-step inference, paraphrasing, synonym/antonym discrimination) yield high rates of model confusion.
- Catastrophic forgetting during Turkish-specific fine-tuning is prevalent unless continual learning techniques (such as Elastic Weight Consolidation) are employed.
- Difficulty is systematically higher in native Turkish items compared to machine-translated Turkic datasets, complicating cross-linguistic benchmarking [2501.00593][2502.07057][2502.11020][2508.13044].

Recommended development directions:
- Invest in Turkish-aware subword tokenizer design with explicit modeling of agglutination and vowel harmony.
- Explore robust fine-tuning protocols (layer freezing, curriculum learning) for reduced forgetting.
- Extend evaluation beyond multiple-choice to open-ended, generative, and multimodal tasks.
- Augment datasets via controlled morphological transformations and synthetic data augmentation [2501.00593][2502.07057][2508.13044].

## 7. Cultural and Linguistic Relevance

TR-MMLU ensures representativeness not only through linguistic alignment but also cultural authenticity:
- Questions systematically cover local canonical knowledge (e.g., Ottoman history, Anatolian geography), idiomatic usage, and statutory terminology.
- Crafting by Turkish educators and experts maintains morphosyntactic awareness (agglutination, suffixation, vowel harmony), demanding deeper understanding from LLMs than dictionary lookups or translated corpora provide [2501.00593][2502.11020].

## Conclusion

TR-MMLU establishes a transparent, rigorous, and natively aligned benchmark for Turkish NLP, effectively extending the multi-paradigm evaluation framework of MMLU to a morphologically and culturally complex resource-limited language. Its methodological rigor in question curation, tokenization assessment, and multi-domain coverage offers a template for similar benchmarks in other low-resource languages. While state-of-the-art LLMs demonstrate strong baseline reasoning, persistent challenges exist in mathematical reasoning, handling linguistic nuance, and mitigating domain-specific catastrophic forgetting. Future work will benefit from continued expansion into open-ended tasks, even tighter linguistic alignment, and elaborated qualitative error analyses, driving equitable advances in Turkish—and broader multilingual—language modeling [2501.00593][2502.07057][2502.11020][2407.12402][2508.13044].

Source: https://www.emergentmind.com/topics/turkish-massive-multitask-language-understanding-tr-mmlu