- The paper evaluates language models' performance on Sinhala using Unicode and Romanized scripts, with Llama-3.1-8B achieving the strongest all-around performance (2.37 Unicode; 77.18 Romanized).
- Key results show that tokenization efficiency and training data exposure to unfiltered web content are crucial for model proficiency.
- Closed-source models Gemini-1.5-pro and DeepSeek excel at Unicode generation while Claude-3.5-Sonnet shows consistent strong performance across both scripts.
This paper presents the first intrinsic, script-aware benchmark of modern foundational LLMs for Sinhala, evaluating thirteen open-source models via perplexity and four closed-source models via qualitative sentence completion on parallel Unicode and Romanized Sinhala corpora (2601.14958). The central finding is a systematic divergence in model performance across scripts: Mistral-Nemo-Base-2407 achieves the lowest perplexity on Unicode text (2.19), while the smaller Mistral-7B-v0.3 leads on Romanized text (74.76), with Llama-3.1-8B providing the strongest all-around performance (2.37 Unicode; 77.18 Romanized). Among closed-source models, Gemini-1.5-pro and DeepSeek excel at Unicode generation, whereas Claude-3.5-Sonnet is superior on Romanized input.
Motivation and research gap
Sinhala is a low-resource, morphologically rich, agglutinative language whose digital use exhibits a script duality: Unicode Sinhala dominates formal contexts, while Romanized Sinhala prevails on social media and messaging platforms. Prior Sinhala NLP work has concentrated on task-specific systems—synthetic data augmentation for rare-word problems in neural machine translation [(Mel et al., 2024) context; Tennage et al.], forward-backward translation for Sinhala-Tamil yielding roughly 3.4 BLEU improvement over statistical baselines, mBART fine-tuning with 2.8–4.4 BLEU gains (2601.14958), and rule-based versus Seq2Seq transliteration models (Mel et al., 2024). Broader multilingual benchmarks (FLORES-200, IndicGLUE, IndoLLM) target translation or downstream tasks rather than intrinsic language modeling ability, and none examine performance differences between native-script and romanized variants—a gap this paper claims to fill for the first time.
Methodology
The authors compiled a 1000-sentence parallel corpus from blogs and social media (e.g., YouTube), manually transliterating each Romanized sentence to its common Unicode equivalent under a phonetically based rule set reviewed by a native speaker. A 200-sentence evaluation set was selected programmatically: LaBSE sentence embeddings were clustered into 200 groups via K-Means, and the sentence nearest each centroid was chosen, mitigating selection bias; PCA visualization confirmed representative coverage.
Open-source models were evaluated with perplexity. Closed-source models (GPT-4o, Claude-3.5-Sonnet, Gemini-1.5-pro, DeepSeek) were prompted to complete fragments of 50 sentences, scored by a single native Sinhala speaker with linguistic research experience on 3-point scales for Coherence and Grammar/Readability (lower is better).
Perplexity results
| Model |
Parameters |
Unicode PPL |
Romanized PPL |
| Mistral-Nemo-Base-2407 |
12B |
2.19 |
105.35 |
| Llama-3.1-8B |
8B |
2.37 |
77.18 |
| Minitron-8B-Base |
8B |
2.53 |
244.33 |
| phi-4 |
14B |
3.19 |
113.97 |
| Mistral-7B-v0.3 |
7B |
3.62 |
74.76 |
| bloom-560m |
560M |
8.88 |
915.6 |
| Seed-X-PPO-7B |
7B |
668.37 |
121.94 |
Two results stand out. First, parameter count is not predictive of Sinhala proficiency: the 8B Llama-3.1-8B outperforms the 14B phi-4 on Unicode text (2.37 vs. 3.19), and Seed-X-PPO-7B—a multilingual translation-focused model—records an anomalous 668.37 perplexity on Unicode despite competitive Romanized performance (121.94). The authors attribute this to tokenization efficiency and architectural optimization rather than scale. Second, the rank reversal between scripts (Mistral-Nemo best on Unicode but only mid-pack on Romanized; Mistral-7B-v0.3 best on Romanized) implies that pre-training corpus composition, not architecture alone, governs cross-script capability. This directly supports the paper's claim that training data exposure to unfiltered web content drives Romanized Sinhala competence.
Qualitative results
The closed-source evaluation shows a pronounced split. Gemini-1.5-pro scores best on Unicode (Coherence 1.22, Grammar 1.46) but degrades sharply on Romanized text (1.76 / 2.02); DeepSeek shows the most extreme divergence, excelling at Unicode (1.28 / 1.44) yet performing worst overall on Romanized (2.46 / 2.80). Claude-3.5-Sonnet is the only model with consistently strong performance across both scripts (Unicode 1.40 / 1.52; Romanized 1.32 / 1.56), making it the top performer on Romanized input. GPT-4o is middling throughout.
Error analysis identified subject–verb agreement as the dominant failure mode—for example, GPT-4o's completion of "monawada meke karanna..." produced a verb form inconsistent with standard Sinhala sentence-final structure. Even top-scoring completions were typically rated "Acceptable" rather than "Excellent" on grammar, indicating that no evaluated closed-source model reliably produces grammatically native Sinhala without fine-tuning.
Limitations
The authors concede several constraints. The 200-sentence evaluation set may not capture the full range of Romanized spelling variation in digital communication. Qualitative scoring by a single rater introduces subjective bias, with no inter-rater reliability assessment. The benchmark evaluates monolingual text in each script separately and does not assess cross-script mixing, which is itself common in real usage. Additionally, the manual transliteration followed one standardized rule set, though the paper acknowledges that many contextual Romanization variants exist—an assumption that may limit ecological validity of the Romanized perplexity figures.
Conclusion
This benchmark establishes baseline intrinsic language-modeling results for both Sinhala scripts and demonstrates that script variation is a first-order factor in low-resource model evaluation, not a peripheral concern. Practically, it suggests hybrid pipelines that route by script, or fine-tuning on balanced dual-script corpora, with Llama-3.1-8B as an efficient dual-script candidate. The open questions left by the paper are concrete: whether larger naturally occurring corpora would shift the perplexity rankings, how models behave on mixed-script text, and whether multi-rater evaluation would confirm the closed-source performance divergence observed here.