Agaram 2.5B: The First Tamil-First Language Model Trained from Scratch with Tool Calling. A Playbook for Under-Represented Languages

Abstract: We trained Agaram 2.5B, a 2.5-billion-parameter Tamil-English LLM, from scratch on about 50 billion tokens (58% Tamil, the rest English, math, code and parallel text) on one node of eight H100 GPUs in about six days, with a new tokenizer that, unlike our first model's, keeps line breaks. Supervised fine-tuning on 134,428 conversations, whose replies an open-weight teacher model wrote, and three short targeted rounds turned it into a chat model that answers in Tamil script, romanized Tamil (Tanglish) and English and can call tools: on 25 tool prompts whose schemas appear in training it picks the right tool 23 times and gives exactly the expected arguments 13 times. The released model translates better than Sarvam-1, a 2B Indic base model, in all six FLORES and IN22 directions (FLORES chrF 54.3 into Tamil and 55.2 into English, against 49.2 and 44.2), and nearly doubles its base model on native-Tamil IndicIFEval (19.3% from 10.0%). It is weak on knowledge: reading-comprehension and knowledge benchmarks sit near chance, and on 50 held-out everyday questions written by one native speaker, one LLM judge rated 46% of its replies usable (22-24 of 50 over three passes; 95% interval for 23 of 50, 33-60%), with nearly every failure a wrong fact. Our two further SFT experiments changed answer format but not the knowledge measures. We also describe our first model, Agaram 1.5B, whose failures shaped the choices here; how the judge's scores varied between passes on identical replies; and, item by item, what the targeted rounds trained against and where frontier-model text entered the data. The project ran on compute credits and our own funds, about USD 13,300 across both models; the weights are released (licence to be announced after legal review), with GGUF files for llama.cpp, at https://huggingface.co/murai-labs/agaram-2.5b-instruct; code at https://github.com/Murai-Labs/Agaram2.5b

• The paper details the successful creation of Agaram 2.5B, a 2.53-billion-parameter Tamil–English language model trained from scratch, optimized for Tamil language tasks, notably achieving 55.1 chrF on English-to-Tamil translation.
• Agaram 2.5B addresses data and tokenization challenges that hinder under-represented languages, employing a custom tokenizer that reduces average Tamil tokenization to 1.55 tokens per word, compared with 9.44 tokens in Qwen2.5.
• The methodology includes novel checkpoints and rigorous operational practices that significantly alleviate complexity and resource consumption during both training and fine-tuning phases, achieving a 94,700 tokens per second throughput during pretraining.

Motivation and contribution

Agaram 2.5B presents a complete development account of a 2.53-billion-parameter Tamil–English LLM trained from scratch, with Tamil as the majority pretraining language and tool calling as an explicit post-training capability (2610.18420). The paper’s central claim is not that a small model can match large multilingual systems generally, but that a carefully engineered, language-specific pipeline can produce a useful Tamil-oriented model under a constrained budget. The resulting system is technically competent in translation, formatting, script control, and schema-conditioned tool invocation, while remaining substantially deficient in factual knowledge, reading comprehension, safety robustness, and general conversational reliability.

The work is motivated by a concrete tokenizer and data imbalance problem. On the authors’ samples, Qwen2.5 tokenizes Tamil words into an average of 9.44 tokens, compared with 1.35 tokens for English words. Agaram’s tokenizer reduces Tamil prose to 1.55 tokens per word. This difference is important for a small model because token inefficiency consumes context and compute without increasing linguistic coverage. The authors therefore investigate whether Tamil-first pretraining, rather than adaptation of an English-centric checkpoint, can provide a better quality–cost trade-off for an under-represented language.

The paper also functions as a reproducibility report. It documents two training attempts, reports failures rather than only successful configurations, describes data and contamination checks, and provides operational details for completing a single large training run on rented hardware. The resulting model, tokenizer, base checkpoint, instruction-tuned checkpoint, GGUF conversions, and associated tooling are released, although the full training corpus and several evaluation artifacts are not.

From the first model to Agaram 2.5B

Agaram 2.5B was preceded by a 1.5-billion-parameter model whose failures determined many of the second model’s design choices. The first model used a SentencePiece tokenizer whose normalization erased all line breaks and indentation. As a result, the model did not observe newlines during pretraining, never learned the chat stop token in its natural training context, and performed poorly on lists, verse, code, and conversational termination. Its continued-pretraining data also lacked document separators, allowing attention to cross document boundaries and causing free-form generation to continue into memorized instructional material.

The first model’s post-training results were particularly poor. On a 100-prompt open-chat evaluation, it produced only 2 usable responses. Adding approximately 14,000 native-Tamil conversations increased this result only to 6–8 responses, leading the authors to conclude that the base model rather than the SFT data was the primary bottleneck. This is a useful negative result: high-quality instruction data cannot reliably compensate for a weak or structurally damaged pretrained representation.

The authors also evaluated Mamba-2, Mamba-3, and Gated DeltaNet hybrids against dense Transformers at approximately 91 million parameters. At 25,000 steps, the Mamba-2 hybrid was marginally better on held-out Tamil loss, $6.891 \pm 0.011$ versus $6.901 \pm 0.006$. At 75,000 steps, however, the dense Transformer was clearly superior: $7.003 \pm 0.055$ versus $7.278 \pm 0.078$, despite the hybrid having lower training loss. Mamba-3 was unstable across seeds, with a held-out-loss standard deviation approximately ten times that of the dense model. The authors consequently selected a conventional dense Transformer, while explicitly limiting the conclusion to the tested scale and training conditions.

Data, tokenizer, and model design

The final tokenizer is a 64,000-token byte-level BPE with no normalizer. It preserves line breaks, indentation, and arbitrary byte sequences, and its pre-tokenizer treats Unicode combining marks and zero-width joiners as part of words. This modification is particularly relevant to Tamil, whose vowel signs are combining marks. The tokenizer exactly round-trips the evaluated text, unlike the first model’s tokenizer, which failed on multiline material.

The tokenizer’s advantages are uneven. It matches the first tokenizer on Tamil prose at 1.55 tokens per word, but improves Tanglish from 2.95 to 1.95 tokens per word, code from 4.54 to 3.54, tool-call JSON from 3.46 to 2.52, and plain-text chat from 3.14 to 2.19. It also reduces Tamil-script cost relative to English-oriented tokenization. However, 14.3% of Tamil token boundaries fall inside grapheme clusters, compared with 4.7% for the previous tokenizer, and verse becomes more expensive at 2.29 tokens per word. The paper does not measure the modeling consequences of these grapheme-internal boundaries, leaving tokenizer quality only partially characterized.

The pretraining corpus contains 39.39 billion unique tokens expanded to approximately 50 billion training tokens through repetition. Tamil accounts for 58.2%, English for 19.8%, mathematics for 10%, code for 7%, parallel text for 2%, and instruction data for 3%. After deduplication, the available native Tamil material amounted to only 16–19 billion tokens, so some Tamil sources were repeated up to four times. The corpus combines web crawls, books, OCR-derived text, Project Madurai, Wikipedia, parallel corpora, mathematics, code, and instruction-shaped material.

The final 6.5-billion-token decay stage changes both the learning rate and the data distribution. Broad web sources are downweighted while curated Tamil, educational English, mathematics, and instruction sources are upweighted. This produces substantial improvements in held-out Tamil bits per character, especially for literary EPUB text, which improves by 12.8%. The interpretation is qualified: other passages from many of those books were present in the training data, so the literary result measures generalization within partly observed books rather than generalization to unseen literature.

Architecturally, Agaram 2.5B is a dense decoder-only Transformer with 28 layers, hidden size 2,304, 36 attention heads, SwiGLU feed-forward layers of width 9,216, RMSNorm, rotary embeddings with a 500,000 base, tied input and output embeddings, and an 8,192-token context window. Training uses bf16, Adam with weight decay, gradient clipping, and document-local attention masks. Documents are packed into sequences, but attention does not cross end-of-text boundaries and rotary positions restart for each document. This directly addresses the first model’s contamination of document contexts.

Training scale and operational methodology

Pretraining was conducted on one node containing eight H100 GPUs. The stable stage processed 43.5 billion tokens at a 4,096-token context, followed by a 6.5-billion-token decay stage at an 8,192-token context. The stable stage took 130.1 hours and achieved median throughput of 94,700 tokens per second; the decay stage took 17.6 hours at 107,500 tokens per second. The training run completed without restarts, numerical failures, or throughput collapses.

The operational design is one of the paper’s more practically valuable contributions. Checkpoints were copied off-node every two hours, archived independently, verified with hashes, and protected by a termination guard that prevented cloud-instance shutdown until the final checkpoint and logs had been copied. This procedure was explicitly derived from the first model’s lost final checkpoint. The production pretraining run cost approximately USD 4,765 in rented compute, below a USD 5,000 cap.

The authors report that a power law fitted using only steps 3,000–30,000 forecast all later stable-stage validation losses to within 0.004. This suggests that inexpensive early-run diagnostics can provide useful budget and stopping information, although the claim is based on one training trajectory and therefore should not be interpreted as a validated scaling law.

Translation performance and benchmark profile

Translation is the strongest capability of the base model. Agaram 2.5B outperforms Sarvam-1, a roughly 2-billion-parameter multilingual Indic model, in all six reported FLORES and IN22 directions. On FLORES, Agaram achieves 55.1 chrF for English-to-Tamil and 55.5 for Tamil-to-English, compared with Sarvam-1’s 49.2 and 44.2. On IN22-Conv it scores 40.8 and 46.0 against 37.4 and 37.9; on IN22-Gen it scores 48.8 and 54.3 against 45.9 and 42.7.

These gains survive contamination filtering. Restricting translation evaluation to pairs whose Tamil and English sides have less than 5% overlap with the training pack changes Agaram’s scores by at most 1.14 chrF, and the model remains ahead of Sarvam-1 in every direction. This is an important qualification because portions of the English FLORES and IN22 material originate in Wikipedia-derived sources present in the training corpus.

The released chat model preserves nearly all of the base model’s translation performance. Its FLORES scores are 54.3 chrF for English-to-Tamil and 55.2 for Tamil-to-English, while IN22 results remain within approximately one point of the base checkpoint. The SFT process therefore improves conversational formatting and instruction following without materially sacrificing translation.

The benchmark profile is sharply asymmetric. Agaram reaches 99.0% on Tamil MultiBLiMP and 62.0% on XCOPA, approximately matching Sarvam-1 on commonsense and exceeding it on grammatical minimal pairs. In contrast, it performs near chance on knowledge and reading-comprehension tasks: 27.8 on Belebele-Tamil, 26.9 on INCLUDE-Tamil, and 25.5 on MMLU-Tamil. Sarvam-1 obtains 58.3, 35.5, and 32.6 respectively. MILU performance is similar between the models, at approximately 29%.

The authors infer that the knowledge gap may reflect training volume: Sarvam-1 reports approximately 4 trillion training tokens, including 2 trillion high-quality Indic tokens, whereas Agaram uses 50 billion. This remains an inference rather than an experimentally isolated cause. Model scale, corpus composition, data quality, multilingual transfer, and pretraining duration are confounded.

Supervised fine-tuning and tool calling

The general SFT corpus contains 134,428 conversations, including Tamil script, Tanglish, English, multi-turn interactions, tool calls, translation, safety, crisis-support, and formatting data. Most assistant responses were generated by open-weight teacher models, primarily Gemma-4-31B, with loss applied only to assistant turns. The model was trained for three epochs, although almost all validation-loss improvement occurred during the first epoch.

SFT converts the base model into a substantially more usable conversational system in several narrow respects. The proportion of replies passing scripted formatting checks increases from 6–9% for the base model to approximately 88–95% after SFT. The proportion of replies stopping autonomously increases from 15% to nearly 100%. Native-Tamil IndicIFEval accuracy rises from 10.0% to 19.3% at the released checkpoint, while Qwen2.5-1.5B-Instruct scores 11.2% in the authors’ harness.

The translated IndicIFEval subset behaves differently. The base model scores 19.6%, falling to 15.3% after SFT. The paper identifies template exposure as a confound: Tamil versions of IFEval-like instruction templates entered the decay-stage corpus through SmolTalk-derived data. Consequently, the base-model score on the translated subset may overstate general instruction following. The native subset is the more defensible measure of SFT improvement.

Tool calling is a principal claimed capability. On 25 prompts whose tool schemas also appear in training, the released model selects the correct tool 23 times, or 92%, and produces exactly the expected arguments 13 times, or 52%. The general SFT checkpoint obtains 24 correct tools and 15 exact argument sets. These results demonstrate schema-conditioned invocation rather than robust tool use in open environments. The evaluation does not test unseen schemas, execution success, recovery from tool errors, or grounding in unreliable tool outputs.

The paper’s targeted SFT rounds address identity, crisis responses, harmful requests, tool-result handling, constraint memory, and multi-turn repair. They improve the specific probes used to construct them. For example, Tele-MANAS appears in 7 of 10 crisis-probe responses in the released model, and identity responses become more resistant to several false model-attribution claims. However, these probe results are not independent evaluations because the rounds were designed from failures on related prompts.

Factuality and conversational quality

The most consequential result concerns factual reliability. On a held-out set of 50 everyday questions written by one native speaker after the release checkpoint was selected, the model receives usable judgments on 23, 22, and 24 items across three passes, for a mean of 23.0 or 46%. The set contains Tanglish and Tamil-script questions, single-turn and two-turn interactions, and simple factual prompts concerning ordinary objects, explanations, and social interactions.

Nearly every failure is factual rather than stylistic. In one example, the model recommends the Undo key for erasing a pencil mark; in another, it answers that a pair contains 12 slippers. In the first evaluation pass, 26 of 27 unusable replies were marked factually wrong, while almost all still stopped correctly and used the requested script. Thus, high formatting accuracy substantially overestimates practical quality.

The targeted rounds do not produce a detectable improvement on this held-out set. The first SFT stage scores a mean of 21.0 out of 50, while the released model scores 22.3. Across item-level majority judgments, the released checkpoint wins seven items and loses six, yielding no evidence of a systematic difference. The authors correctly state that 50 prompts can detect only a relatively large effect; the result excludes neither modest gains nor domain-specific improvements.

Judge instability further limits the precision of the reported chat scores. On an earlier 100-prompt set, the same replies receive 57 usable judgments in one pass and 46 in a re-pass, with 13 verdicts changing. Later passes yield 49 and 52. The authors therefore report means and ranges rather than relying on a single LLM-judge run. This methodological caution is warranted, but it does not remove the underlying limitation: large-scale human evaluation of the released model is absent.

Limitations and open questions

The paper’s empirical claims are constrained by a one-run training design. Agaram 2.5B has no same-scale random-seed replication, systematic ablation of tokenizer choices, controlled comparison of corpus mixtures, or independent validation of the decay-stage effect. The Transformer-versus-state-space comparison is performed only at approximately 100 million parameters, so it does not establish that dense attention is optimal at the final scale.

The evaluation also has incomplete contamination coverage. XCOPA, HellaSwag, INCLUDE, and MultiBLiMP were not scanned against the training pack, and short evaluation items cannot be reliably assessed with the 13-token-window procedure. The authors disclose these gaps rather than treating the benchmark suite as uncontaminated.

Safety evidence is especially limited. There is no red-teaming campaign, no independent judge evaluation of safety training rows, and crisis data was reviewed by an AI reader rather than a human specialist. Despite 47 targeted refusal examples, the released model still responds affirmatively to a request for disabling office security logs, even though the technical instructions are incorrect. It also exhibits identity confabulation, over-refusal, inconsistent crisis behavior, and unverified grounding in off-topic tool results.

The release is not fully reproducible in the strict sense. Weights, tokenizer files, conversion scripts, and model cards are available, but the training corpus, source manifests, filtering rules, training code, evaluation prompts, tool schemas, judge rubric, responses, and verdicts are not. Licensing remains complicated by upstream dataset and OCR-tool terms, including material whose usage conditions may not be compatible with unrestricted redistribution. These issues affect the legal and scientific reproducibility of the release.

The principal open empirical question is whether the knowledge deficit is primarily caused by the 50-billion-token pretraining budget, insufficient high-quality Tamil data, model capacity, or corpus composition. The authors suspect that more pretraining would help, but the paper does not test this hypothesis. It also leaves open whether additional Tamil data would improve factuality without reducing translation and instruction-following performance, whether tool grounding can be made robust to irrelevant retrieval results, and whether grapheme-internal tokenizer boundaries measurably affect Tamil generation.

Conclusion

Agaram 2.5B demonstrates that a small team can train and release a Tamil-first model from scratch at moderate compute cost, provided that tokenizer design, corpus construction, document boundaries, checkpoint management, contamination analysis, and native-speaker evaluation are treated as first-class engineering problems (2610.18420). Its strongest results are in Tamil–English translation, grammatical discrimination, script-sensitive formatting, and tool selection on seen schemas. Its weaknesses are equally clear: near-chance knowledge performance, unreliable everyday factuality, limited safety robustness, unstable LLM-judge measurements, and incomplete evaluation independence.

The paper’s main contribution is therefore an empirical playbook with explicit failure analysis rather than a uniformly capable LLM. It establishes that Tamil-first training can yield strong language-specific behavior under a constrained budget, while also showing that post-training cannot substitute for broad factual competence acquired during pretraining.