---
title: Agaram 2.5B
url: https://www.emergentmind.com/topics/agaram-2-5b
type: topic
---

# Agaram 2.5B

Agaram 2.5B is a 2.53-billion-parameter Tamil–English language model trained from scratch by Murai Labs, with Tamil comprising approximately 58% of its 50-billion-token pretraining mixture. It is a dense decoder-only Transformer designed for Tamil script, romanized Tamil (Tanglish), English, translation, instruction following, and structured tool calling. The released chat model uses a Tamil-oriented byte-level BPE tokenizer, supervised fine-tuning, and targeted behavioral-training rounds. Its strongest reported capabilities are Tamil–English translation, Tamil grammaticality, formatting, and selected tool-use behaviors; its principal limitation is weak factual knowledge and reading comprehension. The model, its base checkpoint, GGUF conversions, and code were released alongside the project report [2610.18420].

## 1. Development context and objectives

Agaram 2.5B was developed as a Tamil-first model rather than as an adaptation of an English-centered pretrained model. The authors identify inefficient tokenization as a central problem for general-purpose language models applied to Tamil. In their measurements, Qwen2.5 tokenized Tamil words into 9.44 tokens on average, compared with 1.35 tokens for English words. This increases training and inference cost and reduces the amount of Tamil that can fit into a fixed context window.

The project therefore adopted four principal design objectives:

1. train from random initialization;
2. construct a tokenizer using Tamil-representative data;
3. make Tamil the majority language in pretraining;
4. include Tamil script, Tanglish, books, web text, OCR, code, mathematics, translation, and conversational data.

Training from scratch also enabled the authors to track the training corpus and conduct contamination checks. By contrast, adapting an existing model would preserve its parent tokenizer; adding Tamil tokens to such a model would leave the new embeddings initially undertrained.

Agaram 2.5B was substantially redesigned in response to the earlier Agaram 1.5B model. The first model used a SentencePiece normalizer that erased newlines and indentation, lacked newline and chat-stop exposure during pretraining, packed documents without separators, and suffered from insufficiently measured data composition. Its final checkpoint was also lost when the training node shut down. Agaram 2.5B introduced newline-preserving tokenization, explicit special tokens, document-isolated attention, a measured 50-billion-token mixture, off-node checkpoint copying, and evaluation-set contamination scanning.

Early architecture experiments also informed the final design. At approximately 91 million parameters, a dense Transformer had slightly higher held-out loss than a Mamba-2 hybrid at 25,000 steps but was clearly better at 75,000 steps. Mamba-3 was unstable, and Gated DeltaNet trailed the dense Transformer. Both Agaram models consequently use dense Transformer blocks.

## 2. Architecture and tokenizer

Agaram 2.5B is a dense decoder-only Transformer with 2,525,757,696 parameters. Its principal configuration is:

| Component | Configuration |
|---|---:|
| Transformer layers | 28 |
| Hidden size | 2,304 |
| Attention | Full multi-head self-attention |
| Attention heads | 36 |
| Head dimension | 64 |
| MLP | SwiGLU, width 9,216 |
| Normalization | RMSNorm |
| Position encoding | RoPE, base 500,000 |
| Vocabulary | 64,000 tokens |
| Context length | 8,192 tokens |
| Precision | bf16 |
| Embeddings | Tied input and output |

The model uses pre-normalized Transformer blocks without bias terms. The tied embedding matrix has shape $64{,}000\times2{,}304$. The parameter breakdown is 21,233,664 parameters per attention component, 63,700,992 per MLP component, and 4,608 for the normalization terms in each layer. The 28 Transformer layers contain 2,378,299,392 parameters; the tied embedding contributes 147,456,000 parameters, and the final norm contributes 2,304 parameters.

### Document-isolated attention

Training examples pack multiple documents into 4,096- or 8,192-token sequences, but attention is prevented from crossing document boundaries. Each document ends with an end-of-text token, rotary positions restart for each document, and each document attends only to itself. Flash attention is retained through packed-batch offsets. This design addresses a failure in Agaram 1.5B, where unseparated document concatenation encouraged free-form chat to continue into memorized instruction text.

### Tokenizer design

The model uses `TOK_FRESH_20260924`, a 64,000-token byte-level BPE tokenizer trained with Hugging Face tokenizers. Its training sample contained 930 million characters:

| Category | Share |
|---|---:|
| Tamil | 58.0% |
| English | 29.0% |
| Code | 5.6% |
| Chat and tool text | 4.3% |
| Tanglish | 3.0% |
| Parallel text | 0.1% |

The tokenizer has no normalizer. The packing pipeline applies Unicode NFC before tokenization. It preserves newlines, indentation, multiline structure, chat boundaries, code formatting, and JSON structure. Byte-level BPE begins with all 256 possible byte values and therefore avoids unknown-character fallback. A SentencePiece tokenizer trained for the same task produced 2,257 fallback tokens on code, whereas the byte-level tokenizer did not require such fallback behavior.

Tamil-aware pre-tokenization extends Qwen2’s word-splitting pattern to include combining marks, U+200C zero-width non-joiners, and U+200D zero-width joiners. This is relevant because Tamil vowel signs are combining marks.

| Sample | TOK_FRESH | Agaram 1.5B | Qwen2.5 |
|---|---:|---:|---:|
| Tamil prose | 1.55 | 1.55 | 9.44 |
| Tamil verse | 2.29 | 1.80 | 9.19 |
| English | 1.41 | 1.67 | 1.35 |
| Tanglish | 1.95 | 2.95 | 2.44 |
| Code | 3.54 | 4.54 | 3.01 |
| Tool-call JSON | 2.52 | 3.46 | 2.35 |
| Plain-text chat | 2.19 | 3.14 | 2.24 |

The new tokenizer did not reduce Tamil prose below the earlier model’s 1.55 tokens per word. Its principal gains were exact preservation of line breaks, lower Tanglish, code, JSON, and chat tokenization cost, elimination of byte fallback, and improved learning of formatting and stopping behavior. Tanglish remains more expensive than Tamil script, at approximately 1.95 tokens per word compared with 1.55 for Tamil prose.

A tokenizer weakness is that 14.3% of Tamil token boundaries fall inside grapheme clusters, compared with 4.7% for the earlier tokenizer. No boundaries fall inside UTF-8 characters, but the effect of grapheme-level splitting was not measured.

Special tokens include end of text, ChatML delimiters, tool-call and tool-response delimiters, padding, and reserved identifiers. During pretraining, documents are encoded with special-token parsing disabled and closed with an end-of-text token. Chat SFT uses ChatML markers, with `<|im_end|>` as the stop token.

## 3. Pretraining data and optimization

The pretraining pack contained 61.3 million documents after deduplication, drawn from 87.7 million initial documents. It contained 39.39 billion unique tokens from 68 sources and was expanded through repetition to approximately 50 billion training tokens. Because high-quality Tamil supply was limited, Tamil sources were repeated up to four times. After deduplication, native Tamil sources contained approximately 16–19 billion tokens rather than the initially estimated 45 billion.

The planned 50-billion-token composition was:

| Category | Share | Approximate tokens |
|---|---:|---:|
| Tamil | 58.2% | 29.1B |
| English | 19.8% | 9.9B |
| Math | 10.0% | 5.0B |
| Code | 7.0% | 3.5B |
| Parallel text | 2.0% | 1.0B |
| Instruction | 3.0% | 1.5B |

Tamil sources included FineWeb2, Sangraha, CulturaX, HPLT, OSCAR, Wikipedia, Project Madurai, OCR of Tamil books, Tamil web crawls, and Tamil e-books and archives. Other sources included FineWeb-Edu, Nemotron-CC, FineMath, Stack-Edu, Nemotron code, BPCC, FineTranslations, and instruction and task data.

### OCR and filtering

OCR-derived text represented approximately 1.8% of stable-stage tokens and 0.9% of decay-stage tokens. Three OCR pipelines were used:

1. Tesseract OCR over Tamil Digital Library scans;
2. indic-ocr over 976 PDFs totaling 142,271 pages;
3. Sarvam re-OCR over 358 books totaling 59,841 pages.

Filtering removed runaway repetition, watermarks, empty table markup, picture descriptions, legacy font encodings, and interleaved columns. The authors manually inspected rejected material because an earlier filter had discarded good Tamil text in 37 of 41 examined cases.

### Stable and decay stages

Pretraining comprised stable and decay stages:

| Quantity | Stable stage | Decay stage |
|---|---:|---:|
| Context | 4,096 | 8,192 |
| Sequences per step | 128 | 64 |
| Steps | 82,970 | 12,398 |
| Tokens | 43.50B | 6.50B |
| Learning rate | Warmup to $3\times10^{-4}$, then constant | $3\times10^{-4}\to3\times10^{-5}$ |
| Wall time | 130.1 h | 17.6 h |
| Median throughput | 94,700 tokens/s | 107,500 tokens/s |
| Validation loss, first | 4.041 | 2.080 |
| Validation loss, last | 2.441 | 1.903 |

Each optimizer step processed 524,288 tokens. Optimization used Adam with $\beta=(0.9,0.95)$, weight decay 0.1, gradient clipping at 1.0, bf16 mixed precision, and optimizer-state distribution across eight GPUs. The decay stage reduced the learning rate to one tenth of its peak and shifted the data mixture toward higher-quality material. FineWeb2 Tamil, verified Sangraha, Project Madurai, FineMath-4+, and high-scoring FineWeb-Edu all received larger proportions in the decay stage. Instruction and task sources represented approximately 23% of decay-stage tokens.

A stable-stage power-law fit was reported as:

$$
L(s)=2.25+54.1s^{-0.5}.
$$

Fitting steps 3,000–30,000 predicted later validation loss to within 0.004.

## 4. Fine-tuning, tool calling, and release

The final supervised fine-tuning corpus contained 134,428 training conversations and 2,736 validation conversations, with an 8,192-token maximum sequence length and loss computed only on assistant turns. The general SFT stage used three epochs. The base dataset included multi-user-turn conversations, tool-calling conversations, Tamil-script conversations, Tanglish conversations, open questions, explanations, memory and correction, casual Tanglish, teaching, practical help, Thirukkural, format following, safety, passage tasks, uncertainty, refusals, reasoning, and translation.

Most assistant replies were generated by open-weight teacher models, primarily Gemma-4-31B, together with scripted templates. The general SFT procedure used full-parameter fine-tuning, packed sequences totaling 65,536 tokens per optimizer step, a learning rate of $2\times10^{-5}$, 3% warmup, cosine decay, AdamW, gradient clipping at 1.0, and bf16. Validation loss decreased from 1.416 for the base model to 1.096 after the first epoch and 1.093 after the second and third epochs.

Three subsequent targeted rounds added 11,226 new rows and replayed earlier data. They addressed identity, tool results, crisis responses, refusals, long answers, false premises, memory, Thirukkural, script switching, and harmful requests. These rounds improved selected targeted behaviors but did not reliably improve broad factual performance.

### Output modes

Agaram 2.5B was trained to answer in:

1. Tamil script;
2. Tanglish;
3. English.

It also supports structured tool calling through dedicated tool-call and tool-response delimiters. On 25 tool prompts whose schemas appeared in training, the released model selected the correct tool 23 times and produced the correct tool and exact arguments 13 times.

| Tool-calling measure | Result |
|---|---:|
| Correct tool selected | 23/25, 92% |
| Correct tool and exact arguments | 13/25, 52% |

The general SFT checkpoint achieved 24/25 correct tool selections and 15/25 exact argument matches; the released checkpoint was slightly worse. Because the tool schemas appeared in training, the evaluation does not measure general schema generalization. Scoring was strict and evaluated the emitted call rather than the usefulness of an external tool’s result.

The model also showed weak grounding. When a simple Wikipedia lookup returned irrelevant text for four of five questions, the model paraphrased the irrelevant result in all four cases rather than recognizing that the tool output was off-topic.

### Targeted behavior

The targeted rounds improved some identity and crisis probes. Identity performance increased from 6/15 fully correct after the second round to 10–11/15 after the third, although the released model was fully correct on 8/15, partly correct on 4/15, and wrong on 3/15. The Tele-MANAS number 14416 appeared in 7/10 released crisis responses, and emergency number 112 appeared in 4/10.

These results are not independent evaluations because the rounds were constructed directly from observed failures. Persistent errors included accepting a false claim that the model has 70 billion parameters, denying that Murai Labs created it after correctly naming Murai Labs, failing to refuse a request to disable office security logs, and hallucinating identity or institutional provenance.

The released model, base model, GGUF files, and code are available through the project’s Hugging Face and GitHub repositories. The release includes safetensors weights, tokenizer files, GGUF conversions, 16-bit, 8-bit, and 4-bit formats, a llama.cpp conversion script, parity-check code, a chat script, model cards, and a crisis footer in the chat script. The license was pending legal review and upstream permissions at the time described in the report.

## 5. Evaluation and capabilities

### Translation

Agaram 2.5B performed strongly on Tamil–English translation. The released chat model exceeded Sarvam-1 in all six reported FLORES and IN22 directions:

| Dataset and direction | Released Agaram | Sarvam-1 |
|---|---:|---:|
| FLORES en→ta | 54.3 | 49.2 |
| FLORES ta→en | 55.2 | 44.2 |
| IN22-Conv en→ta | 41.0 | 37.4 |
| IN22-Conv ta→en | 45.9 | 37.9 |
| IN22-Gen en→ta | 48.4 | 45.9 |
| IN22-Gen ta→en | 53.9 | 42.7 |

The base model achieved 55.1 chrF for FLORES English-to-Tamil and 55.5 for Tamil-to-English. Contamination-controlled evaluation changed scores by at most 1.1 chrF, so the translation advantage did not depend materially on detected contamination.

### Tamil instruction following

IndicIFEval measures exact constraint following, including required keywords, formats, word counts, placeholders, and structural requirements.

| Model/checkpoint | Native strict | Translated strict |
|---|---:|---:|
| Agaram 2.5B base | 10.0% | 19.6% |
| SFT epoch 1 | 19.3% | 16.0% |
| SFT epoch 2 | 18.6% | 16.0% |
| SFT epoch 3 | 18.6% | 17.0% |
| Released model | 19.3% | 15.3% |

The released model achieved 19.3% strict and 23.0% loose performance on native Tamil, compared with 15.3% strict and 17.8% loose performance on translated Tamil. SFT nearly doubled native-Tamil instruction-following performance from the base model, although the translated-subset score declined. The authors regard the native subset as more informative because the decay mixture included Tamil translations of instruction-following data with similar templates.

### Base-model benchmarks

| Benchmark | Agaram 2.5B | Sarvam-1 |
|---|---:|---:|
| FLORES en→ta | 55.1 | 49.2 |
| FLORES ta→en | 55.5 | 44.2 |
| IN22-Conv en→ta | 40.8 | 37.4 |
| IN22-Conv ta→en | 46.0 | 37.9 |
| IN22-Gen en→ta | 48.8 | 45.9 |
| IN22-Gen ta→en | 54.3 | 42.7 |
| MILU-ta, 5-shot | 29.6 | 28.8 |
| Belebele-ta | 27.8 | 58.3 |
| INCLUDE-ta | 26.9 | 35.5 |
| MMLU-ta | 25.5 | 32.6 |
| XCOPA-ta | 62.0 | 62.0 |
| MultiBLiMP-ta | 99.0 | 97.9 |

The model’s strengths were translation, Tamil grammaticality, some commonsense reasoning, and efficient Tamil and Tanglish generation. Its weaknesses were general knowledge, reading comprehension, and everyday factual questions. Belebele performance was 27.8%, close to the 25% chance level, while Sarvam-1 scored 58.3%. INCLUDE and MMLU were similarly close to chance.

The authors hypothesize that the difference may be related to training volume: Agaram used 50 billion tokens, whereas Sarvam-1 reports approximately 4 trillion total tokens, including approximately 2 trillion high-quality Indic tokens. This remains an inference rather than a controlled causal result.

### Open-chat evaluation

On a 50-question held-out everyday set, the same Kimi K3 judge evaluated the released model three times:

| Pass | Usable replies |
|---|---:|
| 1 | 23/50 |
| 2 | 22/50 |
| 3 | 24/50 |
| Mean | 23.0/50, 46% |

The set contained everyday and school-level facts, single-turn and two-turn interactions, Tanglish, and Tamil-script prompts. Almost every failure was factual rather than stylistic. Examples included suggesting the computer Undo key as the way to erase a pencil mistake and answering 12 when asked how many slippers make a pair.

Judge variability was substantial on an earlier 100-prompt set: identical replies received 57 usable judgments in one pass, 46 in a second, 49 in a third, and 52 in a fourth. The first two passes differed on 13 items, and 18 of 100 items did not receive the same verdict across later passes. The authors therefore report means and ranges rather than relying on a single LLM-judge score.

## 6. Limitations, contamination, and significance

Agaram 2.5B’s principal limitation is a weak factual foundation. It can produce fluent, well-formatted Tamil and Tanglish while providing incorrect answers. On the held-out everyday-question set, only approximately 46% of responses were judged usable, and nearly every failure was attributed to factual error. Additional targeted SFT experiments changed answer format, answer-letter tendencies, refusal behavior, and constraint following, but did not improve broad knowledge measures.

The authors consequently distinguish between post-training behavior and pretrained knowledge. SFT can improve stopping, script selection, tool formatting, and local response patterns, but it did not reliably add general factual knowledge. The report suggests that further progress will likely require substantially more clean, diverse, high-quality Tamil pretraining data rather than simply more narrow SFT.

### Contamination and evaluation caveats

The contamination procedure used 13-token windows and classified an item as clean when fewer than 5% of its windows overlapped with the training files. FLORES and IN22 included English source material originating from Wikipedia sources present in the training corpus. XCOPA, HellaSwag, INCLUDE, and MultiBLiMP were not scanned. Items shorter than 13 tokens could not be checked by the principal method.

IndicIFEval translated prompts were not copied directly, but similar instruction templates occurred in Tamil SmolTalk-derived data. The first two open-chat sets influenced targeted SFT and checkpoint selection and therefore are not independent evaluations. The third 50-prompt set was written after checkpoint selection and is the cleanest chat-quality evaluation, although it still contained one short substring match.

The targeted rounds were also not independent tests: they were constructed from earlier failures, including safety items, crisis probes, identity probes, and a security-log refusal scenario. Their improvements therefore measure targeted behavioral patching rather than generalization.

### Data provenance and reproducibility

The project permitted Claude, Codex, GPT, and Gemini to assist with planning, code, and user turns, but assistant replies were intended to come primarily from open-weight models. Exceptions included 37 lines in 122 fixed identity pairs rewritten with Claude. Public datasets generated upstream by frontier models also entered the decay instruction pool, including Tamil Alpaca and Tamil Orca.

The report identifies unresolved licensing issues involving Tamil Alpaca, indic-ocr output, and Sarvam OCR output. It also does not release the full training corpus, pretraining pack, training code, tokenizer-training code, data-pipeline code, source manifests, complete filtering rules, evaluation configurations, contamination scans, chat prompts, reference answers, tool schemas, judge rubrics, or judge verdicts. The work is therefore partially reproducible rather than fully reconstructible.

### Training resources and broader significance

Pretraining used one Lambda node with eight H100 SXM5 GPUs. Training occupied approximately 149 hours of instance time, including 130.1 hours for the stable stage and 17.6 hours for the decay stage. No restarts, NaNs, stalls, or lost checkpoints were reported. Pretraining cost approximately USD 4,765, while total spending across both Agaram models was approximately USD 13,300, excluding the cost of owned DGX Spark machines, electricity, and labor.

Agaram 2.5B demonstrates that a relatively small model trained from scratch with a Tamil-oriented tokenizer and Tamil-majority corpus can achieve strong Tamil–English translation and useful Tamil-script, Tanglish, and tool-calling behavior. Its results indicate the importance of language-specific tokenization, native-language data, document boundaries, contamination-aware evaluation, and native-speaker testing.

At the same time, the model illustrates the limits of post-training when the pretrained knowledge base is small. Its performance profile is asymmetric: translation and grammatical generation are comparatively strong, while factual knowledge, reading comprehension, and everyday question answering remain weak. The project’s central methodological conclusion is that language-specific engineering can substantially improve under-represented-language capabilities, but targeted instruction tuning cannot substitute for broad, high-quality pretraining data.

Source: https://www.emergentmind.com/topics/agaram-2-5b