TigerLLM: Open-Source Bangla LLMs

• TigerLLM is a family of open-source Bangla LLMs defined by its state-of-the-art performance on six Bangla NLP benchmarks using curated data.
• It employs a two-stage training pipeline—continual pretraining on the Bangla-TextBook corpus followed by finetuning with the Bangla-Instruct dataset—for efficiency and reproducibility.
• Benchmark results indicate both TigerLLM-1B and TigerLLM-9B outperform existing open-source and proprietary models, establishing a new reference for Bangla language modeling.

TigerLLM is a family of open-source Bangla LLMs designed to address limitations in both quality and reproducibility of previous Bangla LLMs. Through a combination of carefully curated monolingual corpora and a two-stage training pipeline, TigerLLM achieves state-of-the-art results on six Bangla-specific NLP benchmarks, outperforming all open-source alternatives and surpassing proprietary models such as GPT-3.5 on most tasks. Its architecture, training procedures, datasets, benchmarking results, and open-source protocols establish TigerLLM as a new reference point for Bangla language modeling (Raihan et al., 14 Mar 2025).

1. Model Family Overview

TigerLLM consists of two primary variants, both using standard Transformer-decoder architectures:

• TigerLLM-1B: Built on the LLaMA 3.2 architecture with 1 billion parameters.
• TigerLLM-9B: Based on the Gemma-2 architecture with 9 billion parameters.

Both variants undergo continual pretraining on a domain-specific "Bangla-TextBook" corpus and full-parameter finetuning using the "Bangla-Instruct" instruction tuning dataset. No architectural modifications are introduced beyond their respective base backbones. FlashAttention is employed for training efficiency, reducing memory consumption and accelerating attention computations (Raihan et al., 14 Mar 2025).

2. Training Data and Preparation

TigerLLM's superiority is rooted in its data curation pipeline, consisting of two key datasets:

• Bangla-TextBook Corpus: Derived from 163 open-source textbooks (Grades 6–12), this corpus contains approximately 9.9 million tokens and 697,903 sentences spanning science, mathematics, social studies, and language arts. Preprocessing includes HTML/PDF cleaning, sentence detection, and tokenization using the base model’s native tokenizer.
• Bangla-Instruct Dataset: Starting from 500 seed tasks authored by 50 native experts in 10 distinct domains, iterative self-instruct paradigms using GPT-4o and Claude-3.5-Sonnet expand the set to 100,000 instruction–response pairs. Filtering enforces strict criteria on language, cultural sensitivity, quality, and novelty, combining automated classifiers with human verification.

The filtering pipeline is formalized as:

$$\mathcal{F}(i, r) = \mathbf{1}\bigl[\, \mathcal{L}(i, r)\,\wedge\,\mathcal{C}(i, r)\,\wedge\,\mathcal{Q}(i, r)\,\wedge\,\mathcal{N}(i, r)\bigr]$$

where $\mathcal{L}$, $\mathcal{C}$, $\mathcal{Q}$, and $\mathcal{N}$ denote binary predicates on language adherence, cultural sensitivity, quality, and novelty (Raihan et al., 14 Mar 2025).

3. Training Methodology

Training is composed of continual pretraining followed by instruction finetuning:

• Pretraining:
  • Platform: 8 × NVIDIA A100 40GB GPUs (Lambda Labs cluster)
  • Wall-time: ≈120 hours
  • Hyperparameters: Batch size per GPU = 64, gradient accumulation steps = 16, epochs = 4, linear learning rate warmup (initial rate $5 \times 10^{-6}$), precision = BF16, gradient checkpointing.
• Finetuning:
  • Platform: Single NVIDIA A100 40GB GPU (Google Colab)
  • Wall-time: ≈96 hours per variant
  • Sequence length: 2048 tokens
  • Mixed-precision (8-bit AdamW optimizer), cosine learning-rate schedule

Variant	Batch Size	Grad. Accum.	Epochs	LR (initial)	Weight Decay
TigerLLM-1B	16	4	3	$1 \times 10^{-5}$	0.02
TigerLLM-9B	32	8	3	$1 \times 10^{-6}$	0.04

Both pretraining and finetuning jointly minimize the next-token cross-entropy loss:

$$\mathcal{L}(\theta) = -\sum_{t=1}^T \log p_\theta(x_t \mid x_{<t})$$

Pretraining quality is evaluated using perplexity (PPL):

$$\mathrm{PPL} = \exp\!\left(\frac{1}{T}\mathcal{L}(\theta)\right)$$

4. Benchmark Evaluation and Comparative Results

Performance is measured on six standardized Bangla NLP benchmarks, reporting Pass@1 (%) for generative and QA tasks:

Benchmark	GPT-3.5	LLaMA3.2 (11B)	Gemma2 (27B)	Pangea (7B)	Titu-LLM	Bong-LLaMA	Bangla-LLaMA	Bangla-Gemma	TigerLLM-1B	TigerLLM-9B
MMLU-bn	55	22	35	18	6	5	2	18	61	72
PangBench-bn	55	19	51	15	19	12	8	15	55	68
BanglaQuaD	50	21	43	17	8	8	5	12	68	70
mHumanEval-bn	56	15	64	10	2	2	10	10	61	63
BEnQA	50	18	50	14	17	15	11	22	59	65
BanglaRQA	49	20	56	16	21	13	9	19	62	68

Both TigerLLM variants surpass all existing open-source Bangla LLMs. TigerLLM-9B further outperforms GPT4o-mini on all benchmarks, while TigerLLM-1B outperforms several models with much larger parameter counts (7–27B). Relative improvement over previous open-source Bangla models (e.g., Bong-LLaMA, Bangla-LLaMA) ranges from 30–55% (Raihan et al., 14 Mar 2025).

5. Reproducibility and Open-Source Protocols

TigerLLM provides comprehensive reproducibility:

• All code, pretrained weights (1B & 9B), and datasets are released under permissive open-source licenses via GitHub and Hugging Face.
• Detailed scripts for both continual pretraining and finetuning are available, including full hyperparameter annotation (Appendix A of (Raihan et al., 14 Mar 2025)).
• The curation and filtering processes for Bangla-TextBook and Bangla-Instruct are explicitly documented, supporting full pipeline transparency.

This protocol facilitates downstream research, benchmarking, and adaptation to other under-resourced languages.

6. Limitations and Prospects

Key limitations include:

• The Bangla-TextBook corpus is restricted to Grades 6–12 educational texts (≈10M tokens) and thus underrepresents broader linguistic domains.
• The Bangla-Instruct dataset, while diverse (100,000 instructions), may omit coverage of professional and conversational registers.
• Model sizes are constrained to 1B and 9B parameters; benefits of 10B+ scale are undetermined.

Identified directions for further research include qualitative failure analysis, expansion of the training corpus to news/social/literature domains, investigation into scaling beyond 20B parameters while maintaining data quality, and developing Bangla-specific evaluation metrics beyond Pass@1 (Raihan et al., 14 Mar 2025).

TigerLLM demonstrates that focused data curation and rigorous instruction set design can substantially improve performance in under-resourced languages, establishing an open-source baseline for future work in Bangla language technologies.