--- title: 'TigerLLM: Open-Source Bangla LLMs' url: https://www.emergentmind.com/topics/tigerllm type: topic --- # TigerLLM: Open-Source Bangla LLMs TigerLLM is a family of open-source Bangla large language models (LLMs) designed to address limitations in both quality and reproducibility of previous Bangla LLMs. Through a combination of carefully curated monolingual corpora and a two-stage training pipeline, TigerLLM achieves state-of-the-art results on six Bangla-specific NLP benchmarks, outperforming all open-source alternatives and surpassing proprietary models such as GPT-3.5 on most tasks. Its architecture, training procedures, datasets, benchmarking results, and open-source protocols establish TigerLLM as a new reference point for Bangla language modeling [2503.10995]. ## 1. Model Family Overview TigerLLM consists of two primary variants, both using standard Transformer-decoder architectures: - **TigerLLM-1B**: Built on the LLaMA 3.2 architecture with 1 billion parameters. - **TigerLLM-9B**: Based on the Gemma-2 architecture with 9 billion parameters. Both variants undergo continual pretraining on a domain-specific "Bangla-TextBook" corpus and full-parameter finetuning using the "Bangla-Instruct" instruction tuning dataset. No architectural modifications are introduced beyond their respective base backbones. FlashAttention is employed for training efficiency, reducing memory consumption and accelerating attention computations [2503.10995]. ## 2. Training Data and Preparation TigerLLM's superiority is rooted in its data curation pipeline, consisting of two key datasets: - **Bangla-TextBook Corpus**: Derived from 163 open-source textbooks (Grades 6–12), this corpus contains approximately 9.9 million tokens and 697,903 sentences spanning science, mathematics, social studies, and language arts. Preprocessing includes HTML/PDF cleaning, sentence detection, and tokenization using the base model’s native tokenizer. - **Bangla-Instruct Dataset**: Starting from 500 seed tasks authored by 50 native experts in 10 distinct domains, iterative self-instruct paradigms using GPT-4o and Claude-3.5-Sonnet expand the set to 100,000 instruction–response pairs. Filtering enforces strict criteria on language, cultural sensitivity, quality, and novelty, combining automated classifiers with human verification. The filtering pipeline is formalized as: \[ \mathcal{F}(i, r) = \mathbf{1}\bigl[\, \mathcal{L}(i, r)\,\wedge\,\mathcal{C}(i, r)\,\wedge\,\mathcal{Q}(i, r)\,\wedge\,\mathcal{N}(i, r)\bigr] \] where $\mathcal{L}$, $\mathcal{C}$, $\mathcal{Q}$, and $\mathcal{N}$ denote binary predicates on language adherence, cultural sensitivity, quality, and novelty [2503.10995]. ## 3. Training Methodology Training is composed of continual pretraining followed by instruction finetuning: - **Pretraining**: - Platform: 8 × NVIDIA A100 40GB GPUs (Lambda Labs cluster) - Wall-time: ≈120 hours - Hyperparameters: Batch size per GPU = 64, gradient accumulation steps = 16, epochs = 4, linear learning rate warmup (initial rate $5 \times 10^{-6}$), precision = BF16, gradient checkpointing. - **Finetuning**: - Platform: Single NVIDIA A100 40GB GPU (Google Colab) - Wall-time: ≈96 hours per variant - Sequence length: 2048 tokens - Mixed-precision (8-bit AdamW optimizer), cosine learning-rate schedule | Variant | Batch Size | Grad. Accum. | Epochs | LR (initial) | Weight Decay | |---------------|------------|--------------|--------|--------------|--------------| | TigerLLM-1B | 16 | 4 | 3 | $1 \times 10^{-5}$ | 0.02 | | TigerLLM-9B | 32 | 8 | 3 | $1 \times 10^{-6}$ | 0.04 | Both pretraining and finetuning jointly minimize the next-token cross-entropy loss: \[ \mathcal{L}(\theta) = -\sum_{t=1}^T \log p_\theta(x_t \mid x_{