Banzhida: Multilingual Tibetan Language Model

• Banzhida is a multilingual large language model designed to enhance Tibetan language representation by using a 72GB curated corpus from diverse sources.
• The model adapts tokenization with an expanded 165K-token vocabulary for Tibetan, reducing over-segmentation and computational load.
• Continual pre-training using a Qwen2.5-7B-base backbone and stage-based approaches leads to significant performance gains over comparable models.

Searching arXiv for the specified paper and closely related context. Banzhida is a multilingual LLM supporting Tibetan, Chinese, and English, developed to address the underrepresentation of Tibetan in existing LLMs caused by the scarcity of high-quality training corpora. It is built by curating the largest Tibetan pre-training corpus to date, extending the tokenizer for Tibetan, and continually pre-training a multilingual base model. The reported system couples corpus construction, tokenizer adaptation, continual pre-training, and benchmark creation, and the reported experiments state that it consistently and significantly outperforms both open-source models of similar scale and Tibetan-tailored models across a wide range of tasks (Pan et al., 12 Jul 2025).

1. Corpus construction and data curation

The Tibetan pre-training corpus aggregates over 72 GB of cleaned text drawn from four complementary sources: open-source datasets, web-crawled content, synthetic data, and private data. After aggressive cleaning and deduplication, the final Tibetan monolingual corpus comprises approximately 72.3 GB of raw text, which tokenizes to about 8.5 billion tokens under the extended tokenizer (Pan et al., 12 Jul 2025).

Source	Volume (GB)	Tokens (approx.)
Open-source	9.53	1.12 B
Web-crawled	4.18	0.50 B
Synthetic translations	17.46	2.53 B
Private archives	41.13	4.35 B
Total	72.30	8.50 B

The open-source component includes Tibetan slices of multilingual corpora—Community OSCAR, CulturaX, HPLT-v2, and MC²—and monolingual collections including Tibetan News and Buddhist Digital Resource. The web-crawled component contains 204,789 pages from 55 Tibetan-language websites, covering news, books, Wikipedia, and forums. The synthetic component consists of parallel sentences translated into Tibetan from FineWeb-Edu and Cosmopedia using Google Translate plus instruction templates. The private component consists of legally obtained Tibetan books and religious texts from institutional archives (Pan et al., 12 Jul 2025).

The cleaning and processing pipeline is tailored for Tibetan’s script and linguistic characteristics. Language identification uses a FastText classifier with Tibetan confidence $\geq 0.5$. Quality filtering applies five successive filters: Gopher duplicate filtering, Gopher quality filtering, C4 rules, FineWeb heuristics, and a manually curated sensitive-word blacklist. Deduplication uses MinHash-based block removal via DataTrove. The duplicate-filter thresholds for repeated $n$-grams are reported as $p_2 < 0.20$, $p_3 < 0.18$, $p_4 < 0.16$, and for $n = 5 \ldots 10$ as gradually decreasing from $p_5 < 0.15$ to $p_{10} < 0.10$, where $$p_n = (\#\text{ characters in repeated }n\text{-grams})/(\text{total characters})$$. Documents are also removed if more than $20\%$ of characters lie in duplicated lines, or more than $n$0 of lines or paragraphs are exact duplicates (Pan et al., 12 Jul 2025).

2. Tokenization and vocabulary adaptation

Banzhida adapts tokenization by training a 15 K-merge Byte-Level BPE model on representative Tibetan text and then merging it with Qwen2.5’s original vocabulary, with duplicates removed. The resulting vocabulary size is 165,428 tokens. The report states that this adaptation reduces over-segmentation and yields both higher tokenization coverage for Tibetan and lower computational load (Pan et al., 12 Jul 2025).

On held-out Tibetan text, the tokenizer achieves a compression ratio of 3.9644, compared with 0.7315 for Qwen3-8B and 0.4885 for Llama3-8B. The reported interpretation is that this means fewer tokens per input length and faster inference. Because the tokenizer is a central interface between Tibetan orthography and Transformer computation, this result is consequential for both throughput and context utilization in subsequent continual pre-training (Pan et al., 12 Jul 2025).

The vocabulary expansion is described as incorporating 15 K new BPE merges of frequent Tibetan character combinations, yielding $n$1. In the reported system, tokenizer adaptation is therefore not a peripheral preprocessing step but a structural modification of the model interface for Tibetan (Pan et al., 12 Jul 2025).

3. Backbone architecture and continual pre-training

Banzhida’s backbone is Qwen2.5-7B-base, described as a 32-layer Transformer with hidden dimension $n$2, $n$3 attention heads per layer, and approximately 7 billion parameters. This architecture is preserved for all continual pre-training stages (Pan et al., 12 Jul 2025).

The self-attention mechanism is reported in the standard form

$n$4

where $n$5, $n$6, and $n$7 are the projected queries, keys, and values, and $n$8. The training objective is the standard autoregressive cross-entropy loss over a sequence $n$9:

$p_2 < 0.20$0

with perplexity monitored as $p_2 < 0.20$1 (Pan et al., 12 Jul 2025).

The continual pre-training methodology is divided into two stages. Stage 1, termed language-balanced pre-training, uses approximately 15 B Chinese, 15 B English, and 8.5 B Tibetan tokens after tokenization, with a context window of 4 K tokens. Optimization uses AdamW with decoupled weight decay, maximum gradient norm $p_2 < 0.20$2, learning rate $p_2 < 0.20$3, warmup $p_2 < 0.20$4 of steps, and cosine decay. The reported batch size is 128 tokens per GPU in bf16, and the hardware configuration is 32 $p_2 < 0.20$5 NVIDIA A800 GPUs under DeepSpeed ZeRO Stage 2 (Pan et al., 12 Jul 2025).

Stage 2, termed long-context pre-training, uses 0.56 B Chinese, 2.5 B English, and 1.5 B Tibetan tokens, for a total of 4.56 B tokens for long contexts. The context window is extended to 32 K tokens with RoPE $p_2 < 0.20$6. The reported training schedule uses 1,000 steps, learning rate $p_2 < 0.20$7, and the batch size remains 128. The hardware stack is DeepSpeed ZeRO-3 Offload on the same GPU cluster (Pan et al., 12 Jul 2025).

4. Benchmark suite and evaluation protocol

To evaluate Tibetan capabilities, the report introduces four new multiple-choice reasoning and commonsense benchmarks by translation: HellaSwag $p_2 < 0.20$8 HellaSwag-bo, ARC $p_2 < 0.20$9 ARC-bo, Xcopa $p_3 < 0.18$0 Xcopa-bo, and XStoryCloze $p_3 < 0.18$1 XStoryCloze-bo. The translations were generated with Claude-Sonnet-3.7 and validated by native speakers for semantic, grammatical, and cultural accuracy. These are complemented by existing public benchmarks: TibetanQA, a reading-comprehension dataset with 20 K QA pairs evaluated by Exact Match and $p_3 < 0.18$2, and FLORES-200 for translation quality on zh$p_3 < 0.18$3bo and en$p_3 < 0.18$4bo, evaluated by BLEU (Pan et al., 12 Jul 2025).

The performance metrics are specified as follows. Accuracy is used for multiple-choice and classification tasks. Exact Match and $p_3 < 0.18$5 are used for question answering. BLEU is computed as

$p_3 < 0.18$6

where $p_3 < 0.18$7 are $p_3 < 0.18$8-gram precisions, $p_3 < 0.18$9 are uniform, and BP is the brevity penalty (Pan et al., 12 Jul 2025).

This benchmark design suggests an evaluation strategy that spans reasoning, commonsense, reading comprehension, and translation rather than restricting Tibetan capability assessment to a single task family. The inclusion of both translated and public benchmarks also provides a mixed regime for probing generalization under Tibetan-specific evaluation conditions (Pan et al., 12 Jul 2025).

5. Reported performance and comparative findings

On the reasoning and question-answering benchmarks, the report compares Banzhida variants against Qwen2.5-7B-base, Qwen3-8B, LLaMA3.1-8B-Instr., and DeepSeek-R1-Distill. Qwen2.5-7B-base scores 26.69 on Hellaswag-bo, 29.60 on ARC-bo, 51.80 on Xcopa-bo, 49.97 on XStoryCloze-bo, and 19.31 / 32.38 on TibetanQA EM / F1. Qwen3-8B scores 27.07, 29.16, 50.40, 50.37, and 17.80 / 30.22. LLaMA3.1-8B-Instr. scores 27.28, 29.31, 51.60, 51.69, and 36.09 / 53.04. DeepSeek-R1-Distill scores 25.96, 30.38, 50.40, 48.50, and 9.88 / 17.99. By comparison, Banzhida-Base scores 30.30, 44.32, 59.80, 60.80, and 49.43 / 66.15; Banzhida-32K scores 30.09, 44.62, 58.60, 60.60, and 38.42 / 55.52; and Banzhida-SFT scores 30.64, 48.39, 57.80, 61.86, and 46.15 / 63.15 (Pan et al., 12 Jul 2025).

On FLORES-200 translation, Yak_Llama2_7B reports 11.81 BLEU for zh$p_4 < 0.16$0bo and 11.10 for en$p_4 < 0.16$1bo; Tibetan-Alpaca-7B reports 4.87 and 5.69; Tibetan-Llama2-7B reports 11.81 and 11.10; Banzhida-Base reports 13.81 and 9.20; Banzhida-32K reports 20.86 and 13.56; and Banzhida-SFT reports 21.56 and 11.42 (Pan et al., 12 Jul 2025).

The comparative analysis highlights three explicit gains: on ARC-bo, Banzhida-SFT at 48.39% versus Qwen2.5 at 29.60% for a gain of +18.8 percentage points; on TibetanQA EM, Banzhida-Base at 49.43% versus Qwen2.5 at 19.31% for a gain of +30.1 percentage points; and on zh$p_4 < 0.16$2bo BLEU, Banzhida-SFT at 21.56 versus Yak_Llama2_7B at 11.81 for a relative gain of +82%. The qualitative error analysis further states that Banzhida excels in resolving Tibetan-specific morphology and honorific particles, but occasionally confuses rare religious terms, an issue traced to limited domain coverage in private data (Pan et al., 12 Jul 2025).

A common oversimplification would be to treat the reported variant progression as uniformly monotonic across tasks. The tabulated results do not show that pattern. For example, Banzhida-32K exceeds Banzhida-SFT on en$p_4 < 0.16$3bo BLEU, 13.56 versus 11.42, while Banzhida-Base exceeds Banzhida-32K on TibetanQA EM / F1, 49.43 / 66.15 versus 38.42 / 55.52. The report therefore supports a stronger claim about overall advancement than about a single dominant variant across every benchmark (Pan et al., 12 Jul 2025).

6. Applications, limitations, and projected extensions

The reported applications are machine translation for zh$p_4 < 0.16$4bo and en$p_4 < 0.16$5bo in governmental and educational use, interactive Tibetan educational tools such as vocabulary drills and reading comprehension, and digital preservation and search over Tibetan cultural and religious archives. These applications align with the model’s multilingual scope and the domain composition of the corpus, especially its coverage of books, religious texts, news, and educational material (Pan et al., 12 Jul 2025).

The limitations are stated explicitly. OCRed PDF texts, including many Tibetan books, are not yet included because of licensing and budget constraints. Model scale is limited to 7 B parameters, and larger scales may yield further gains. Domain imbalance remains: news and religious texts are well-covered, while technical and colloquial domains remain sparse. These constraints are material for interpreting both the benchmark results and the error analysis, particularly the confusion over rare religious terms and the absence of stronger evidence in technical or colloquial registers (Pan et al., 12 Jul 2025).

The future directions listed in the report are to incorporate OCR-derived book corpora to expand literary Tibetan; explore 13 B+ parameter variants based on Qwen3-8B to further boost performance; pursue domain-adaptive continual pre-training for Tibetan scientific and legal domains; and use parameter-efficient fine-tuning, specifically LoRA and prefix tuning, for rapid adaptation to downstream tasks. A plausible implication is that Banzhida is intended not only as a fixed multilingual model but also as a basis for subsequent specialization under low-resource constraints (Pan et al., 12 Jul 2025).