Tibetan Adaptation Dynamics for LLMs

• Tibetan adaptation dynamics for LLMs are techniques that specialize large language models for Tibetan using continual pretraining, supervised fine-tuning, and parameter-efficient adapters.
• Methodologies include two-stage pipelines, dynamic configuration, and retrieval-augmented generation that enhance performance metrics like perplexity and BLEU scores.
• Empirical evaluations demonstrate significant gains in parameter localization, semantic manifold formation, and downstream task alignment while ensuring reproducibility and scalability.

Tibetan adaptation dynamics for LLMs refers to the empirical, architectural, and algorithmic mechanisms by which LLMs are specialized for Tibetan—a morphologically rich, low-resource language—via techniques such as continual pretraining, supervised fine-tuning, parameter-efficient adapters, and dynamic configuration strategies. This domain investigates the impact of data scarcity, morphological complexity, cross-lingual drift, and resource-efficient adaptation on parameter distributions, representational manifolds, and downstream task alignment in LLMs. Recent work provides the first comprehensive quantitative analyses of adaptation regimes, parameter localization, performance metrics, and the reproducibility of Tibetan-centric LLM pipelines (Chen et al., 3 Dec 2025, Pan et al., 12 Jul 2025, Huang et al., 24 Mar 2025, Mingjun et al., 2023, Kumar et al., 28 May 2024).

1. Methodological Foundations for Adapting LLMs to Tibetan

Tibetan adaptation methodologies rest on rigorous two-stage or multi-stage paradigms to address both language grounding and downstream task alignment. The canonical pipeline comprises:

• Continual Pretraining (CPT): An initial phase recalibrating LLMs on Tibetan-only corpora to construct a semantic representation manifold for the target language. This stage updates embedding matrices and early-to-mid MLP projections based on domain token distributions (Chen et al., 3 Dec 2025, Pan et al., 12 Jul 2025).
• Supervised Fine-Tuning (SFT): Application of cross-entropy training on curated instruction-following, translation, and question-answering datasets—often including cross-lingual anchors—to specialize for targeted tasks without catastrophic forgetting (Chen et al., 3 Dec 2025, Huang et al., 24 Mar 2025).

Some models, such as Sun-Shine, extend this regime with Direct Preference Optimization (DPO), aligning model behavior directly with preference pairs using a contrastive loss (Huang et al., 24 Mar 2025).

Parameter-efficient approaches—such as prompt-tuning, adapter modules (LoRA), and their combinations—enable lightweight adaptation by updating either a small subset of learned vectors or inserting low-rank trainable matrices into transformer sub-blocks, allowing full backbone freezing (Mingjun et al., 2023, Huang et al., 24 Mar 2025).

2. Data Curation, Tokenization, and Corpus Construction

Performance and stability in Tibetan adaptation directly depend on meticulous corpus curation, tokenization scheme design, and preprocessing:

• Corpus Assembly: Leading efforts aggregate texts from open-source datasets, web crawls, synthetic parallel data (Tibetan/Chinese/English), and private documents. For example, Banzhida’s 72 GB corpus incorporates 8.5B Tibetan tokens and is balanced with Chinese and English to leverage cross-lingual transfer (Pan et al., 12 Jul 2025). Sun-Shine's TIB-STC achieves domain coverage through expert-vetted literature, web, and media corpora (Huang et al., 24 Mar 2025).
• Preprocessing: Includes sentence segmentation, de-duplication, language filtering (fastText), Gopher-based quality filtering, and MinHash deduplication. Orthographic normalization (Uchen vs. Umeh) and morphological challenges are addressed by vocabulary expansion and BPE/SentencePiece tokenizers tailored for the 2-D Tibetan syllable structure (Pan et al., 12 Jul 2025, Huang et al., 24 Mar 2025).
• Tokenization Expansion: To avoid excessive fragmentation and loss compression, token vocabularies are augmented with thousands of Tibetan-specific tokens (e.g., 15K for Banzhida; ~64K for Sun-Shine) (Pan et al., 12 Jul 2025, Huang et al., 24 Mar 2025).

3. Training Regimes and Quantitative Dynamics

Adaptation regimes for Tibetan involve exhaustive hyperparameter sweeps and quantitative monitoring of language grounding, task specialization, and convergence:

• Hyperparameters: CPT and SFT stages employ batch sizes of 128–256, BF16 mixed precision, AdamW optimizers, cosine learning-rate decay, sequence lengths up to 8,192 in pretraining and 4,096 for SFT (Chen et al., 3 Dec 2025, Pan et al., 12 Jul 2025, Huang et al., 24 Mar 2025).
• Performance Metrics: Core metrics include perplexity ($\mathrm{PPL}$) on Tibetan text, BLEU and chrF for Chinese→Tibetan and English→Tibetan translation, as well as accuracy, EM, and F1 on reasoning and QA benchmarks (Chen et al., 3 Dec 2025, Pan et al., 12 Jul 2025, Huang et al., 24 Mar 2025).
• Layerwise Adaptation: Parameter L2-deltas are measured across all layers (e.g., 430 transformer blocks + embeddings, output heads in Qwen2.5-3B), revealing adaptation is concentrated in embeddings, lm_head, and mid–late MLP projections (Chen et al., 3 Dec 2025).

The dynamics of adaptation consistently show:

• Perplexity improvements from 2.98 $\rightarrow$ 1.54 after CPT+SFT (Chen et al., 3 Dec 2025)
• BLEU score increases for CN→BO translation from 0.046 $\rightarrow$ 0.261; chrF from 2.2 $\rightarrow$ 6.6 (Chen et al., 3 Dec 2025)
• Strong cross-lingual generalization and multitask robustness after SFT/DPO (Huang et al., 24 Mar 2025).

4. Parameter Localization, Task Alignment, and Semantic Manifold Formation

Systematic analysis of where and how Tibetan adaptation manifests in LLMs yields several key findings:

• Localization: CPT induces global but highly localized shifts in the model: embeddings and lm_head weights are most highly perturbed, with domain-specific transformations encoded in late MLP layers (Chen et al., 3 Dec 2025). SFT induces smaller but colocalized updates, with near-perfect (Pearson $r \approx 1.0$) alignment to the CPT-induced parameter directions.
• Semantic Manifold Construction: CPT re-anchors token embeddings to Tibetan lexical semantics and re-shapes logits to prioritize Tibetan output, establishing a structurally distinct Tibetan “semantic manifold” (Chen et al., 3 Dec 2025).
• Task Specialization Consolidation: SFT and preference optimization sharpen and solidify this manifold, minimally disrupting linguistic representations while encoding instruction-following and translation competencies (Chen et al., 3 Dec 2025, Huang et al., 24 Mar 2025).

In parameter-efficient approaches, LoRA adapters and soft prompt vectors focus adaptation to parameter-efficient “subspaces,” yielding competitive downstream performance with minimal resource cost (sub-1% parameter update) (Mingjun et al., 2023, Huang et al., 24 Mar 2025).

5. Dynamic, Retrieval-Augmented, and Multilingual Adaptation Strategies

To further mitigate resource constraints and leverage cross-lingual capacity, dynamic and retrieval-augmented adaptation frameworks are utilized:

• Dynamic Configuration Selection: At inference, a learned neural selector chooses among LLM, prompt, and embedding model tuples based on task embeddings, maximizing query-level F1 over static or random selection (Kumar et al., 28 May 2024). Offline and online dynamic adaptation drive fast convergence on Tibetan by collecting ground-truth per-configuration F1 and updating a selector head over time.
• Retrieval-Augmented Generation (RAG): Tibetan queries are augmented with context retrieved from Tibetan Wikipedia, Common Crawl, or monastic corpora, using cosine similarity in an XLM-R or Cohere embedding space, before LLM generation (Kumar et al., 28 May 2024).
• Prompting Strategies: Approaches include monolingual Tibetan prompts, translation-pivoting, and aggregation/voting methods to compensate for limited instruction-following data (Kumar et al., 28 May 2024).
• Empirical Gains: Dynamic adaptation beats static best-single configurations by 10–20 percentage points of F1 across low-resource and typologically diverse languages (Kumar et al., 28 May 2024).

6. Empirical Results, Ablations, and Framework Reproducibility

Experimental evaluation on a spectrum of classification, QA, and translation benchmarks yields the following verified outcomes:

• Performance Summary: SFT and DPO deliver substantial gains—BLEU jumps (e.g., 0.046 → 0.261) and F1 improvements are repeatably observed (Chen et al., 3 Dec 2025, Pan et al., 12 Jul 2025, Huang et al., 24 Mar 2025).
• Benchmark Comparisons: Banzhida and Sun-Shine surpass prior Tibetan and multilingual baselines on HellaSwag-bo, ARC-bo, Xcopa-bo, TLUE (few-shot), and translation tasks by 5–20 points (Pan et al., 12 Jul 2025, Huang et al., 24 Mar 2025). LoRA adapters and prompt-tuning via PEFTT match or exceed conventional full-parameter fine-tuning in macro-F1 for news headline classification, with <0.2% extra parameters (Mingjun et al., 2023).
• Ablation Studies: Removing parameter-efficient adapters, classical-text segments, or bilingual mining results in 3–12% drops in accuracy or response rates (Huang et al., 24 Mar 2025).
• Framework Reproducibility: Public codebases and detailed recipes are made available for CPT/SFT, RAG, and evaluation (e.g., https://github.com/clf28/Tibetan-Finetuning/tree/main) (Chen et al., 3 Dec 2025). Extension to new low-resource languages follows the same procedural template: corpus curation, tokenizer expansion, two-stage adaptation, and dynamic evaluation.

7. Limitations, Open Problems, and Future Research Directions

Open issues in Tibetan LLM adaptation include:

• Data Scarcity and Domain Coverage: Current results remain constrained by the scarcity of high-quality, domain-balanced Tibetan corpora and reliable QA ground-truth annotations (Pan et al., 12 Jul 2025, Kumar et al., 28 May 2024).
• Morphological Representation: The agglutinative and irregular morphology of Tibetan is only partially addressed by current tokenization and adapter-driven methods. Morphological segmentation models and further token vocabulary expansions are identified as promising avenues (Pan et al., 12 Jul 2025).
• Scaling and Generalization: Larger-scale LLMs (e.g., Qwen3, LLaMA-3.1 8B), longer context adaptation (>32 K tokens), and better cross-lingual transfer capacity are active areas for scalable improvement (Pan et al., 12 Jul 2025, Huang et al., 24 Mar 2025).
• Parameter-Efficient Adaptation Extension: Expanding PEFT methods (prompt+adapter) to generative and complex reasoning tasks remains to be fully explored (Mingjun et al., 2023).
• Broader Multilingual Dynamics: The transferability of observed adaptation localizations (embeddings/lm_head/MLP) and consolidation dynamics to other under-resourced or non-Latin languages is an empirical and theoretical frontier (Chen et al., 3 Dec 2025, Kumar et al., 28 May 2024).

By delineating and quantifying parameter shift localization, semantic manifold formation, and dynamic adaptation strategies, Tibetan adaptation dynamics for LLMs lay the methodological and analytical foundation for extending LLMs to typologically complex, low-resource, and under-served languages (Chen et al., 3 Dec 2025, Pan et al., 12 Jul 2025, Huang et al., 24 Mar 2025, Mingjun et al., 2023, Kumar et al., 28 May 2024).