--- title: 'TransLLM: Chat Model Language Transfer' url: https://www.emergentmind.com/topics/transllm type: topic --- # TransLLM: Chat Model Language Transfer Searching arXiv for the exact TransLLM paper and closely related uses/contexts of the term. arxiv_search(query="TransLLM OR \"Why Not Transform Chat Large Language Models to Non-English?\" OR multilingual translation-tailored LLMs translation chain-of-thought low-resource language", max_results=10) TransLLM is a framework for transforming an English-centric chat large language model into a non-English chat model while attempting to preserve the source model’s advanced chat abilities, especially helpfulness, multi-turn dialogue skill, and safety alignment. Its canonical instantiation transforms **LLaMA-2-chat-7B** from **English** to **Thai** through a combination of **translation chain-of-thought (TCOT)**, target-language and translation pre-training on public data, and a forgetting-mitigation scheme built from **LoRA** and **recovery knowledge distillation (recovery KD)** [2405.13923]. ## 1. Concept and problem setting TransLLM addresses a specific transfer problem: how to convert an English-centric **chat** LLM into a strong non-English chat model **without** reproducing the original expensive chat-model training pipeline in the target language. The framework is motivated by the observation that chat LLMs encode more than general language modeling knowledge. They already contain supervised chat behavior from SFT, multi-turn conversational structure, human-preference alignment, and safety refusal behavior. The paper therefore treats chat-model transfer as substantially harder than transferring a base LLM [2405.13923]. Two challenges define the method. The first is **how to transfer advanced chat abilities without supervised chat data** in the target language. The second is **how to avoid catastrophic forgetting of the original knowledge** during language transfer. These challenges are central because the proprietary multi-turn SFT and preference data used to build chat LLMs are usually unavailable in the target language, while naïve continued training risks overwriting the original English chat prior [2405.13923]. The framework adopts source-language notation \(\alpha\) and target-language notation \(\beta\), with \(\alpha=\) English and \(\beta=\) Thai in the reported experiments. This formulation is presented as a general recipe, but the demonstrated case is specifically **English \(\rightarrow\) Thai** transformation of **LLaMA-2-chat-7B** [2405.13923]. ## 2. Architecture and training pipeline TransLLM is organized as a four-stage pipeline. The stages are **model extension**, **target language pre-training**, **translation pre-training**, and **transfer fine-tuning**. In the model-extension stage, the method adds **LoRA** modules to the chat LLM and extends the tokenizer with **fine-grained target-language vocabulary** learned using **SentencePiece**. The final vocabulary size is **43,012**, including the three special control tokens **`<α>`**, **`<β>`**, and **``**, and the new embeddings are randomly initialized [2405.13923]. The target-language pre-training stage continues pretraining on target-language monolingual data to improve fluency and localization. In the reported Thai setting, the monolingual source is **mC4-TH**, filtered with a sensitive word list and deduplicated with **MinHashLSH**, yielding about **11B Thai tokens**. The translation pre-training stage uses bidirectional **EN–TH** parallel corpora from **CCAligned**, **Tatoeba Challenge Data**, and **OpenSubtitles**, while mixing in English monolingual data from **The Pile** to protect English embeddings [2405.13923]. The pretraining objective is the standard language-model objective \[ J_{\text{PT}} = \arg\max_{\theta}\sum_{i=1}^{|y|} \log P(y_i|y_{}, q^{\alpha},\text{}, a^{\alpha},\text{<}\beta\text{>}, a^{\beta}), \] with input \[ x=q^{\beta}. \] This makes translation the bridge between English and Thai “step-by-step,” while retaining all stages inside one generation [2405.13923]. The method supplements TCOT with **translation SFT** so that the model can still handle explicit translation requests. For a parallel pair \((q^\alpha, q^\beta)\), the framework wraps the source in a translation prompt and predicts the target with the appropriate control token. This component is introduced because TCOT alone can confuse pure translation with chat-style answering [2405.13923]. The second core mechanism is **recovery KD**, which is designed to recover the source chat model’s original English behavior rather than distilling from an external stronger teacher. The framework constructs English query-response pairs from **Alpaca** queries, but the responses are generated by the **source chat LLM itself**, namely **LLaMA-2-chat-7B**. For an English query \(q^\alpha\) and answer \(a^\alpha\), \[ x=q^{\alpha}, \qquad y=\text{cat}(\text{}, a^{\alpha}). \] The paper’s argument is that this is synergistic with LoRA: because the original backbone weights are frozen, the model can learn a shortcut that reuses the preserved English knowledge already encoded in those frozen parameters [2405.13923]. The transfer fine-tuning stage does not use a hand-written composite loss with explicit mixture coefficients. Instead, the paper states that **recovery KD**, **TCOT**, and **translation SFT** data are **randomly mixed**, and the model is fine-tuned by the SFT objective. This makes the final model jointly support English answering, Thai answering via TCOT, and bidirectional translation [2405.13923]. ## 4. Inference behavior and multilingual interpretation In single-turn use, the final model automatically selects among three behaviors: **English response**, **Thai response via TCOT**, and **bidirectional translation**. In multi-turn Thai dialogue, the framework keeps prior turns in **English** and appends the current Thai query. The input is \[ x=\text{cat}(q^{\alpha}_1,a^{\alpha}_1,\dots,q^{\alpha}_n,a^{\alpha}_n,q^{\beta}_{n+1}), \] and the model tends to generate a TCOT-style continuation for the new turn: \[ y=\text{cat}(\text{<}\alpha\text{>}, q^{\alpha}_{n+1},\text{}, a^{\alpha}_{n+1},\text{<}\beta\text{>}, a^{\beta}_{n+1}). \] Experiments use **greedy decoding**, and the paper stresses that the training data are only **single-turn**, even though the transferred behavior targets **multi-turn** interaction [2405.13923]. This design differs from a standard translate-test pipeline. In a classic external pipeline, the non-English input is translated into English before the LLM sees it, and the English output is translated back afterward. TransLLM instead generates the English query translation, the English response, and the target-language response in one sequence, allowing the final Thai answer to condition on both the Thai and English query forms as well as the English answer. The paper further reports attention-map analysis in which the Thai response attends most to **Thai response context**, then **English response**, then **English query**, and then **Thai query**, which the authors interpret as evidence that the final target-language answer is conditioned on more than a literal back-translation step [2405.13923]. Related research places this behavior in a broader translation-mediated picture of multilingual LLMs. **MALT** argues that, in small English-dominant LLMs handling Urdu, the model’s “internal latent response” is often formed in English and the final layers act as translators into the target language; removing a late-layer translation direction and delegating final rendering to an external MT model improved the extracted latent response quality [2502.00041]. This suggests that TransLLM’s TCOT bridge is compatible with a broader view in which multilingual capability is often mediated by an English-centered response pathway rather than uniformly native non-English generation. ## 5. Empirical performance and diagnostic evidence The main evaluation covers **helpfulness**, **safety**, and **translation quality**. On **MT-bench** human evaluation in Thai, TransLLM versus **ChatGPT** achieves **win 53.75%, tie 27.50%, loss 18.75%, \(\Delta = 35.00\)** on the first turn and **win 48.75%, tie 26.25%, loss 25.00%, \(\Delta = 23.75\)** on the second turn. Against **GPT-4**, it remains behind, with **\(\Delta=-15.00\)** on the first turn and **\(\Delta=-27.50\)** on the second. On **AlpacaEval**, TransLLM versus **ChatGPT** records **win 72.50%**, **tie 13.75%**, **loss 13.75%**, for **\(\Delta = 58.75\)**, while remaining below **GPT-4** with **\(\Delta = -20.00\)** [2405.13923]. On **AdvBench**, the reported safety numbers are especially strong. Rejection rates are **79.81%** for **ChatGPT**, **85.96%** for **GPT-4**, **63.46%** for **Ours w/ GPT-4 KD**, and **94.61%** for **Ours**. The corresponding bypass rates are **10.96%**, **10.38%**, **31.15%**, and **2.69%**. The paper emphasizes that this safety behavior is obtained **without safety-specific training data**, and that replacing recovery KD with **GPT-4 KD** harms safety substantially [2405.13923]. On **Flores-200**, the framework also functions as a translation system. For **EN→TH**, **Ours** reports **COMET 86.96 / BLEU 35.04**, compared with **ChatGPT 85.47 / 31.26** and **NLLB 83.88 / 28.53**. For **TH→EN**, **Ours** reports **86.97 / 27.68**, compared with **ChatGPT 86.29 / 23.47** and **NLLB 87.14 / 30.78** [2405.13923]. Several diagnostics are used to support the recovery claim. On held-out English recovery-KD validation data, average generation probabilities \(P(y|x)\) are **0.2363** for **LLaMA2-Chat (EN)**, **0.1666** for **Ours w/o transfer fine-tuning**, **0.1972** for **Ours w/ GPT-4 KD**, and **0.2352** for **Ours**. The difference from the original is therefore **0.0697**, **0.0391**, and **0.0055**, respectively. The paper also compares hidden states from the frozen-output path \(\Tilde{h}\) and LoRA-updated output \(\hat{h}\) on TCOT validation data, reporting cosine similarity **0.6191** for English responses and **0.2522** for Thai, which the authors interpret as evidence that the model relies more heavily on the original path for English than for Thai [2405.13923]. Ablations show that all major components matter. Relative to the full system on **MT-bench**, the paper reports clear degradations for **w/o chat model**, **w/o TH pre-train**, **w/o translation pre-train**, **w/ GPT-4 KD**, and **w/o LoRA**. The LoRA ablation is especially severe, reinforcing the paper’s claim that freezing the original backbone is central to preventing catastrophic forgetting [2405.13923]. | Benchmark | TransLLM result | Comparison | |---|---:|---| | MT-bench human vs ChatGPT, first turn | \(\Delta = 35.00\) | Win 53.75%, tie 27.50%, loss 18.75% | | MT-bench human vs ChatGPT, second turn | \(\Delta = 23.75\) | Win 48.75%, tie 26.25%, loss 25.00% | | AdvBench rejection rate | 94.61% | ChatGPT 79.81%, GPT-4 85.96% | | AdvBench bypass rate | 2.69% | ChatGPT 10.96%, GPT-4 10.38% | | Flores-200 EN→TH | COMET 86.96 / BLEU 35.04 | ChatGPT 85.47 / 31.26 | | Flores-200 TH→EN | COMET 86.97 / BLEU 27.68 | ChatGPT 86.29 / 23.47 | ## 6. Limitations, broader significance, and later extensions The framework’s limitations are explicit. It is tested only on **Thai** and only on **LLaMA-2-chat-7B**, so generality to other languages and chat backbones is plausible but unverified. It remains **heavily dependent on translation**, which the paper links to weakness on target-language-specific phenomena such as Thai wordplay or text-centric tasks. Inference overhead is also high because TCOT explicitly generates intermediate English steps. The authors further suggest that an **adaptive mode selection** mechanism—deciding when to answer directly in Thai and when to invoke TCOT—would be a natural extension. Finally, math and coding remain weak, tracking limitations already present in the English source model [2405.13923]. Within the broader translation-tailored LLM literature, TransLLM can be read as one point in a larger design space. **Language Aware Instruction Tuning** treats translation-specialized LLMs as failing primarily at **language-direction adherence**, and reports an average **-53.3% OTR**, **+5.7 SacreBLEU**, and **+16.4 BLEURT** over a translation-finetuned LLaMA baseline by adding instruction-conflicting samples and an unlikelihood loss [2403.14399]. A plausible implication is that TransLLM’s TCOT formulation and recovery KD address a different but adjacent failure mode: preserving aligned chat behavior across languages rather than only improving direct translation direction following. Later multilingual MT work pushes translation-specialized LLMs further. **LMT** introduces a family centered on both **Chinese** and **English**, covering **60 languages** and **234 translation directions**, and identifies **directional degeneration** under symmetric multilingual SFT. Its remedy, **Strategic Downsampling**, keeps **En/Zh→X** at **100%** and retains only **5%** of **X→En/Zh** samples during SFT [2511.07003]. This suggests a broader research trajectory in which TransLLM-style systems evolve from single target-language transformation toward explicitly multilingual, direction-balanced translation LLMs. The label itself is not unique. A later and unrelated paper also uses **TransLLM** to denote a **unified multi-task foundation framework for urban transportation via learnable prompting**, combining a spatiotemporal encoder, LLM reasoning, and an instance-level prompt router [2508.14782]. In arXiv usage, therefore, “TransLLM” designates both a specific chat-model language-transfer framework and, later, a separate transportation foundation model. Source: https://www.emergentmind.com/topics/transllm