TransLLM: Chat Model Language Transfer

• TransLLM is a framework that converts an English chat LLM into a robust non-English version by employing translation chain-of-thought and recovery knowledge distillation.
• It uses a four-stage pipeline—including model extension with LoRA, target-language pre-training, translation pre-training, and transfer fine-tuning—to ensure effective language transfer.
• Experimental results demonstrate competitive performance in helpfulness, safety, and translation quality while mitigating catastrophic forgetting of original chat behaviors.

Searching arXiv for the exact TransLLM paper and closely related uses/contexts of the term. arxiv_search(query="TransLLM OR \"Why Not Transform Chat LLMs to Non-English?\" OR multilingual translation-tailored LLMs translation chain-of-thought low-resource language", max_results=10) TransLLM is a framework for transforming an English-centric chat LLM into a non-English chat model while attempting to preserve the source model’s advanced chat abilities, especially helpfulness, multi-turn dialogue skill, and safety alignment. Its canonical instantiation transforms LLaMA-2-chat-7B from English to Thai through a combination of translation chain-of-thought (TCOT), target-language and translation pre-training on public data, and a forgetting-mitigation scheme built from LoRA and recovery knowledge distillation (recovery KD) (Geng et al., 2024).

1. Concept and problem setting

TransLLM addresses a specific transfer problem: how to convert an English-centric chat LLM into a strong non-English chat model without reproducing the original expensive chat-model training pipeline in the target language. The framework is motivated by the observation that chat LLMs encode more than general language modeling knowledge. They already contain supervised chat behavior from SFT, multi-turn conversational structure, human-preference alignment, and safety refusal behavior. The paper therefore treats chat-model transfer as substantially harder than transferring a base LLM (Geng et al., 2024).

Two challenges define the method. The first is how to transfer advanced chat abilities without supervised chat data in the target language. The second is how to avoid catastrophic forgetting of the original knowledge during language transfer. These challenges are central because the proprietary multi-turn SFT and preference data used to build chat LLMs are usually unavailable in the target language, while naïve continued training risks overwriting the original English chat prior (Geng et al., 2024).

The framework adopts source-language notation $\alpha$ and target-language notation $\beta$, with $\alpha=$ English and $\beta=$ Thai in the reported experiments. This formulation is presented as a general recipe, but the demonstrated case is specifically English $\rightarrow$ Thai transformation of LLaMA-2-chat-7B (Geng et al., 2024).

2. Architecture and training pipeline

TransLLM is organized as a four-stage pipeline. The stages are model extension, target language pre-training, translation pre-training, and transfer fine-tuning. In the model-extension stage, the method adds LoRA modules to the chat LLM and extends the tokenizer with fine-grained target-language vocabulary learned using SentencePiece. The final vocabulary size is 43,012, including the three special control tokens <α>, <β>, and <RESPONSE>, and the new embeddings are randomly initialized (Geng et al., 2024).

The target-language pre-training stage continues pretraining on target-language monolingual data to improve fluency and localization. In the reported Thai setting, the monolingual source is mC4-TH, filtered with a sensitive word list and deduplicated with MinHashLSH, yielding about 11B Thai tokens. The translation pre-training stage uses bidirectional EN–TH parallel corpora from CCAligned, Tatoeba Challenge Data, and OpenSubtitles, while mixing in English monolingual data from The Pile to protect English embeddings (Geng et al., 2024).

The pretraining objective is the standard language-model objective

$$J_{\text{PT}} = \arg\max_{\theta}\sum_{i=1}^{|y|} \log P(y_i|y_{<i};\theta),$$

and the supervised fine-tuning objective is

$$J_{\text{SFT}} = \arg\max_{\theta} \sum_{i=1}^{|y|}\log P(y_i|x, y_{<i};\theta).$$

The paper states that, in SFT, the loss is calculated only on the label $y$, not on the query $x$ (Geng et al., 2024).

LoRA is used not only for efficiency but also as a preservation mechanism. For a weight matrix $W \in \mathbb{R}^{d\times k}$, the framework writes the update as

$\beta$0

with frozen original $\beta$1. LoRA is applied to q_proj, v_proj, k_proj, o_proj, gate_proj, down_proj, and up_proj; embed_tokens and lm_head are also trainable. The reported LoRA hyperparameters are rank $\beta$2, alpha $\beta$3, and dropout $\beta$4. Trainable parameters total 512.27M, and after merging LoRA into the backbone the final model size is 6.83B (Geng et al., 2024).

3. Translation chain-of-thought and recovery knowledge distillation

The central conceptual device in TransLLM is translation chain-of-thought. Rather than teaching the model to answer directly in Thai as a monolithic skill, the framework decomposes non-English chat into a sequence of sub-tasks: translate the target-language query into English, answer in English, and then generate the final target-language answer. For a Thai query $\beta$5, the TCOT training target is

$\beta$6

with input

$\beta$7

This makes translation the bridge between English and Thai “step-by-step,” while retaining all stages inside one generation (Geng et al., 2024).

The method supplements TCOT with translation SFT so that the model can still handle explicit translation requests. For a parallel pair $\beta$8, the framework wraps the source in a translation prompt and predicts the target with the appropriate control token. This component is introduced because TCOT alone can confuse pure translation with chat-style answering (Geng et al., 2024).

The second core mechanism is recovery KD, which is designed to recover the source chat model’s original English behavior rather than distilling from an external stronger teacher. The framework constructs English query-response pairs from Alpaca queries, but the responses are generated by the source chat LLM itself, namely LLaMA-2-chat-7B. For an English query $\beta$9 and answer $\alpha=$0,

$\alpha=$1

The paper’s argument is that this is synergistic with LoRA: because the original backbone weights are frozen, the model can learn a shortcut that reuses the preserved English knowledge already encoded in those frozen parameters (Geng et al., 2024).

The transfer fine-tuning stage does not use a hand-written composite loss with explicit mixture coefficients. Instead, the paper states that recovery KD, TCOT, and translation SFT data are randomly mixed, and the model is fine-tuned by the SFT objective. This makes the final model jointly support English answering, Thai answering via TCOT, and bidirectional translation (Geng et al., 2024).

4. Inference behavior and multilingual interpretation

In single-turn use, the final model automatically selects among three behaviors: English response, Thai response via TCOT, and bidirectional translation. In multi-turn Thai dialogue, the framework keeps prior turns in English and appends the current Thai query. The input is

$\alpha=$2

and the model tends to generate a TCOT-style continuation for the new turn: $\alpha=$3 Experiments use greedy decoding, and the paper stresses that the training data are only single-turn, even though the transferred behavior targets multi-turn interaction (Geng et al., 2024).

This design differs from a standard translate-test pipeline. In a classic external pipeline, the non-English input is translated into English before the LLM sees it, and the English output is translated back afterward. TransLLM instead generates the English query translation, the English response, and the target-language response in one sequence, allowing the final Thai answer to condition on both the Thai and English query forms as well as the English answer. The paper further reports attention-map analysis in which the Thai response attends most to Thai response context, then English response, then English query, and then Thai query, which the authors interpret as evidence that the final target-language answer is conditioned on more than a literal back-translation step (Geng et al., 2024).

Related research places this behavior in a broader translation-mediated picture of multilingual LLMs. MALT argues that, in small English-dominant LLMs handling Urdu, the model’s “internal latent response” is often formed in English and the final layers act as translators into the target language; removing a late-layer translation direction and delegating final rendering to an external MT model improved the extracted latent response quality (Bajwa, 27 Jan 2025). This suggests that TransLLM’s TCOT bridge is compatible with a broader view in which multilingual capability is often mediated by an English-centered response pathway rather than uniformly native non-English generation.

5. Empirical performance and diagnostic evidence

The main evaluation covers helpfulness, safety, and translation quality. On MT-bench human evaluation in Thai, TransLLM versus ChatGPT achieves win 53.75%, tie 27.50%, loss 18.75%, $\alpha=$4 on the first turn and win 48.75%, tie 26.25%, loss 25.00%, $\alpha=$5 on the second turn. Against GPT-4, it remains behind, with $\alpha=$6 on the first turn and $\alpha=$7 on the second. On AlpacaEval, TransLLM versus ChatGPT records win 72.50%, tie 13.75%, loss 13.75%, for $\alpha=$8, while remaining below GPT-4 with $\alpha=$9 (Geng et al., 2024).

On AdvBench, the reported safety numbers are especially strong. Rejection rates are 79.81% for ChatGPT, 85.96% for GPT-4, 63.46% for Ours w/ GPT-4 KD, and 94.61% for Ours. The corresponding bypass rates are 10.96%, 10.38%, 31.15%, and 2.69%. The paper emphasizes that this safety behavior is obtained without safety-specific training data, and that replacing recovery KD with GPT-4 KD harms safety substantially (Geng et al., 2024).

On Flores-200, the framework also functions as a translation system. For EN→TH, Ours reports COMET 86.96 / BLEU 35.04, compared with ChatGPT 85.47 / 31.26 and NLLB 83.88 / 28.53. For TH→EN, Ours reports 86.97 / 27.68, compared with ChatGPT 86.29 / 23.47 and NLLB 87.14 / 30.78 (Geng et al., 2024).

Several diagnostics are used to support the recovery claim. On held-out English recovery-KD validation data, average generation probabilities $\beta=$0 are 0.2363 for LLaMA2-Chat (EN), 0.1666 for Ours w/o transfer fine-tuning, 0.1972 for Ours w/ GPT-4 KD, and 0.2352 for Ours. The difference from the original is therefore 0.0697, 0.0391, and 0.0055, respectively. The paper also compares hidden states from the frozen-output path $\beta=$1 and LoRA-updated output $\beta=$2 on TCOT validation data, reporting cosine similarity 0.6191 for English responses and 0.2522 for Thai, which the authors interpret as evidence that the model relies more heavily on the original path for English than for Thai (Geng et al., 2024).

Ablations show that all major components matter. Relative to the full system on MT-bench, the paper reports clear degradations for w/o chat model, w/o TH pre-train, w/o translation pre-train, w/ GPT-4 KD, and w/o LoRA. The LoRA ablation is especially severe, reinforcing the paper’s claim that freezing the original backbone is central to preventing catastrophic forgetting (Geng et al., 2024).

Benchmark	TransLLM result	Comparison
MT-bench human vs ChatGPT, first turn	$\beta=$3	Win 53.75%, tie 27.50%, loss 18.75%
MT-bench human vs ChatGPT, second turn	$\beta=$4	Win 48.75%, tie 26.25%, loss 25.00%
AdvBench rejection rate	94.61%	ChatGPT 79.81%, GPT-4 85.96%
AdvBench bypass rate	2.69%	ChatGPT 10.96%, GPT-4 10.38%
Flores-200 EN→TH	COMET 86.96 / BLEU 35.04	ChatGPT 85.47 / 31.26
Flores-200 TH→EN	COMET 86.97 / BLEU 27.68	ChatGPT 86.29 / 23.47

6. Limitations, broader significance, and later extensions

The framework’s limitations are explicit. It is tested only on Thai and only on LLaMA-2-chat-7B, so generality to other languages and chat backbones is plausible but unverified. It remains heavily dependent on translation, which the paper links to weakness on target-language-specific phenomena such as Thai wordplay or text-centric tasks. Inference overhead is also high because TCOT explicitly generates intermediate English steps. The authors further suggest that an adaptive mode selection mechanism—deciding when to answer directly in Thai and when to invoke TCOT—would be a natural extension. Finally, math and coding remain weak, tracking limitations already present in the English source model (Geng et al., 2024).

Within the broader translation-tailored LLM literature, TransLLM can be read as one point in a larger design space. Language Aware Instruction Tuning treats translation-specialized LLMs as failing primarily at language-direction adherence, and reports an average -53.3% OTR, +5.7 SacreBLEU, and +16.4 BLEURT over a translation-finetuned LLaMA baseline by adding instruction-conflicting samples and an unlikelihood loss (Zan et al., 2024). A plausible implication is that TransLLM’s TCOT formulation and recovery KD address a different but adjacent failure mode: preserving aligned chat behavior across languages rather than only improving direct translation direction following.

Later multilingual MT work pushes translation-specialized LLMs further. LMT introduces a family centered on both Chinese and English, covering 60 languages and 234 translation directions, and identifies directional degeneration under symmetric multilingual SFT. Its remedy, Strategic Downsampling, keeps En/Zh→X at 100% and retains only 5% of X→En/Zh samples during SFT (Luo et al., 10 Nov 2025). This suggests a broader research trajectory in which TransLLM-style systems evolve from single target-language transformation toward explicitly multilingual, direction-balanced translation LLMs.

The label itself is not unique. A later and unrelated paper also uses TransLLM to denote a unified multi-task foundation framework for urban transportation via learnable prompting, combining a spatiotemporal encoder, LLM reasoning, and an instance-level prompt router (Leng et al., 20 Aug 2025). In arXiv usage, therefore, “TransLLM” designates both a specific chat-model language-transfer framework and, later, a separate transportation foundation model.