SALAMANDRATA: Multilingual Translation LLM

• SALAMANDRATA is a family of multilingual translation-specialized LLMs that uses continual pre-training on parallel data and supervised fine-tuning to enhance performance.
• It leverages multiple bridge languages and vocabulary adaptation to support both European and non-European languages, reducing English-centric bias.
• Quality-aware decoding methods like Minimum Bayes Risk and Tuned Re-ranking significantly boost its translation quality across diverse evaluation tasks.

SALAMANDRATA is a family of translation-specialized LLMs developed by the Barcelona Supercomputing Center as an improved iteration of the SALAMANDRA LLMs, specifically trained to achieve strong performance in translation-related tasks for 38 European languages and later extended for additional WMT25 directions. The family comprises 2B and 7B parameter variants and is characterized by a two-stage adaptation recipe: continual pre-training on parallel data, followed by supervised fine-tuning on high-quality instructions. The BSC submission to the WMT25 General Machine Translation shared task is based on the 7B variant, with additional vocabulary adaptation, a second phase of continual pre-training and supervised fine-tuning, and quality-aware decoding via Minimum Bayes Risk Decoding and Tuned Re-ranking using COMET and COMET-KIWI respectively (Gilabert et al., 18 Aug 2025).

1. Model lineage and architectural basis

SALAMANDRATA extends the original SALAMANDRA LLMs rather than replacing their architectural substrate. Both SALAMANDRATA-2B and SALAMANDRATA-7B are based on the pre-existing SALAMANDRA LLM architecture, which is a causal decoder-only transformer, trained from scratch on broad multilingual monolingual data. The principal distinction from previous SALAMANDRA models is not a change in core architecture but the incorporation of parallel corpora at later training stages.

This design choice is central to the family’s specialization. SALAMANDRA models were not pre-trained on parallel data, whereas SALAMANDRATA introduces continual pre-training on parallel data and thereby targets translation performance directly. The paper also identifies a departure from standard English-centric practice: instead of relying on a single pivot language, the continual pre-training recipe uses three bridge languages—English, Spanish, and Catalan. This suggests an explicit attempt to broaden cross-lingual transfer and reduce English-centric bias, particularly for zero-shot and non-English-pair translation (Gilabert et al., 18 Aug 2025).

2. Continual pre-training on parallel data

The training pipeline is staged. SALAMANDRATA starts from checkpoints of SALAMANDRA 2B and 7B in the pre-annealing phase to maintain wide generalization, and then applies continual pre-training on parallel data. In the first continual pre-training stage, CPT-V1, the corpus covers 38 languages, 172 supervised directions, and three bridge languages—English, Spanish, and Catalan—at a scale of 6.57B sentence pairs and 424B tokens. Data filtering uses LABSE for alignment, Lingua2 for language ID, and Bifixer for cleanup (Gilabert et al., 18 Aug 2025).

The CPT prompt format is:

{source_lang}: {source} {target_lang}: {target}

The second continual pre-training stage, CPT-V2, is introduced for the newer SALAMANDRATA-V2 model and focuses on new WMT25 directions, especially non-European and new low-resource pairs. It covers 14 languages, 15 directions, and approximately 0.39B sentences and 27B tokens. The paper notes that EN->HI was also added to facilitate Bhojpuri transfer, and that subsampling was used to avoid catastrophic forgetting for previously seen directions.

The optimization setup is specified as Distributed Fused Adam with a learning rate linearly warmed up to $3\times10^{-5}$ and then cosine-decayed to $3\times10^{-6}$. Training uses bfloat16 precision, sequence length 8192, gradient clipping with norm 1.0, and NVIDIA NeMo. The 2B model is trained for 50k steps on CPT-V1 and the 7B model for 105k steps, with a similar setup reported for CPT-V2. The methodological significance of this stage lies in the fact that parallel-data exposure is the defining improvement over SALAMANDRA, and the paper treats it as the main mechanism for translation specialization (Gilabert et al., 18 Aug 2025).

3. Supervised instruction fine-tuning and task composition

After continual pre-training, SALAMANDRATA undergoes supervised instruction fine-tuning in two aligned stages. IT-V1 is used for the 2B and 7B base models and consists of approximately 135k instructions, mostly translation-related, spanning sentence, paragraph, and document settings and built from sources such as TOWERBLOCKS, FLORES-200, EUROPARL, and NTREX. Task balancing in IT-V1 is pivoted through five bridge languages—English, Spanish, Catalan, Basque, and Galician—but random sampling is used to avoid target-side collapse due to duplicate alignments (Gilabert et al., 18 Aug 2025).

The instruction prompt format is ChatML-style:

Translate the following text from {source_lang} to {target_lang}: {source_lang}: {source} {target_lang}: {target}

IT-V2 contains approximately 51k instructions and focuses on paragraph/context-aware translation, especially for the new WMT25 pairs. The fine-tuning framework is FastChat, with hyperparameters reported as a learning rate of $1\times10^{-5}$, warmup ratio 0.03, and maximum sequence length 8192.

The ablation results are especially consequential for translation-centric model design. They confirm that adding pre/post-MT tasks does not significantly boost translation, and that adding chat/code tasks is detrimental. A common misconception in instruction-tuning practice is that broader task diversity necessarily improves downstream MT quality; the reported ablations indicate the opposite for this model family, with MT-only instruction tuning yielding the best translation quality. The paper also reports that instruction tuning confers a major boost in translation quality, on the order of approximately +3.5 to 4.4 COMET points on average (Gilabert et al., 18 Aug 2025).

4. Vocabulary adaptation and tokenizer extension

To support the non-European languages added in WMT25—Chinese, Korean, Japanese, Arabic, and Bhojpuri—the tokenizer was adapted rather than leaving the original vocabulary unchanged. The approach involves training a new tokenizer on a corpus combining existing and newly added languages’ monolingual data. The embedding initialization strategy is mixed: common tokens copy embeddings from the old model, while new tokens are initialized to the average of existing embeddings to avoid outliers. Corresponding embedding and unembedding layer modifications allow the combination of reused and new tokens (Gilabert et al., 18 Aug 2025).

The paper reports that the new and old tokenizers share approximately 58% vocabulary. Recovery of new token embeddings is then carried out in the subsequent continual pre-training stage. This makes vocabulary adaptation an integrated component of the training pipeline rather than a detached preprocessing step.

The effect of tokenizer adaptation is quantified through fertility, defined as

$$\text{Fertility}(T, S) = \frac{\#\text{tokens in } T(S)}{\#\text{words in } S}$$

where $T$ is the tokenizer and $S$ is a corpus.

Under this measure, the adapted tokenizer in SALAMANDRATA-v2 achieves lower fertility than MADLAD-400 and NLLB; the paper gives an average fertility of 1.88 for SALAMANDRATA, compared to 2.00 for NLLB. Within the paper’s framing, lower fertility indicates more efficient tokenization, especially for the added non-European languages. A plausible implication is that the tokenizer extension was not merely a compatibility measure for new scripts, but a component of the system’s translation efficiency profile (Gilabert et al., 18 Aug 2025).

5. Decoding and quality-aware inference

SALAMANDRATA’s inference stack includes both conventional search and quality-aware reranking. Standard beam search is used with beam size 5. Diverse beam search uses beam size 20 with 5 beam groups, following the cited diverse beam search method. These candidate-generation procedures are then augmented by two quality-aware strategies: Minimum Bayes Risk decoding and Tuned Re-ranking (Gilabert et al., 18 Aug 2025).

Minimum Bayes Risk decoding is implemented with the mbrs library and selects the output with minimum expected loss with respect to a quality metric; COMET-22 is used for this purpose. Tuned Re-ranking generates $n$-best candidates from beam or diverse beam search, scores them with a reference-free quality estimator, and selects the best candidate, with COMET-KIWI used as the scoring metric.

The empirical outcome reported in the paper is consistent across evaluation settings: both MBR and TRR show significant quality improvements over simple beam search, and MBR gives the best results in most cases. In operational terms, this means that model quality is not treated as a function of parametric training alone; decoding is explicitly optimized against external MT quality estimators. This suggests a system design in which generation-time decision rules are a substantial contributor to final translation performance (Gilabert et al., 18 Aug 2025).

6. Empirical performance, WMT25 submission, and release

The BSC system submitted to the WMT25 General Machine Translation shared task is based on the 7B variant of SALAMANDRATA. For that submission, the model vocabulary was first adapted to support the additional non-European languages included in the task, after which a second phase of continual pre-training and supervised fine-tuning was carried out to optimize performance across all translation directions for that year’s shared task. The reported evaluation outcomes include several distinct findings (Gilabert et al., 18 Aug 2025).

First, on the WMT24++ test set, MBR decoding achieves the best scores in nearly every direction and metric, and SALAMANDRATA-7B together with SALAMANDRATA-v2 strongly outperform TOWER-V2 7B, NLLB 3.3B, and MADLAD400 7B baselines. Second, vocabulary adaptation and continual pre-training for low-resource languages materially affect performance: the inclusion of supporting languages such as Hindi to improve Bhojpuri raises EN→Bhojpuri performance from near-zero to more than 11 BLEU. Third, instruction-tuned SALAMANDRATA-7B exhibits increased resilience to character-level noise, with 17–21 BLEU percentage-point improvement at high noise levels over the base model. Fourth, MT-only instruction tuning is again confirmed as optimal, while chat/code instructions degrade BLEU scores.

The paper also summarizes the released checkpoints. SALAMANDRATA-2B corresponds to CPT-V1 plus IT-V1, SALAMANDRATA-7B likewise corresponds to CPT-V1 plus IT-V1, and SALAMANDRATA-7B-v2 combines CPT-V1+V2, IT-V2, and vocabulary adaptation. All major checkpoints and versions—SALAMANDRATA-2B, SALAMANDRATA-7B, and SALAMANDRATA-v2—are publicly released on Hugging Face through the cited repository. In the paper’s own synthesis, the family is presented as an open, massively multilingual, translation-specialized LLM suite whose strongest configuration combines multi-bridge continual pre-training, focused instruction tuning, tokenizer adaptation for new languages, and quality-aware decoding (Gilabert et al., 18 Aug 2025).