OMT-LLaMA: Decoder-Only MT for 1,600+ Languages

• The paper presents OMT-LLaMA as a decoder-only adaptation that uses vocabulary expansion and multilingual pre-training to boost translation fidelity for over 1,600 languages.
• It demonstrates that smaller models (1B-8B parameters) can match or exceed a 70B LLM, dramatically reducing compute while supporting long-tail language generation.
• A comprehensive training strategy combining monolingual, parallel, and synthetic data with reinforcement learning enables robust performance and scalable on-device deployment.

OMT-LLaMA is a family of decoder-only machine translation models within Omnilingual Machine Translation (OMT), a system designed to support more than 1,600 languages. It specializes Meta’s LLaMA3 models for extreme-scale multilingual MT through vocabulary expansion, multilingual continued pre-training on monolingual and parallel data, supervised fine-tuning, and reinforcement learning. Within the OMT framework, OMT-LLaMA is one of two specialization paths explored for LLMs, the other being OMT-NLLB as a module in an encoder-decoder architecture. The central empirical claim is that specialized OMT-LLaMA models from approximately 1 B to 8 B parameters match or exceed the MT performance of a 70 B LLM baseline, while substantially extending coherent generation into the long tail of undersupported languages (Team et al., 17 Mar 2026).

1. Position within Omnilingual Machine Translation

OMT-LLaMA is introduced against the background that current multilingual MT systems, despite high quality at moderate scale, cover only a small fraction of the world’s roughly 7,000 languages. The paper characterizes the state of practice as offering about 200 target-side languages and perhaps a few hundreds more on the source side through cross-lingual transfer. It presents OMT as the first MT system supporting more than 1,600 languages, and situates OMT-LLaMA as the decoder-only branch of that system (Team et al., 17 Mar 2026).

The model family addresses two related limitations. The first is coverage: long-tail languages are poorly represented in standard multilingual benchmarks and training corpora. The second is evaluability: the paper argues that reliable assessment has been difficult because of the lack of robust benchmarks and metrics at this scale. OMT-LLaMA is therefore not only a model adaptation effort but part of a broader attempt to operationalize “omnilingual” MT through training data construction, benchmark design, and metric selection.

A key framing in the paper is the distinction between cross-lingual “understanding” and target-language generation. The reported results indicate that baseline models can often interpret undersupported languages, but frequently fail to generate them with meaningful fidelity. OMT-LLaMA is presented as a specialization strategy aimed specifically at closing that generation gap, rather than merely improving latent multilingual representation.

2. Architectural specialization of LLaMA

OMT-LLaMA starts from Meta’s LLaMA3 decoder-only Transformer, specifically the “8B-Instruct,” “3.2B,” and “1.2B” variants. The number of layers, attention heads, hidden dimensions, and feed-forward dimensions is retained from the corresponding base models. The architectural intervention is deliberately narrow: vocabulary and the layers tied to vocabulary are expanded, while all other parameters are inherited (Team et al., 17 Mar 2026).

The principal changes are the following:

• Vocabulary expansion: the tokenizer vocabulary is enlarged from 128K to 256K tokens.
• Improved pre-tokenizer: the pre-tokenizer is modified to better handle non-Latin scripts and diacritics.
• Embedding/output adaptation: the token-embedding and output-projection layers are enlarged to match the 256K vocabulary.
• Scale transfer: for smaller models, LLaMA3.2-1.2B and LLaMA3.2-3.2B are used as bases, followed by the same vocabulary extension and training recipe.

Model	Base variant	Approximate scale
OMT-LLaMA-1.2B	LLaMA3.2-1.2B	≈1 B
OMT-LLaMA-3.2B	LLaMA3.2-3.2B	≈3 B
OMT-LLaMA-8B	LLaMA3 “8B-Instruct”	≈8 B

The paper reports that experimental downscaling to 1.2B and 3.2B shows that even at 1 B parameters the model retains strong cross-lingual “understanding” ability, while the 8 B variant provides the best overall quality. This suggests that, in the OMT setting, architectural specialization and tokenization coverage contribute substantially to multilingual MT quality even when parameter count is modest.

3. Data composition, balancing, and training objectives

OMT-LLaMA is trained through continued pre-training (CPT) on a mixture of monolingual and parallel data, followed by supervised fine-tuning and reinforcement learning. The monolingual component consists of CC-2000-Web, with approximately 20 M sentences over 2,000+ languages, and CC-2000-Pdf, with approximately 5 M sentences over about 1,700 languages. The parallel component includes Bible, Panlex dictionary entries, Tatoeba, CC-NLLB-200, OMT-Primary, OMT-Langwise, and LTPP; synthetic bitext is added through back-translation and bitext mining (Team et al., 17 Mar 2026).

The parallel and synthetic resources are described as follows:

• Bible: ~600 M aligned sentences, ~1,600 languages
• Panlex dictionary entries: 2 B word-level pairs, ~1,000 languages
• Tatoeba: ~25 M sentence pairs, ~500 languages
• CC-NLLB-200: ~500 M sentences, 200 languages
• OMT-Primary: ~55 M sentences spanning ~500 languages, including SMOL, Gatitos, KreyolMT, AmericasNLP, AfroLingu-MT
• OMT-Langwise: ~8 M sentences over 100 languages
• LTPP: 1.4 M sentence pairs across 18 community-sourced projects
• Back-translated data: ~230 M sentences, stratified by resource level and filtered by OmniSONAR similarity
• Bitext mining: ~100 K high-confidence pairs across ~60 languages using global similarity in OmniSONAR space

Data balancing is a central part of the recipe. The paper states that monolingual and parallel streams are sampled in proportion to predefined budgets, with each language’s monolingual data capped at 50 K documents. It uses “Unimax sampling” over languages, which squashes the top-100 high-resource languages to a uniform share, upsamples the tail up to $\times 100$, and applies manual boosts for underrepresented scripts. For synthetic back-translation, the data are stratified by high, mid, low, very low, and zero-resource buckets, then filtered by latent-space similarity thresholds calibrated on a Bible development set. Ablations show that mixing back-translation at 10–20% of tokens is optimal for low-resource gains without harming high-resource quality.

Both monolingual and parallel streams use causal autoregressive cross-entropy. For a parallel pair with source $s$ and target $t$, a prompt tag is prepended and training minimizes

$$\mathcal{L}_{\mathrm{CE}} = -\sum_{i=1}^{|t|} \log P\bigl(t_i \mid t_{<i},\,\text{src}=s \bigr).$$

After CPT and supervised fine-tuning, reinforcement learning is applied with reward

$$R = 0.5\;\bigl(\mathrm{ChrF++}(output,ref)\bigr) + 0.5\;\bigl(\mathrm{MetricX}(output,ref)\bigr).$$

This combination of monolingual language modeling, direct translation supervision, and synthetic bitext implies that OMT-LLaMA is optimized simultaneously for broad multilingual lexical coverage and for translation-conditioned target generation.

4. Optimization procedure and decoding strategy

The optimization pipeline is staged. CPT uses AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.95$, weight decay $= 0.1$, and base learning rate $= 5 \times 10^{-5}$, on 8,192-token sequences for 50 K steps on 256 A100 GPUs. A vocabulary-adaptation warm-up precedes this stage: 10 K steps at learning rate $= 2 \times 10^{-4}$, training only the expanded embedding and output-projection layers on 80 GPUs. This warm-up directly targets the new 256K vocabulary introduced by the architectural modification (Team et al., 17 Mar 2026).

Supervised fine-tuning uses AdamW with learning rate $s$0, cosine decay with 1 K warmup, a 0.2 final scale, and 10 K steps, validating every 100 steps. The training stack uses FSDP with FP32 reduction and layer-wise checkpointing. Reinforcement learning is then conducted on the SFT checkpoint via Decoupled Clip + Dynamic Sampling Policy Optimization (DAPO), with group size $s$1, asymmetric clipping to preserve exploration, re-added KL penalty, and no reward shaping.

Decoding for standard translation uses greedy or small-beam search, with beam size 1–5, no sampling, and default length penalties matched to prompt lengths. For Retrieval-Augmented Generation, the pipeline is more elaborate:

1. The source is split into sentences or $s$2-grams.
2. Chunks are embedded via OmniSONAR.
3. Up to 64 examples per chunk are retrieved via FAISS quantized KNN with $s$3 plus BM25.
4. Candidates are filtered by quality signals.
5. They are reranked by a mixture of cosine similarity, BM25, and coverage.
6. The top 80 examples are selected and prepended as few-shot in-context examples.

For directions without parallel examples, examples are mined from all targets via on-the-fly similarity with $s$4. In practical terms, this decoding design means that OMT-LLaMA supports both direct translation and example-augmented inference, with the latter primarily targeted at difficult or undersupported directions.

5. Evaluation framework and empirical performance

The quantitative evaluation uses ChrF++, xCOMET, MetricX adjusted by GlotLID confidence, and BLEU. The benchmarks are Bible with 1,561 languages, BOUQuET with 275 languages across 8 domains, and FLoRes+ with 213 languages across 3 domains. The paper also introduces a leaderboard and identifies BOUQuET and Met-BOUQuET as the main human-created evaluation datasets, both described as dynamically evolving toward omnilinguality and freely available (Team et al., 17 Mar 2026).

Across these benchmarks, OMT-LLaMA-8B is reported to match or exceed a 70 B LLaMA baseline and OMT-NLLB on nearly every metric across all 1,600 languages. Several headline results are specified:

• On Bible $s$5 in ChrF++, OMT-LLaMA-8B maintains high quality at $s$6 ChrF++ for approximately 1,200 languages, whereas most baselines drop to random at approximately $s$7 ChrF++ by 300–400 languages.
• On BOUQuET $s$8 and $s$9, OMT-LLaMA-8B leads by 3–8 ChrF++ points over the best general LLMs, including Gemma-27B and GPT-OSS-120B, and by 5–15 points over NLLB-200-3B.
• In scaling analyses, the 3.2B model is approximately at parity with the 8B model for $t$0 on BOUQuET, and the 1.2B model is only 5–10% behind.

The paper characterizes this as a specialization advantage: OMT-LLaMA models from 1.2 B to 8 B outperform a 70 B LLM by a wide margin on MT metrics, forming a favorable Pareto frontier of cost versus performance. In the low-resource regime, the quality-threshold analysis based on $t$1 using a MetricX-to-XSTS proxy shows OMT-LLaMA covering approximately 440 languages, versus approximately 220 for NLLB-200.

The RAG setting produces additional gains in the hardest 20 BOUQuET directions: OMT-LLaMA + RAG achieves approximately 29 ChrF++ versus approximately 20 for base LLaMA, and approximately 0.34 BLASER3 versus approximately 0.26. Human evaluation on Met-BOUQuET Round 2, covering 57 directions, shows OMT-LLaMA winning 77% of pairwise comparisons against the best external baselines, with mean $t$2 of 3.38 versus 2.67. The largest gains occur for high-to-low-resource directions.

The paper also reports a toxicity analysis using OmniTOX, described as a 1,600+-language classifier with ROC AUC 0.86. On BOUQuET, all systems exhibit at most 1.5% flagging rates. OMT-LLaMA shows a small directionality bias, slightly higher in $t$3 than $t$4, which the paper states is consistent with model architecture.

6. Technical interpretation and implications

A central conclusion is that multilingual MT at extreme scale is constrained less by latent language understanding than by target-language generation fidelity. The paper explicitly frames this as an “understanding” versus generation bottleneck: large LLMs can understand many languages, but often fail to generate coherently in the long tail. OMT-LLaMA’s specialized CPT recipe, combining bitext, monolingual, and synthetic data with vocabulary extension, is presented as the mechanism that unlocks robust generation up to approximately 1,200 languages (Team et al., 17 Mar 2026).

The efficiency implications are equally important. The paper states that small OMT-LLaMA models in the 1–3 B range match 70 B LLM baselines at a tiny fraction of compute, making high-quality MT accessible in low-compute or on-device deployments. This suggests that, for multilingual MT, specialization can dominate sheer scale when the training objective and data curriculum are aligned with the task.

The framework is also described as extensible. Because OMT-LLaMA is decoder-only and instruction-tuned, it supports easy fine-tuning on new languages using MeDLEy seed data or domain-specific RAG, with further gains requiring minimal effort. A plausible implication is that the model family is intended not as a fixed endpoint, but as an expandable substrate for long-tail language inclusion.

The evaluation stack carries a separate methodological implication. By combining reference-based and reference-free metrics—MetricX, xCOMET, and BLASER3—with large-scale human annotation through BOUQuET and Met-BOUQuET, the paper proposes a more robust framework for omnilingual MT assessment than benchmarking on a few hundred languages alone. This is significant because claims of multilingual capability at the 1,600-language scale depend not only on generation quality, but on whether quality can be measured consistently across domains, scripts, and resource levels.

In sum, OMT-LLaMA is best understood as a task-specialized decoder-only LLaMA adaptation for extreme-scale MT. Its defining features are narrow architectural modification, aggressively multilingual data curation, causal training over both monolingual and parallel streams, optional RAG at inference time, and an evaluation protocol designed for the long tail of global language diversity.