XGLM: Multilingual Transformer Models

Updated 3 July 2026

XGLM is a family of multilingual, autoregressive decoder-only Transformer models that excel at few-shot learning, cross-lingual transfer, and translation tasks.
It leverages a GPT-3–style architecture with unified SentencePiece/BPE tokenization across 30 languages, balancing high-, medium-, and low-resource data.
Empirical results show XGLM outperforms comparable models on cross-lingual benchmarks, with efficient prompt tuning and scaling enhancing its performance.

XGLM is a family of multilingual, autoregressive, decoder-only Transformer LLMs explicitly designed for broad-coverage multilingual few-shot learning, cross-lingual transfer, and generative tasks such as machine translation. XGLM models are distinctive in their large parameter counts, up-sampled data coverage of 30 languages spanning high, medium, and low-resource families, and a training and inference paradigm focused on eliciting strong zero- and few-shot performance across typologically diverse languages. Empirical results consistently show that XGLM can outperform prior multilingual and monolingual models of comparable or even larger size on a variety of cross-lingual benchmarks, and that its scaling properties, tokenization choices, and fine-tuning strategies are central to its robust cross-linguistic generalization (Lin et al., 2021).

1. Model Architecture and Pretraining

XGLM models follow a GPT-3-style autoregressive decoder-only Transformer architecture with layer-norm, residual connections, and causal self-attention. Several public variants exist:

Variant	Layers	Hidden size	Attention heads	Parameters
XGLM-564M	24	1,024	16	~564M
XGLM-1.7B	24	2,048	16	~1.7B
XGLM-2.9B	48	2,048	16	~2.9B
XGLM-4.5B	N/A	N/A	N/A	~4.5B
XGLM-7.5B	32	4,096	32	~7.5B

All models use a learned positional embedding scheme and a joint Byte-Pair Encoding (BPE)/SentencePiece vocabulary of 250,000 subwords over 30 languages. Pretraining is conducted on roughly 500 billion tokens drawn from CC100-XL, Wikipedia, news, and web data (2013–2020), with multilingual balancing via up-sampling to mitigate dominance by high-resource languages. Training objective is standard left-to-right cross-entropy minimization:

$L(θ) = - \sum_{t=1}^T \log p_θ(x_t \mid x_{< t})$

Larger models (e.g., XGLM-7.5B) incorporate wider layers and more attention heads, linearly scaling context and expressivity (Lin et al., 2021, Li et al., 2023, Michaelov et al., 2022).

2. Multilingual Coverage and Tokenization Strategy

XGLM’s multilingual design covers 30 typologically diverse languages, from English, German, French, Russian, and Chinese (high-resource) to Catalan, Swahili, Tamil, Urdu, and others (medium- and low-resource). Tokenization is performed by fitting a single SentencePiece/BPE model on the combined multilingual corpus, producing shared subword units.

Tokenization “fertility” (subwords per word) is language-dependent: high-resource languages exhibit 1.3–1.4, while low-resource languages often show higher fertility (∼1.5–1.6), reflecting finer-grained subword segmentation due to sparse data. This property strongly impacts prompt tuning effectiveness, especially for low-resource languages, as continuous prompt vectors can exploit subword regularities without disturbing the main model parameters (Park et al., 2023):

Language group	Average fertility (φₗ)
High-resource	1.30
Low-resource	1.49

Chinese and Japanese are edge cases, typically tokenized at the character level with φₗ > 2 (Park et al., 2023).

3. Few-Shot Learning, Prompt Tuning, and Cross-Lingual Transfer

XGLM achieves strong zero-shot and few-shot performance on tasks such as XNLI, PAWS-X, NER, POS, XWinograd, and social-value tasks, often outperforming monolingual English-centric models of similar or greater size (e.g., GPT-3 6.7B) on cross-lingual NLU (Lin et al., 2021). Key findings include:

Prompt tuning (“P-tuning v2”) is highly parameter-efficient. Adding ≤0.2% of parameters as continuous prompts achieves or exceeds full fine-tuning performance, especially for low-resource languages where gains of +4%–8% (XNLI accuracy) are observed.
Multilingual prompting: English-language templates generalize well to non-English data; cross-lingual transfer of both templates and demonstration examples boosts few-shot performance.
Scaling effects: Increasing model size monotonically improves cross-lingual task accuracy, though with diminishing returns for high-resource languages.
Correlation to subword fertility: Gains from prompt tuning on low-resource languages scale with tokenization granularity (Pearson’s r ≈ 0.72).

Practical prompt tuning recommendations include using prompt length m ∼15 (0.13% parameters), initializing with random vectors or language-symbol embeddings, and composing distinct prompts for domain adaptation (Park et al., 2023).

4. Translation and Multilingual Instruction Tuning

XGLM-7.5B demonstrates intrinsic but further unlockable translation capabilities. Simple multilingual finetuning with instruction-style templates (mFTI) such as “Translation: [l_s]: x [l_t]:” and 1,000 parallel sentences per direction leads to BLEU improvements of ≈3 points compared to 8-shot in-context learning, with dramatic gains for non-English directions (+8 BLEU for Catalan).

Key translation findings (Li et al., 2023):

Typological similarity to English, more than pretraining data volume, is the strongest predictor of translation performance (ρ = +0.80 for syntax proximity).
mFTI-trained XGLM-7.5B approaches the performance of supervised NMT models (e.g., M2M-100-1.2B), though it remains behind state-of-the-art supervised models (NLLB-3B) by ≈10 BLEU.
Generalization: XGLM-7.5B with mFTI can translate zero-shot between language pairs never seen during instruction tuning, benefiting from exposure to a wide variety of translation instructions and language-pair templates.
Error reduction: Increasing the number of language pairs and including monolingual instructions reduces rates of source-copying, off-target translations, and under- or over-generation.

Pivot-based translation (X→En→Y) leads to smaller BLEU gains after mFTI (<2 points), indicating learned direct X–Y alignment. Adding English as a pivot during training further improves unseen language-pair translation quality (Li et al., 2023).

5. Structural and Psycholinguistic Properties

XGLM captures linguistic phenomena at a level matching or exceeding state-of-the-art masked LMs and monolingual LMs on both structural probing and psycholinguistic evaluation.

Zero-pronoun resolution: XGLM-2.9B/4.5B/7.5B models align almost perfectly with human reading-time data on Italian zero pronouns across five Carminati (2005) experiments. Lower surprisal on human-preferred referents (subject > object, pronoun > name, first/second person bias) reflects native-like expectations (Michaelov et al., 2022).
Agreement circuits: Causal analysis of XGLM’s internal representations reveals two distinct “agreement circuits”: mid-layer neurons (layers 8–12) for short-range subject–verb agreement and late-layer neurons (14–20) for long-range dependencies. Notably, the same neurons mediate agreement in typologically related languages, a property more pronounced in XGLM than in masked models such as mBERT (Mueller et al., 2022).
Neuron-level density: XGLM distributes agreement processing across ∼4–16% of neurons (for total effect and maximal natural indirect effect), denser than in masked LMs.

These empirical observations support a probabilistic, distributional account of anaphora and agreement in neural models and suggest cross-linguistic parameter sharing is supported by the autoregressive objective and joint tokenization (Mueller et al., 2022).

6. Limitations, Implications, and Recommendations

English-centricity: Despite balanced upsampling, XGLM’s translation and transfer performance show a persistent bias toward English-proximate languages; typological closeness has a stronger effect than source-token count (Li et al., 2023).
Curse of multilinguality: XGLM lags monolingual English LMs by ∼10% accuracy on English-only NLU tasks, reflecting tradeoffs in training data allocation (Lin et al., 2021).
Prompting and tokenization: Fine-grained tokenization in low-resource languages acts as implicit prompt anchors exploitable by prompt tuning, but may also fragment rare morphology. Prompt tuning is superior to full fine-tuning for low-resource transfer.
Scaling and parameter updates: Larger XGLM variants show linear improvements on cross-lingual tasks; lightweight prompt tuning enables rapid adaptation to new languages and domains with minimal parameter updates (<0.2%).

For practitioners, XGLM is recommended as a foundation for multilingual generative and classification tasks where cross-linguistic transfer, resource efficiency, and human-like linguistic competence are priorities.

7. Comparative Performance and Broader Impact

Across multiple benchmarks, XGLM sets new standards for zero-/few-shot cross-lingual learning and instruction-tuned translation in medium-sized LMs. In zero-pronoun resolution, XGLM is the only family (alongside GePpeTto) to fully replicate all human behavioral benchmarks, outperforming both masked multilingual LMs and monolingual Italian BERTs (Michaelov et al., 2022).

Comparison Table: Model Success Rates on Carminati Italian Zero-Pronoun Experiments

Model	# Experiments (out of 5)
XGLM-2.9B/4.5B/7.5B	5
GePpeTto	4
XLM-R Large	3
Italian BERTs	3
mBERT/XLM-100	≤2

XGLM’s design and empirical strengths support unified multilingual NLP with efficient adaptation and robust human-like generalization across languages, morphological systems, and linguistic tasks (Lin et al., 2021, Park et al., 2023, Li et al., 2023, Mueller et al., 2022, Michaelov et al., 2022).