---
title: 'XGLM: Multilingual Transformer Models'
url: https://www.emergentmind.com/topics/xglm
type: topic
---

# XGLM: Multilingual Transformer Models

XGLM is a family of multilingual, autoregressive, decoder-only Transformer language models explicitly designed for broad-coverage multilingual few-shot learning, cross-lingual transfer, and generative tasks such as machine translation. XGLM models are distinctive in their large parameter counts, up-sampled data coverage of 30 languages spanning high, medium, and low-resource families, and a training and inference paradigm focused on eliciting strong zero- and few-shot performance across typologically diverse languages. Empirical results consistently show that XGLM can outperform prior multilingual and monolingual models of comparable or even larger size on a variety of cross-lingual benchmarks, and that its scaling properties, tokenization choices, and fine-tuning strategies are central to its robust cross-linguistic generalization [2112.10668].

## 1. Model Architecture and Pretraining

XGLM models follow a GPT-3-style autoregressive decoder-only Transformer architecture with layer-norm, residual connections, and causal self-attention. Several public variants exist:

| Variant      | Layers | Hidden size | Attention heads | Parameters |
|--------------|--------|-------------|-----------------|------------|
| XGLM-564M    | 24     | 1,024       | 16              | ~564M      |
| XGLM-1.7B    | 24     | 2,048       | 16              | ~1.7B      |
| XGLM-2.9B    | 48     | 2,048       | 16              | ~2.9B      |
| XGLM-4.5B    | N/A    | N/A         | N/A             | ~4.5B      |
| XGLM-7.5B    | 32     | 4,096       | 32              | ~7.5B      |

All models use a learned positional embedding scheme and a joint Byte-Pair Encoding (BPE)/SentencePiece vocabulary of 250,000 subwords over 30 languages. Pretraining is conducted on roughly 500 billion tokens drawn from CC100-XL, Wikipedia, news, and web data (2013–2020), with multilingual balancing via up-sampling to mitigate dominance by high-resource languages. Training objective is standard left-to-right cross-entropy minimization:

$$
L(θ) = - \sum_{t=1}^T \log p_θ(x_t \mid x_{< t})
$$

Larger models (e.g., XGLM-7.5B) incorporate wider layers and more attention heads, linearly scaling context and expressivity [2112.10668][2305.15083][2208.14554].

## 2. Multilingual Coverage and Tokenization Strategy

XGLM’s multilingual design covers 30 typologically diverse languages, from English, German, French, Russian, and Chinese (high-resource) to Catalan, Swahili, Tamil, Urdu, and others (medium- and low-resource). Tokenization is performed by fitting a single SentencePiece/BPE model on the combined multilingual corpus, producing shared subword units.

Tokenization “fertility” (subwords per word) is language-dependent: high-resource languages exhibit 1.3–1.4, while low-resource languages often show higher fertility (∼1.5–1.6), reflecting finer-grained subword segmentation due to sparse data. This property strongly impacts prompt tuning effectiveness, especially for low-resource languages, as continuous prompt vectors can exploit subword regularities without disturbing the main model parameters [2311.07820]:

| Language group      | Average fertility (φₗ) |
|---------------------|------------------------|
| High-resource       | 1.30                   |
| Low-resource        | 1.49                   |

Chinese and Japanese are edge cases, typically tokenized at the character level with φₗ > 2 [2311.07820].

## 3. Few-Shot Learning, Prompt Tuning, and Cross-Lingual Transfer

XGLM achieves strong zero-shot and few-shot performance on tasks such as XNLI, PAWS-X, NER, POS, XWinograd, and social-value tasks, often outperforming monolingual English-centric models of similar or greater size (e.g., GPT-3 6.7B) on cross-lingual NLU [2112.10668]. Key findings include:

- **Prompt tuning (“P-tuning v2”) is highly parameter-efficient.** Adding ≤0.2% of parameters as continuous prompts achieves or exceeds full fine-tuning performance, especially for low-resource languages where gains of +4%–8% (XNLI accuracy) are observed.
- **Multilingual prompting**: English-language templates generalize well to non-English data; cross-lingual transfer of both templates and demonstration examples boosts few-shot performance.
- **Scaling effects**: Increasing model size monotonically improves cross-lingual task accuracy, though with diminishing returns for high-resource languages.
- **Correlation to subword fertility**: Gains from prompt tuning on low-resource languages scale with tokenization granularity (Pearson’s r ≈ 0.72).

Practical prompt tuning recommendations include using prompt length m ∼15 (0.13% parameters), initializing with random vectors or language-symbol embeddings, and composing distinct prompts for domain adaptation [2311.07820].

## 4. Translation and Multilingual Instruction Tuning

XGLM-7.5B demonstrates intrinsic but further unlockable translation capabilities. Simple multilingual finetuning with instruction-style templates (mFTI) such as “Translation: [l_s]: x [l_t]:” and 1,000 parallel sentences per direction leads to BLEU improvements of ≈3 points compared to 8-shot in-context learning, with dramatic gains for non-English directions (+8 BLEU for Catalan).

Key translation findings [2305.15083]:

- **Typological similarity to English**, more than pretraining data volume, is the strongest predictor of translation performance (ρ = +0.80 for syntax proximity).
- **mFTI-trained XGLM-7.5B** approaches the performance of supervised NMT models (e.g., M2M-100-1.2B), though it remains behind state-of-the-art supervised models (NLLB-3B) by ≈10 BLEU.
- **Generalization**: XGLM-7.5B with mFTI can translate zero-shot between language pairs never seen during instruction tuning, benefiting from exposure to a wide variety of translation instructions and language-pair templates.
- **Error reduction**: Increasing the number of language pairs and including monolingual instructions reduces rates of source-copying, off-target translations, and under- or over-generation.

Pivot-based translation (X→En→Y) leads to smaller BLEU gains after mFTI (<2 points), indicating learned direct X–Y alignment. Adding English as a pivot during training further improves unseen language-pair translation quality [2305.15083].

## 5. Structural and Psycholinguistic Properties

XGLM captures linguistic phenomena at a level matching or exceeding state-of-the-art masked LMs and monolingual LMs on both structural probing and psycholinguistic evaluation.

- **Zero-pronoun resolution**: XGLM-2.9B/4.5B/7.5B models align almost perfectly with human reading-time data on Italian zero pronouns across five Carminati (2005) experiments. Lower surprisal on human-preferred referents (subject > object, pronoun > name, first/second person bias) reflects native-like expectations [2208.14554].
- **Agreement circuits**: Causal analysis of XGLM’s internal representations reveals two distinct “agreement circuits”: mid-layer neurons (layers 8–12) for short-range subject–verb agreement and late-layer neurons (14–20) for long-range dependencies. Notably, the same neurons mediate agreement in typologically related languages, a property more pronounced in XGLM than in masked models such as mBERT [2210.14328].
- **Neuron-level density**: XGLM distributes agreement processing across ∼4–16% of neurons (for total effect and maximal natural indirect effect), denser than in masked LMs.

These empirical observations support a probabilistic, distributional account of anaphora and agreement in neural models and suggest cross-linguistic parameter sharing is supported by the autoregressive objective and joint tokenization [2210.14328].

## 6. Limitations, Implications, and Recommendations

- **English-centricity**: Despite balanced upsampling, XGLM’s translation and transfer performance show a persistent bias toward English-proximate languages; typological closeness has a stronger effect than source-token count [2305.15083].
- **Curse of multilinguality**: XGLM lags monolingual English LMs by ∼10% accuracy on English-only NLU tasks, reflecting tradeoffs in training data allocation [2112.10668].
- **Prompting and tokenization**: Fine-grained tokenization in low-resource languages acts as implicit prompt anchors exploitable by prompt tuning, but may also fragment rare morphology. Prompt tuning is superior to full fine-tuning for low-resource transfer.
- **Scaling and parameter updates**: Larger XGLM variants show linear improvements on cross-lingual tasks; lightweight prompt tuning enables rapid adaptation to new languages and domains with minimal parameter updates (<0.2%).

For practitioners, XGLM is recommended as a foundation for multilingual generative and classification tasks where cross-linguistic transfer, resource efficiency, and human-like linguistic competence are priorities.

## 7. Comparative Performance and Broader Impact

Across multiple benchmarks, XGLM sets new standards for zero-/few-shot cross-lingual learning and instruction-tuned translation in medium-sized LMs. In zero-pronoun resolution, XGLM is the only family (alongside GePpeTto) to fully replicate all human behavioral benchmarks, outperforming both masked multilingual LMs and monolingual Italian BERTs [2208.14554].

Comparison Table: Model Success Rates on Carminati Italian Zero-Pronoun Experiments

| Model              | # Experiments (out of 5) |
|--------------------|-------------------------|
| XGLM-2.9B/4.5B/7.5B| 5                       |
| GePpeTto           | 4                       |
| XLM-R Large        | 3                       |
| Italian BERTs      | 3                       |
| mBERT/XLM-100      | ≤2                      |

XGLM’s design and empirical strengths support unified multilingual NLP with efficient adaptation and robust human-like generalization across languages, morphological systems, and linguistic tasks [2112.10668][2311.07820][2305.15083][2210.14328][2208.14554].

Source: https://www.emergentmind.com/topics/xglm