GLAP Multilingual Pretraining

• GLAP is a multilingual pretraining paradigm that harnesses large-scale monolingual and parallel corpora to achieve strong cross-lingual alignment and transfer.
• It employs varied architectures—encoder-decoder, decoder-only, and dual-encoder—with objectives like denoising autoencoding, autoregressive modeling, and symmetric contrastive loss.
• Extensions include vocabulary expansion, algorithmic data filtering, and explicit cross-lingual mapping to boost performance in low-resource and multimodal applications.

Multilingual Pretraining (GLAP)

Multilingual Pretraining, hereafter referred to using the abbreviation "GLAP" as introduced in "Multilingual Translation with Extensible Multilingual Pretraining and Finetuning" (Tang et al., 2020), encompasses a suite of methods for constructing large language and translation models with broad multilingual capacity. GLAP-style pipelines focus on leveraging monolingual or parallel corpora for many languages to create models that can be flexibly extended, efficiently finetuned, and achieve strong transfer—particularly to low-resource languages. Since its introduction, the GLAP paradigm has evolved to cover both text and multimodal (notably audio-text) domains, incorporate advanced sampling schemes, and tightly couple pretraining objectives with model architecture for improved cross-lingual alignment and transferability.

1. Foundational Objectives and Model Architectures

GLAP's foundational methodology centers on large-scale pretraining over multilingual corpora using self-supervised objectives tailored to the available data and the model class (encoder-only, decoder-only, or encoder-decoder). The original GLAP formulation in (Tang et al., 2020) uses the denoising autoencoder objective of mBART:

$$L(\theta) = \sum_{i=1}^N \sum_{x \in \mathcal{D}_i} \log P_\theta\bigl(x \mid g(x)\bigr)$$

where $g(x)$ applies random span masking and permutation to the sentence $x$ in language $i$. The model is a 12-layer Transformer encoder+decoder stack, using a shared subword vocabulary (typically constructed via SentencePiece) across all supported languages.

Extensions of GLAP to decoder-only transformer LLMs (e.g., LLaMA, GEMMA, Qwen2) (Wang et al., 2024, Wang et al., 18 Feb 2025) employ the standard autoregressive causal language modeling (CLM) objective:

$$\mathcal{L} = -\sum_{t=1}^T \log P(x_t \mid x_{<t})$$

For multimodal GLAP frameworks (audio-text), as in (Dinkel et al., 12 Jun 2025), the core objective is a symmetric, sigmoid-based contrastive loss computed on pairs of text and audio embeddings projected into a common space, allowing aligned representations across modalities and languages.

Table: Representative Objectives in GLAP Frameworks

Setting	Model class	Core Pretraining Objective
Text (mBART)	Enc-Dec	Denoising autoencoder (span-masking)
Text (LLM)	Decoder-only	Autoregressive CLM
Text (mT5)	Enc-Dec	MLM + span-masked seq2seq
Audio-Text	Dual-encoder	Symmetric contrastive (sigmoid variant)

2. Corpus Construction, Data Curation, and Language Balancing

A core component of GLAP is the construction of diverse, high-quality multilingual corpora. Original efforts paired Wikipedia/CommonCrawl with monolingual news/web data (mBART-25/50 (Tang et al., 2020)). Recent GLAP-style pipelines have proposed algorithmic data filtering for quality (JQL (Ali et al., 28 May 2025)) and massive synthetic corpora via machine translation from a single high-resource source (typically English):

• Machine-translated corpora: FineWeb-Edu, a 100B-token English collection, is translated into multiple languages using scaleable NMT (NLLB-200-1.3B (Wang et al., 18 Feb 2025)) or instruction-tuned LLMs (Mistral-7B-Instruct (Wang et al., 2024)).
• Data quality filtering: JQL leverages frozen multilingual text-embedding backbones and tiny regressors distilled from LLM judges to filter web-scale corpora, outperforming heuristic filtered baselines (e.g., FineWeb2) and boosting downstream LLM accuracy by up to 7% in token-normalized benchmarks (Ali et al., 28 May 2025).
• Sampling for fairness: GLAP sampling initially used heuristic temperature-based upsampling for low-resource languages. The UniMax algorithm (Chung et al., 2023) supersedes this by capping the number of epochs per language, achieving nearly uniform language representation while preventing overfitting on extremely low-resource data.

3. Extensibility: Adding New Languages and Vocabulary

GLAP is designed to be extensible: new languages or scripts can be added post hoc without architecture changes. The protocol is:

1. Vocabulary extension: Add new language-ID tokens and subwords as needed (randomly initialized or with semantically informed vectors).
2. Continued pretraining: Resume denoising autoencoder or CLM objective with pooled corpora including the new language(s), typically for several hundred thousand updates.
3. Parameter efficiency: The OFA framework (Liu et al., 2023) accelerates convergence and reduces parameter count by factorizing embedding tables and initializing subwords using external aligned static vectors, rather than random initialization.

Ablations confirm that performance on previously supported languages does not degrade after extension (Tang et al., 2020), provided continued pretraining is used and sampling is balanced.

4. Multilingual Finetuning and Transfer

GLAP emphasizes one-shot, truly multilingual finetuning over many translation directions or tasks simultaneously. For N language pairs, parallel data are mixed with probabilities upsampled by pairwise resource, controlled via a temperature hyperparameter or (in newer work) explicit caps (Chung et al., 2023). Each training sample receives source and target language-ID tokens (in BART/mBART derivatives), or language/script embeddings (LangSAMP (Liu et al., 2024)) incorporated into the Transformer outputs to maximize cross-lingual representation sharing.

Key findings:

• Multilingual finetuning delivers consistent BLEU gains (1–3 over strongest baselines, up to +18 on low-resource pairs) for MT (Tang et al., 2020).
• Mixed multilingual pretraining is generally superior to continued pretraining on a single target, especially for overall cross-lingual transfer and knowledge consistency (Gao et al., 2024).

5. Pretraining Objectives: Translation, Language Modeling, and Explicit Cross-Lingual Mapping

The choice of pretraining objective in GLAP is tightly coupled to the downstream architecture and resource setting:

• Translation objective: Encoder-decoder models benefit most from explicit translation pretraining, with substantial gains for token-level tasks (POS, NER) over denoising autoencoding or masked language modeling (Li et al., 2024).
• Language modeling objectives: For decoder-only LLMs, causal LM or span-masked objectives suffice for most transfer settings. When parallel data is abundant, combining translation and language modeling objectives—especially via a cross-lingual mapping loss as in (Zheng et al., 12 Apr 2026)—dramatically enhances alignment (up to +12 BLEU, +6.7 BERTScore-Precision).
• Contrastive objectives: For audio-text GLAP (Dinkel et al., 12 Jun 2025), symmetric contrastive objectives over batch-wise audio-text pairs yield strong multilingual retrieval and zero-shot classification with minimal adaptation.

Single-source MT corpora, when carefully curated and filtered, can yield performance closely matching multi-source or proprietary corpora for both reasoning and translation tasks (Wang et al., 2024, Wang et al., 18 Feb 2025).

6. Cross-Lingual Alignment and Evaluation

GLAP research has developed new analytic tools to assess and quantify cross-lingual consistency and transfer:

• CLiKA framework (Gao et al., 2024): Evaluates cross-lingual knowledge alignment at the levels of performance (rescaled accuracy), consistency (overlap of correct answers), and conductivity (retrieval of knowledge acquired in one language by prompts in another). Empirically, mixed multilingual pretraining and instruction tuning boost performance/consistency but have limited effect on deep knowledge conductivity.
• Language Alignment Coefficient (LAC) (Zheng et al., 12 Apr 2026): Measures cross-lingual embedding similarity across Transformer layers, robustly estimating alignment even with limited data. Higher LAC correlates with stronger transfer in MT, QA, and NLU.
• Representation neutrality: Incorporation of explicit language/script embeddings (LangSAMP (Liu et al., 2024)) ensures that language-specific information is disentangled from token embeddings, resulting in more language-neutral contextual representations and improved zero-shot transfer.

7. Impact on Low-Resource and Multimodal NLP

A hallmark of the GLAP paradigm is its ability to deliver large performance gains in extremely low-resource settings:

• Pretrained models incorporating large-scale monolingual data enable translation models for languages with only 4–30 K parallel pairs, achieving up to +18 BLEU over bilingual baselines (Tang et al., 2020).
• Extensive benchmarks (e.g., ML50 (Tang et al., 2020), CuatroBen (Wang et al., 2024)) document consistent gains (10–15 BLEU, 2–5% absolute accuracy) over leading closed-data LLMs when pivoting through high-quality MT-augmented corpora.
• In multimodal domains, GLAP enables unified audio-text models that support retrieval, classification, and keyword spotting across 145 languages without task-specific finetuning (Dinkel et al., 12 Jun 2025), outperforming prior state-of-the-art in both speech and sound understanding domains.

8. Recommendations and Open Directions

GLAP-based multilingual pretraining pipelines benefit most from:

• Balanced, algorithmically sampled corpora (UniMax or temperature-based, not naive frequency sampling).
• Incorporation of high-quality synthetic (MT) data and rigorous, scalable filtering (JQL) for both efficiency and downstream performance.
• Explicit cross-lingual mapping losses and/or auxiliary language/script-aware embeddings in the architecture to boost alignment stability and transfer.
• For low-resource or new languages, vocabulary extension plus continued pretraining with well-initialized embeddings is preferable to random extension or monolingual-only adaptation.
• Mixed pretraining (rather than monolingual continuation) is favored for comprehensive transfer, as confirmed by CLiKA and cross-lingual NLU/QA benchmarks (Gao et al., 2024).

Open challenges include optimizing GLAP at scale for hundreds of languages, improving cross-lingual knowledge conductivity, and generalizing the paradigm across modalities, domains, and instruction formats. The continued convergence of large-scale multilingual, multimodal, and high-efficiency pretraining frameworks exemplifies GLAP's sustained impact on global language AI research.