EuroLLM: Multilingual LLM for Europe

• EuroLLM is a comprehensive, open-weight multilingual language modeling initiative targeting all 24 EU languages and additional low-resource languages.
• It employs innovative methodologies including tailored data filtering, tokenizer development, instruction tuning, and extensive evaluation frameworks.
• The project demonstrates significant impact through scalable deployment, modular integration, and open licensing, fostering further multilingual NLP research.

The EuroLLM project is a large-scale, open-weight initiative advancing multilingual large language modeling for all official European Union (EU) languages and additional relevant languages. The project targets both general-purpose NLP tasks and machine translation, addressing the chronic underrepresentation of non-English and low-resource European languages in previous open LLMs. EuroLLM’s scope encompasses architectural innovation, multilingual data filtering, tokenizer development, instruction tuning, evaluation frameworks, and infrastructure for open research (Martins et al., 2024, Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026, Thellmann et al., 2024, Ali et al., 2024, Tantaroudas et al., 7 Apr 2026).

1. Motivation and Scope

The design and training of open-weight LLMs have historically prioritized English and a narrow set of high-resource languages, leading to systematic underperformance on minority, regional, and low-resource European languages. EuroLLM aims to remedy this disparity by supporting all 24 official EU languages (BG, HR, CS, DA, NL, EN, ET, FI, FR, DE, EL, HU, GA, IT, LV, LT, MT, PL, PT, RO, SK, SL, ES, SV), as well as Arabic, Catalan, Chinese, Galician, Hindi, Japanese, Korean, Norwegian, Russian, Turkish, and Ukrainian. The project constructs a suite of open-weight LLMs (1.7B, 9B, 22B parameters) optimized for Europe-centric downstream tasks, including instruction following, reasoning, and high-fidelity machine translation (Martins et al., 2024, Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026).

2. Data Collection, Filtering, and Curation

Data collection for EuroLLM is characterized by extensive web and curated sources, engineered for multilingual balance:

• Web Data: Sources include FineWeb-Edu (English; score>2 for educational value), RedPajama-v2 (DE/ES/FR/IT), HPLT, MADLAD-400, CulturaX, and mC4. Deduplication, language identification, and perplexity-based heuristics (KenLM, CCNet pipeline) are employed to ensure quality (Martins et al., 2024, Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026).
• Parallel Data: Large-scale, sentence-aligned bilingual corpora (Europarl, ParaCrawl, CCMatrix, OPUS, TICO-19, MultiUN, ELITR-ECA) filtered by Bicleaner (≥0.6 for PT, ≥0.5 for others) and CometKiwi (≥0.7), with deduplication via Bifixer (Martins et al., 2024, Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026).
• High-Quality and Domain Data: Wikipedia, arXiv, public-domain books, Apollo, Cosmopedia (synthetic educational content), and curated mathematical/code datasets (The Stack, Open-Web-Math, GSM8K, synthetic MathInstruct/Qwen2.5 samples) (Martins et al., 2024, Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026).
• Filtering Innovations: The EuroFilter classifier, based on an mDeBERTa backbone, is used to score document quality across all languages, facilitating multi-phase, tiered data exposure and quality upsampling (Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026).
• Instruction-Tuning Data: EuroBlocks-Synthetic, containing up to 10.6M instruction–response pairs, is constructed using both human-written and LLM-generated prompts with ranking by reward models such as ArmoRM and Skywork-Gemma2-27B. This dataset enhances coverage especially for low-resource and domain-specific language pairs (Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026).

During training, data mixtures shift to increase non-English fractions in later "annealing" phases (e.g., final 10% of tokens: 32.5% English, 60.5% other languages, 7% code/math) (Martins et al., 2024). This approach systematically improves multilingual generalization without sacrificing high-resource performance.

3. Model and Tokenizer Architectures

Across all EuroLLM versions, a decoder-only transformer backbone is employed, incorporating modular design choices tailored for scalable multilinguality:

• Tokenizer: A 128,000-token shared BPE vocabulary is trained on multilingual text, covering Latin, Cyrillic, Greek, Arabic, Devanagari, CJK, and Hangul scripts. Byte-fallback and script-specific tokens are present. Fertility (pieces/word) is matched or surpassed compared to Gemma 256k, LLaMA-3 128k, and Mistral 32k, and balanced across morphologically complex languages (Martins et al., 2024, Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026).
• Model Architecture: Parameterizations vary by scale, shown below for primary EuroLLM releases (Ramos et al., 5 Feb 2026):

Model	Layers	d_model	d_ff	Heads	GQA KV-groups	Params (B)
EuroLLM-1.7B	24	2048	5632	16	8	1.657
EuroLLM-9B	42	4096	12288	32	8	9.153
EuroLLM-22B	54	6144	16384	48	8	22.639

• All models use SwiGLU activation, pre-layer RMSNorm normalization (as in Zhang & Sennrich), RoPE positional encodings (θ=10,000–1,000,000 for context extension), and Grouped-Query Attention for efficient scaling and inference (Ramos et al., 5 Feb 2026, Martins et al., 4 Jun 2025, Martins et al., 2024).

Instruction-tuning layers employ chat-style control tokens (<|im_start|>, <|im_end|>) and apply specialized fine-tuning for translation or instruction-following (Martins et al., 2024, Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026).

4. Training Methodologies and Scaling Laws

EuroLLM models are trained on ∼4 trillion tokens using Megatron-LM on EuroHPC-scale clusters, with batch sizes up to 12M tokens and mixed-precision (bfloat16) Adam/AdamW optimization. Schedulers typically follow a "trapezoid" with 10% warm-up, constant rate, and final decay, with empirical ablations favoring this over cosine schedules for both general and MT benchmarks (Martins et al., 2024, Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026).

Scaling law analysis, extending Kaplan et al., incorporates the proportion $p$ of parallel-data or high-quality data, with empirical fitting revealing optimal performance as a function of both model size $N$ and data mixture. Added parallel data up to 25% of total corpus does not degrade web/Wikipedia generalization but substantially improves translation; over-augmentation yields diminishing returns (Martins et al., 2024).

Preprocessing pipelines involve multi-stage deduplication, heuristic and probabilistic filtering, and tiered exposure for both text and parallel data. Context extension (4K to 32K) is supported by architectural upgrades to RoPE and memory-optimized fused kernels (Ramos et al., 5 Feb 2026).

Post-training/instruction tuning is performed with Axolotl or analogous frameworks, using cross-entropy loss (targets only), >3 epochs, and domain/multilingual packing. Synthetic instructions are ranked with reward models and filtered for readability, diversity, and domain coverage (Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026).

5. Evaluation Frameworks and Benchmarking

Evaluation of EuroLLM employs a comprehensive multilingual suite combining both custom and standardized benchmarks:

• General tasks: ARC Challenge, HellaSwag, MMLU, MMLU-Pro, TruthfulQA, IFEval, BBH, MGSM, GSM8K, HumanEval, GPQA (varied shot settings) (Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026, Thellmann et al., 2024).
• Machine Translation: FLORES-200 (zero-shot greedy, COMET-22 scoring), WMT-23, WMT-24, WMT-25, with comparisons to Gemma-2B/7B/27B, Meta NLLB-200-distilled-600M, and Mistral/Mixtral variants (Martins et al., 2024, Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026, Tantaroudas et al., 7 Apr 2026).
• Evaluation Initiatives: The EuroLLM evaluation framework (EU20-Benchmarks: EU20-MMLU, EU20-HellaSwag, EU20-ARC, EU20-TruthfulQA, EU20-GSM8K) provides fully parallel benchmarks for 21 European languages, employing DeepL Pro for translation, XML tagging for context fidelity, and both automatic (COMET-KIWI) and manual spot validation (Thellmann et al., 2024).

Results demonstrate robust cross-lingual performance, notably:

• EuroLLM-1.7B-Instruct outperforms Meta NLLB-200-distilled-600M by +5 BLEU points on English→French and executes slightly faster (0.529s vs 0.596s) (Tantaroudas et al., 7 Apr 2026).
• On Flores-200, EuroLLM-1.7B-Instruct (en→xx: 86.8, xx→en: 87.2 COMET-22) outperforms Gemma-2B-Instruct and matches Gemma-7B-Instruct, despite smaller non-embedding param counts (Martins et al., 2024).
• EuroLLM-9B-Instruct and EuroLLM-22B-Instruct consistently achieve top-ranked COMET-22 scores among fully-open European models on MT and general understanding (Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026).

The EU20 evaluation framework enables performance comparison for 40+ LLMs (1.7B–70B) on 21 languages, finding that EuroLLM maintains strong accuracy on high-resource languages, with cross-lingual gaps narrowing at ≥22B scale (Thellmann et al., 2024).

6. System Integration and Deployment

EuroLLM is engineered for containerized, cloud-native deployment. A typical integration, as detailed in the modular XR education platform, involves:

1. Speech capture and ASR (OpenAI Whisper)
2. Translation API (EuroLLM-1.7B-Instruct or alternatives via RESTful microservice)
3. Downstream processing (RoBERTa-based emotion, Flan-T5 summarisation)
4. TTS (AWS Polly) and International Sign avatar animation.

Deployment uses Docker containers with horizontal autoscaling on AWS ECS/EKS, scheduling GPU-backed nodes for translation jobs. End-to-end system latency (STT→translation→TTS or IS rendering) remains below 800 ms for up to 1,000 concurrent users, supporting real-time immersive education use-cases (Tantaroudas et al., 7 Apr 2026). Sub-€0.01 per-translation costs are achieved in large-scale runs.

The modular design enables model selection by feature flags, rapid integration of new language pairs or translation styles, and independent scaling or tuning of translation, TTS, and sign-language components (Tantaroudas et al., 7 Apr 2026).

7. Open Research, Licensing, and Impact

EuroLLM releases all major model checkpoints, tokenizers, data filters, instruction corpora (e.g., EuroBlocks-SFT-2512), and code under permissive licenses (Apache-2.0 for models, CC-BY-4.0 for data, MIT for code), fostering an open ecosystem for research, domain specialization, and public-sector adoption (Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026). Model cards and data documentation accompany each asset.

Implications include:

• Enabling EU institutions, educational platforms, and developers to deploy state-of-the-art, privacy-preserving, and locally-controlled LLM solutions.
• Supporting compliance with the EU Digital Education Action Plan and Accessibility Act via accessible, high-fidelity multilingual and translation services (Tantaroudas et al., 7 Apr 2026).
• Providing a foundation for further research in cross-lingual NLP, resource-augmentation for low-resource and regional languages, and extensible benchmarking (Thellmann et al., 2024, Ramos et al., 5 Feb 2026).

A plausible implication is that EuroLLM’s scaling, open distribution, and modular design set methodological standards for future multilingual LLM development in regional and policy-constrained contexts.

---

References:

(Martins et al., 2024, Martins et al., 4 Jun 2025, Ramos et al., 5 Feb 2026, Thellmann et al., 2024, Ali et al., 2024, Tantaroudas et al., 7 Apr 2026)