EuroBERT 210M Encoder

• EuroBERT-210M is a multilingual transformer encoder featuring 12 layers, 210M parameters, and support for long contexts up to 8,192 tokens.
• It employs innovations such as grouped-query attention, SwiGLU activation, and rotary positional embeddings to boost efficiency and performance.
• Empirical evaluations demonstrate its competitive edge over larger models in tasks like IR, code search, and classification across diverse multilingual benchmarks.

EuroBERT-210M is a multilingual encoder model with 210 million parameters developed as part of the EuroBERT family, designed to provide general-purpose vector representations for retrieval, regression, and classification tasks across a wide range of European and globally spoken languages. It revisits encoder-centric architectures, adapting recent innovations typically associated with large-scale decoder models to a transformer-based encoder backbone, and demonstrates competitive or superior performance relative to larger encoder models, especially in multilingual and long-context settings (Boizard et al., 7 Mar 2025).

1. Architecture and Parameterization

EuroBERT-210M is a 12-layer transformer encoder with a model dimension $d=768$, 12 attention heads of size $d/h=64$, and a feed-forward dimension $d_\text{ff}=3072$. Key architectural innovations inherited from Llama 3 include:

• Full removal of bias terms from all layers.
• Grouped-query attention (GQA).
• SwiGLU activation in the feed-forward network.
• RMS layer normalization with $\epsilon=10^{-5}$.
• Rotary position embeddings (RoPE) with a frequency parameter $\theta$ increased to $250\,000$ during annealing to support longer context lengths.

Parameterization follows the formula:

$$P \approx Vd + L\left(4d^2 + 2d\,d_\text{ff}\right)$$

with $V=128\,000$ (vocabulary size), yielding approximately $210 \times 10^6$ parameters. The encoder supports context windows up to $8\,192$ tokens, enabled by RoPE modification and “crop-and-shift” segment sampling.

2. Tokenization and Vocabulary Construction

EuroBERT-210M uses the LLaMA 3 SentencePiece tokenizer with byte-level encoding, fixed at 128,000 tokens. The vocabulary is shared across all 15 supported languages, including code and mathematical expressions, with no language-specific token splits or script separation. This unified tokenization is critical for multilingual and cross-domain applicability.

Supported languages: English, French, German, Spanish, Chinese, Italian, Russian, Polish, Portuguese, Japanese, Vietnamese, Dutch, Arabic, Turkish, Hindi.

3. Pre-training Objective and Scheduling

The model is trained via masked language modeling (MLM). Formally, for a sequence $d/h=64$0 with masked positions $d/h=64$1:

$d/h=64$2

A dual-phase masking schedule is employed:

• Pre-training: 4.8T tokens, 50% mask ratio.
• Annealing: 200B tokens, 10% mask ratio.

The increased mask ratio during initial pre-training directly benefits retrieval and IR tasks, while annealing to a lower ratio enhances classification performance.

4. Training Data, Pipeline, and Preprocessing

EuroBERT-210M leverages a diverse multilingual and multi-domain corpus:

• Pre-training mixture (4.8T tokens):
  • FineWeb (English): 2.00T (41.3%)
  • CulturaX (multilingual): ~1.9T (38.9%)
  • Unbabel parallel (to/from English): ~0.35T (7.2%)
  • Proof-Pile-2 & The Stack v2 (code & math): ~0.25T (5.1%)
• Annealing mixture (200B tokens):
  • EuroLLM quality-filtered data (threshold >3)
  • Balanced to 26% English, ~46% other human languages, 4% code, 6% math, 5% parallel data
  • No instruction fine-tuning data

Preprocessing steps include deduplication, language filtering, fixed-length chunking, asymmetric masking for parallel pairs (concatenated with <|parallel_sep|>), and random cropping for variable-length document simulation.

5. Optimization, Infrastructure, and Code Efficiency

Key training hyperparameters:

Phase	Context	Mask Ratio	LR Scheduler	LR Schedule
Pre-training	2,048 tokens	50%	Warmup–Decay	Warmup 2k steps → 1e-4 const
Annealing	≤8,192 (crop)	10%	Cosine Decay	1e-4 → 0

• Optimizer: AdamW ($d/h=64$3, $d/h=64$4, $d/h=64$5, weight decay 0.1), gradient-norm clip 1.0.
• Per-GPU micro-batch: 24 sequences.
• ≈92 MI250X GPUs (192 logical FSDP devices), aggregating $d/h=64$69.4M effective tokens/step.
• Total compute: $d/h=64$715,000 GPU-hours.

Code-level optimizations include FlashAttention 2, LigerKernel for fused cross-entropy, torch.compile, and hybrid FSDP sharding.

6. Empirical Evaluation and Benchmarking

EuroBERT-210M achieves strong or state-of-the-art results relative to 300–400M parameter baselines across diverse tasks, with results aggregated over the supported European languages:

Task	Metric	mGTE-305M	XLM-RoBERTa-Large (560M)	EuroBERT-210M
MIRACL (IR)	NDCG@10	93.8	91.6	95.1
MLDR (long)	NDCG@10	73.2	65.2	73.4
CC-News (IR)	NDCG@10	71.5	72.1	69.0
XNLI (CLS)	Acc. (%)	78.4	84.1	79.9
PAWS-X (CLS)	Acc. (%)	89.8	92.4	89.9
WMT (QE)	ρ (%)	37.7	39.0	40.5
SeaHorse	ρ (%)	55.8	34.3	59.3
CodeSearchNet	NDCG@10	34.0	40.8	58.9
CodeComplexity	Acc. (%)	74.5	83.6	91.9
MathShepherd	Acc. (%)	77.2	71.9	84.0

EuroBERT-210M outperforms all 300M–400M baselines on MIRACL, MLDR, CodeSearchNet, CodeComplexity, and MathShepherd. It is competitive on XNLI and PAWS-X despite having significantly fewer parameters than XLM-RoBERTa-Large. Importantly, robustness to input sequences up to 8,192 tokens is maintained, outperforming XLM-RoBERTa in long-context settings.

7. Design Choices and Empirical Trade-offs

EuroBERT-210M’s development illustrates several significant trade-offs:

• The initial 50% mask ratio benefits IR at the expense of classification, necessitating annealing to 10% for downstream task balance.
• Upsampling code and math data improves IR performance but can negatively impact classification metrics—final mixture uses 4% code and 6% math as a compromise.
• Increasing parallel data proportion uniformly benefits both IR and CLS tasks.
• Random crop-and-shift sampling in long-context training is crucial for model stability at extended context lengths.
• Quality filtering finds optimal balance by mixing mid- and high-quality data samples, rather than selecting only top data.

8. Availability and Impact

EuroBERT-210M and its associated training scripts, along with intermediate checkpoints, are publicly released at https://huggingface.co/EuroBERT. The model establishes a new empirical standard for compact, scalable multilingual encoders, supporting out-of-the-box application to IR, regression, classification, and code/math tasks in diverse multilingual settings, particularly where long context and robust multilinguality are required (Boizard et al., 7 Mar 2025).