EuroBERT: Scaling Multilingual Encoders for European Languages (2503.05500v2)

Abstract: General-purpose multilingual vector representations, used in retrieval, regression and classification, are traditionally obtained from bidirectional encoder models. Despite their wide applicability, encoders have been recently overshadowed by advances in generative decoder-only models. However, many innovations driving this progress are not inherently tied to decoders. In this paper, we revisit the development of multilingual encoders through the lens of these advances, and introduce EuroBERT, a family of multilingual encoders covering European and widely spoken global languages. Our models outperform existing alternatives across a diverse range of tasks, spanning multilingual capabilities, mathematics, and coding, and natively supporting sequences of up to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering insights into our dataset composition and training pipeline. We publicly release the EuroBERT models, including intermediate training checkpoints, together with our training framework.

• The paper introduces EuroBERT, a set of multilingual encoders that leverage recent advances in generative modeling to enhance performance on various tasks.
• The models achieve superior performance across multilingual tasks, mathematics, and coding while supporting sequences up to 8,192 tokens.
• The study provides detailed insights into its dataset and training, publicly releasing models with intermediate checkpoints for the research community impatiently.

EuroBERT advances multilingual language encoding for European languages and broad global use.

• EuroBERT introduces multilingual encoders that leverage advances from generative model innovations for broader utility (2503.05500).
• The models achieve superior performance across multilingual tasks, mathematics, and coding while supporting sequences up to 8,192 tokens (2503.05500).
• The paper provides detailed insights into the dataset composition, training pipeline, and public release of models with intermediate checkpoints (2503.05500).