---
title: BERT-base-multilingual-cased
url: https://www.emergentmind.com/topics/bert-base-multilingual-cased
type: topic
---

# BERT-base-multilingual-cased

BERT-base-multilingual-cased (commonly abbreviated as mBERT) is a Transformer-based masked language model pretrained on Wikipedia text across 104 languages, sharing a single parameter set and WordPiece vocabulary with case sensitivity. It serves as a widely adopted model in multilingual representation learning, cross-lingual transfer, and the evaluation of typological phenomena in NLP. The following sections summarize mBERT’s architecture, pretraining, transfer behavior, linguistic properties, and performance across core linguistic and applied tasks, referencing principal results from recent arXiv literature.

## 1. Model Architecture and Pretraining Scope

mBERT is a direct implementation of the original BERT-Base specification, with key architectural features:

- 12 Transformer encoder layers ($L=12$)
- Hidden size 768 ($H=768$)
- 12 attention heads per layer ($A=12$)
- Feed-forward dimension 3072
- Shared WordPiece vocabulary, size $\approx$119,000 (cased)
- Total parameters $\approx$110–178M (a discrepancy across sources due to counting embedding table vs. transformer weights)

Pretraining is performed on Wikipedia dumps for 104 languages. The shared vocabulary is derived over all languages, preserving case distinctions. The pretraining objective is the sum of the Masked Language Modeling (MLM) loss, where 15% of tokens in each input sequence are masked and predicted, and the Next Sentence Prediction (NSP) loss, which predicts segment contiguity. Standard maximal input sequence length is 512 tokens, enabling large context windows that are shown to be crucial for the model’s eventual cross-lingual transfer capacity [2010.10938, 2004.09205].

## 2. Pretraining Data, Tokenization, and Multilingual Vocabulary

Pretraining encompasses approximately 2.5 billion words ($\approx$12.5B tokens) across typologically diverse languages ranging from high-resource (e.g., English, Spanish) to mid-/low-resource (e.g., Greek, Hindi). The single WordPiece vocabulary facilitates implicit cross-lingual signal transmission via shared subword representations, including extensive overlap in Latin-script substrings, numbers, and punctuation. English comprises the largest segment of the token pool, shaping the statistical properties of the model’s learned representations [2210.05619].

Subword overlap and case preservation increase model flexibility and serve both to anchor alignment (by shared numerals, punctuation) and to enforce finer-grained language-specific patterns, e.g., German noun capitalization [2010.10938].

## 3. Cross-Lingual Transfer and Representational Properties

mBERT exhibits strong zero-shot transfer capacity, as substantiated by multilingual benchmarks such as word retrieval (MRR on MUSE dictionaries) and zero-shot NLI classification (XNLI accuracy). At 200K sentences per language, performance trails static models, but with 1M+ sentences, mBERT delivers a dramatic increase in MRR and XNLI accuracy (e.g., MRR$\approx$0.51, XNLI Acc$\approx$62%) and consistently surpasses static embeddings in open-class word alignment, particularly for adjectives and nouns [2010.10938, 2004.09205].

Context window size is decisive: truncating down to 20 tokens reduces word-retrieval and XNLI score by up to 30 and 10 points, respectively. For robust cross-lingual generalization, both large monolingual data per language ($>$1M sentences) and long context windows ($>$128 tokens) are mandatory [2010.10938].

Manipulating mBERT’s latent representations—inserting per-language bias vectors—allows direct control of output language emissions and enables unsupervised token-level translation. mBERT’s ability to switch output vocabulary space via simple mean-difference vector arithmetic is empirically validated for multiple scripts, with conversion rates approaching 100% (e.g., $en\to el$ at $\alpha=3$) [2004.09205].

## 4. Linguistic Universals and Grammatical Biases

mBERT internalizes grammatical relations that co-cluster according to the Universal Dependencies taxonomy. Structural probe analyses show that a learned subspace ($B$) recovers syntactic tree distances and dependency labels across languages with high fidelity (UUAS$\approx$74.1%, DSpr$\approx$0.804 on joint subspaces). Notable universal clusters include prenominal vs. postnominal adjective modification, core arguments, determiners, and expletives [2005.04511].

Despite capturing cross-lingual structure, mBERT shows measurable “grammatical structure bias”—the tendency to reproduce grammatical properties favored in dominant training languages (principally English). For Spanish pronoun-drop and Greek subject–verb order, mBERT assigns 10–15% higher probability to English-like structures relative to strong monolingual baselines (BETO, GreekBERT), thus overproducing explicit subjects and rigid SVO order (bootstrap $p<.05$) [2210.05619]. This structural drift can degrade fluency and register in generation or pseudo-probing tasks.

Table: Grammatical Structure Bias (English-like preference ratio $r_{model}$)

| Language | Monolingual Model $r_{mono}$ | mBERT $r_{multi}$ | p-value | Bias Direction         |
|----------|------------------------------|-------------------|---------|-----------------------|
| Spanish  | 0.93 (95% CI [0.89,0.97])   | 1.06 (95% CI [1.02,1.10]) | $<.05$   | Prefers explicit subject |
| Greek    | 0.98 (95% CI [0.94,1.02])   | 1.12 (95% CI [1.07,1.17]) | $<.05$   | Prefers SVO order       |

Mitigation strategies include fine-tuning on in-domain data, language-specific adapters, and rebalancing pretraining corpora [2210.05619].

## 5. Morphosyntactic Probing and Embedding Space Structure

Extensive probing shows mBERT encodes rich morphology: average test accuracy 90.4% across 247 tasks in 42 languages (XLM-RoBERTa: 91.8%, chLSTM: 85%). Probing setups extract the last subword vector of target words, aggregate across layers, and train a classifier atop these features. Shapley-value and masking perturbations reveal that preceding context is systematically more morphologically informative than following context (left/right context ratio $\approx$1.45), consistent with typological markedness—even though BERT’s attention is bidirectional [2306.06205]. Cases of reversed asymmetry (postpositions, agreement) correspond to known language-specific morphological triggers.

Table: Morphological Probing—Model Accuracy

| Model     | Mean accuracy (%) |
|-----------|------------------|
| mBERT     | 90.4             |
| XLM-R     | 91.8             |
| chLSTM    | 85.0             |
| fastText  | 83.3             |
| Stanza    | 91.4             |

## 6. Embedding Space Geometry and Isotropy

Unlike monolingual BERT, mBERT’s embedding space is highly anisotropic but exhibits no “rogue” dimensions—that is, no mean activation exceeding $3\sigma$ from the mean. Anisotropy is instead distributed evenly across several principal components, with the top dimension contributing only $\approx$0.02–0.04 to pairwise cosine similarity (vs. 0.38 in monolingual BERT). Isotropy-enhancing cluster-based postprocessing (PCA subtraction per cluster) yields 10–20-point improvements in semantic similarity Spearman correlation and allows zero-shot transfer of the isotropy correction between languages due to shared degenerate directions [2110.04504].

## 7. Downstream Performance, Fine-tuning, and Applied Recommendations

mBERT, when fine-tuned for downstream tasks in a target language—POS/morph tagging, sentiment, NER, machine comprehension—consistently outperforms non-contextual or traditional neural baselines.

- Estonian NLP: POS/morph tagging accuracy up to 98%, sentiment accuracy 70.2%, NER F1 86.5% [2010.00454].
- Indic languages: transforming vanilla mBERT to “multilingual SBERT” via synthetic NLI/STS fine-tuning raises Hindi sentence similarity Spearman $\rho$ from 0.49 to 0.75, besting alternatives (LaBSE, LASER) [2304.11434].
- Sentiment/hate speech (English, Korean, Japanese, Chinese, French): accuracy 90.2% (with 8-layer freezing); XLM-RoBERTa outperforms on morphologically rich languages by 2 points [2601.02697].
- Machine comprehension (English–Hindi QA): SQuAD F1 up to 64.5 (cross-lingual), setting new SOTA for these pairs [2006.01432].

Layer freezing (typically the first eight layers) mitigates overfitting and preserves general multilingual representation, especially when data are scarce; unfreezing decreases mean accuracy by 2.7 points in hate speech and sentiment tasks [2601.02697]. Explainability via LIME or SHAP confirms model decisions respect semantic cues in each language.

## References

- [2210.05619] Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models
- [2010.10938] What makes multilingual BERT multilingual?
- [2304.11434] L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT
- [2110.04504] An Isotropy Analysis in the Multilingual BERT Embedding Space
- [2005.04511] Finding Universal Grammatical Relations in Multilingual BERT
- [2004.09205] A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
- [2006.01432] BERT Based Multilingual Machine Comprehension in English and Hindi
- [2601.02697] Boosting Accuracy and Interpretability in Multilingual Hate Speech Detection Through Layer Freezing and Explainable AI
- [2010.00454] Evaluating Multilingual BERT for Estonian
- [2306.06205] Morphosyntactic probing of multilingual BERT models

## Summary

BERT-base-multilingual-cased represents a foundational architecture for multilingual representation learning, providing a robust, broadly deployable encoder for over 100 languages. Its strength derives from joint subword vocabulary, deep bidirectional modeling of long-range context, and empirically validated cross-lingual capacity. Notwithstanding its efficacy, users must be alert to grammatical structure bias and anisotropic representation space, applying mitigation strategies—data balancing, targeted fine-tuning, isotropy correction—as needed for fluency-critical and typologically sensitive tasks.

Source: https://www.emergentmind.com/topics/bert-base-multilingual-cased