BERTje: Dutch Monolingual BERT

Updated 19 July 2026

BERTje is a monolingual Dutch BERT model trained on a curated 12GB corpus, offering enhanced performance on NER, POS tagging, and sentiment analysis.
It employs whole-word masking and sentence order prediction in its pretraining phase, leading to superior results compared to multilingual approaches.
BERTje excels in general Dutch NLP tasks but shows varied effectiveness in domain-specific applications such as clinical note analysis and bias measurement.

BERTje is a monolingual Dutch BERT model developed as a language-specific alternative to multilingual BERT. It uses the same general BERT $_\text{base}$ architecture, specifically 12 transformer blocks, but is pretrained on a much larger and more diverse Dutch corpus than the Dutch component of multilingual BERT. In the original evaluation, BERTje consistently outperformed equally sized multilingual BERT on Dutch named-entity recognition, part-of-speech tagging, semantic role labeling, spatio-temporal relations, and sentiment analysis, establishing it as a reusable pretrained model for Dutch NLP (Vries et al., 2019). Subsequent work positioned BERTje both as a strong general-domain Dutch encoder and as a component whose effectiveness depends sharply on task formulation, domain shift, input length constraints, and evaluation protocol, particularly in HR de-identification, Dutch clinical NLP, and bias measurement (Toledo et al., 2021, Mosteiro et al., 2022, Strazda et al., 22 Jul 2025, Murphy et al., 25 Jul 2025).

1. Origin and rationale

BERTje was introduced in response to two limitations identified in the then-current Dutch NLP landscape. First, the literature was dominated by English BERT variants. Second, multilingual BERT, although it includes Dutch, was trained only on Wikipedia text across 104 languages and therefore was neither domain-diverse nor specialized for Dutch. The BERTje paper explicitly positions the model as a demonstration that a Dutch-specific BERT, trained on multi-genre Dutch text, can outperform multilingual BERT on downstream tasks (Vries et al., 2019).

The model is monolingual rather than multilingual, and that design choice is central to its motivation. The underlying claim is not merely that Dutch is represented in multilingual BERT, but that representation quality depends on corpus scale, corpus composition, and language specialization. In that sense, BERTje belongs to the class of language-specific pretrained encoders whose performance gain derives from concentrating model capacity and pretraining signal on one language rather than distributing both across a multilingual mixture.

2. Pretraining corpus and objective design

BERTje was pretrained on a Dutch corpus assembled from several sources: books, including contemporary and historical fiction novels, with 4.4 GB; TwNC, a Dutch news corpus, with 2.4 GB; SoNaR-500, a multi-genre reference corpus, with 2.2 GB; web news from four Dutch news websites dated 2015-01-01 to 2019-10-01, with 1.6 GB; and the October 2019 Wikipedia dump, with 1.5 GB. After cleaning and filtering, the final pretraining dataset contained 12 GB of uncompressed text, corresponding to about 2.4 billion tokens (Vries et al., 2019).

The preprocessing pipeline was designed to reduce evaluation leakage. Chat and Twitter material were removed from SoNaR, Wikipedia documents already present in SoNaR were removed to prevent overlap with the full Wikipedia dump, and any SoNaR-500 documents appearing in the manually annotated SoNaR-1 and Lassy Small test datasets were excluded. For tokenization, the authors built a 30K WordPiece vocabulary using SentencePiece on the raw pretraining dataset and then translated it into WordPiece format for compatibility with BERT (Vries et al., 2019).

BERTje follows the BERT pretraining paradigm with two modifications. Next Sentence Prediction is replaced by Sentence Order Prediction, in which the second sentence is either the next or the previous sentence rather than a random sentence. Masked Language Modeling uses whole-word-style masking over WordPieces: 15% of all tokens are selected, of those 80% are replaced with a special mask token, 10% are replaced with a random token, and 10% are left unchanged. The model was pretrained for 1 million iterations, and the paper also evaluates a checkpoint at 850K iterations to study the effect of additional pretraining (Vries et al., 2019).

3. Benchmark profile on Dutch NLP tasks

The original evaluation compares multilingual BERT base, BERTje $_{850k}$ , and the final BERTje checkpoint under the same fine-tuning conditions for each task, namely four epochs with the same hyperparameters. The intent is to isolate the effect of the pretrained LLM rather than task-specific tuning. On CoNLL-2002 Dutch NER, multilingual BERT reached 80.7 test F1, BERTje $_{850k}$ 87.6, and BERTje 88.3. On SoNaR-1 NER, the corresponding scores were 79.7, 81.1, and 82.1. The paper notes that BERTje does not surpass the published state of the art of 90.9 F1 on CoNLL-2002, but attributes the remaining gap to task tuning rather than to limitations of Dutch-specific pretraining itself (Vries et al., 2019).

The gains extend beyond NER. On UD Lassy Small POS tagging, multilingual BERT scored 92.5, BERTje $_{850k}$ 96.6, and BERTje 96.3. On SoNaR-1 coarse POS, the scores were 98.3, 98.6, and 98.5; on SoNaR-1 fine-grained POS, 96.2, 96.6, and 96.8. On predicate-argument semantic role labeling, the scores were 80.4, 85.2, and 85.3; on SRL modifiers, 62.4, 67.3, and 67.2; and on spatio-temporal relations, 57.3, 62.5, and 64.3. On the 110k Dutch Book Reviews Dataset for sentiment analysis, multilingual BERT scored 89.1, BERTje $_{850k}$ 92.8, and BERTje 93.0 (Vries et al., 2019).

A recurring observation in the paper is that linguistic phenomena do not saturate uniformly during pretraining. POS tagging performance is already near its final level at 850K iterations, whereas NER and some higher-level tasks continue to improve with longer pretraining. This suggests that simpler syntactic information is learned earlier, while more abstract structure benefits from extended training. The result is a model whose empirical profile is especially strong on token-level and sentence-level Dutch NLP, but whose pretraining signal remains general-domain rather than domain-specific.

4. Role in Dutch HR de-identification

In Dutch HR de-identification, BERTje was not used as a standalone anonymizer. Instead, it was inserted into the modular TKS de-identification method as the NER component, replacing the older FROG-based recognizer and complementing baseline NER systems trained on the CoNLL 2002 and SoNaR corpora. The paper constructs four transformer-based NER models: CoNLL + BERT Multilingual cased, SoNaR + BERT Multilingual cased, CoNLL + BERTje, and SoNaR + BERTje. BERTje is described there as a Dutch pretrained BERT model, and evaluation retains only the entity classes relevant to the HR task: Person, Location, and Organisation (Toledo et al., 2021).

The HR evaluation uses a separate annotation set of 2,017 HR e-mails with 13,496 annotations and about 133 tokens per e-mail. For the NER task alone, CoNLL + BERTje yielded Person precision 0.72, recall 0.88, F1 0.80; Organisation precision 0.59, recall 0.69, F1 0.64; and Location precision 0.41, recall 0.65, F1 0.50. SoNaR + BERTje yielded Person precision 0.77, recall 0.83, F1 0.80; Organisation precision 0.63, recall 0.54, F1 0.58; and Location precision 0.34, recall 0.57, F1 0.42. On this comparison, CoNLL + BERTje is stronger on recall than SoNaR + BERTje for all three entity types (Toledo et al., 2021).

The paper distinguishes strict and loose de-identification. In the strict setting, a suppression is correct only if the entity is identified and labeled with the correct class; the paper states that if a person is identified as an organisation, the person identifier gets a false negative. In the loose setting, only the removal of the sensitive text span matters, regardless of the assigned class. This distinction is important because the authors treat recall as the most important metric for de-identification, since false negatives are riskier than false positives (Toledo et al., 2021).

The strongest de-identification setup reported in the abstract is TKS + CoNLL + BERTje, with recall 0.94 for suppressing persons and recall 0.82 for suppressing locations. In the strict or standard de-identification setting, TKS + CoNLL BERTje achieved recall 0.94 for Person, 0.78 for Organisation, 0.82 for Location, 0.97 for Date, 0.89 for Number, 0.16 for Gender, 0.48 for E-mail, 0.60 for Code, 0.18 for Title, 0.51 for Job title, and 0.00 for Website. The comparison with DEDUCE is complementary rather than absolute: DEDUCE performs best for suppressing gender, with recall 0.53 in the abstract and loose-evaluation gender recall 0.49, whereas BERTje-based TKS is better for persons, organisations, and locations (Toledo et al., 2021).

A common misconception is that the result implies a complete transformer replacement of rule-based de-identification. The paper does not support that interpretation. It explicitly shows a modular pipeline in which BERTje strengthens the NER backbone inside TKS, while rule-based resources remain useful for categories such as gendered prefixes and titles. It also states that no method achieves perfect anonymisation and that manual checking remains necessary. Annotator agreement for the HR annotation is reported with observed agreement $P_r(a)=0.99$ , expected agreement $P_r(e)=0.82$ , and resulting $\kappa = 0.92$ (Toledo et al., 2021).

5. Performance in violence risk assessment from Dutch clinical notes

In violence risk assessment for psychiatric admission periods, BERTje appears in a markedly less favorable role. The model, referred to in Hugging Face Transformers as BERT-base-dutch-cased, was used as the deep-learning alternative to conventional machine learning methods for document classification. A linear classification layer was added on top of the pretrained LLM and the system was fine-tuned on Dutch psychiatric admission-period notes from UMC Utrecht. The data consisted of 835k anonymized notes collected between 2012-08-01 and 2020-03-01, aggregated into 4280 admission periods from 2892 patients, with only 425 violent periods out of 4280. A period note concatenated all notes written from 28 days before to 1 day after the beginning of the admission period, and the outcome was positive if a violence incident occurred between day 1 and day 28 after admission began (Mosteiro et al., 2022).

The modeling constraints were substantial. BERTje could only accept sequences up to 512 tokens, so the notes had to be shortened either by truncating to the last 512 tokens or by summarizing the note with Gensim TextRank before tokenization; the summary had to remain under 512 words so that BERT tokenization would still fit the 512-token limit. Fine-tuning used the Hugging Face sequence-classification implementation with AdamW, learning rate $2\times10^{-5}$ , dropout 0.1, AdamW epsilon $10^{-8}$ , and a linear warm-up and decay schedule with warm-up lasting 10% of the training set. Under a nested 5-fold cross-validation scheme, the best epoch count was 1 for all BERTje configurations (Mosteiro et al., 2022).

The results were clearly below the conventional baselines. With summarization and 1 epoch, BERTje obtained AUC $_{850k}$ 0, AUPRC $_{850k}$ 1, and AUK $_{850k}$ 2. With truncation and 1 epoch, it obtained AUC $_{850k}$ 3, AUPRC $_{850k}$ 4, and AUK $_{850k}$ 5. Training for 2 epochs did not help. The best conventional results were around AUC 0.792, achieved by doc2vec+SVM and doc2vec+LDA+structured features+RF, with AUPRC around 0.315 and AUK around 0.136. BERTje also trailed the questionnaire-based Brøset Violence Checklist baseline in AUC, which reached $_{850k}$ 6 under a 00:00 assumption and $_{850k}$ 7 under a 23:59 assumption (Mosteiro et al., 2022).

The discussion attributes this underperformance to a poor fit between vanilla BERTje and the task conditions rather than to a general failure of transformer encoders. The main reasons offered are the 512-token limit and consequent information loss, many idiosyncratic or out-of-vocabulary forms in the clinical notes, overfitting after two epochs, label noise arising from potentially unreported violence incidents, and the small size of the labeled dataset for fine-tuning a large transformer model. The paper therefore suggests that a larger Dutch clinical corpus and possibly a “medical-BERTje” pretrained on Dutch clinical notes from multiple medical domains would likely be needed for the model family to realize its full potential in this setting (Mosteiro et al., 2022).

6. Benchmark role in Dutch clinical adverse drug event detection

A later Dutch clinical benchmark on adverse drug event detection again included BERTje, but under a different task decomposition and with more favorable results. In that study, BERTje was treated as a general-purpose monolingual Dutch BERT model with 108 million parameters and was benchmarked against RobBERT, MedRoBERTa.nl, NuNER-multilingual, and a Bi-LSTM baseline. It was fine-tuned for named entity recognition with a CRF layer using BIO tags B-Drug, I-Drug, B-Disorder, I-Disorder, and O. Training used 5-fold cross-validation, hyperparameter tuning on the first fold, up to 10 epochs, early stopping patience 3, learning rate 3e-5, max sequence length 512, batch size 8, warmup ratio 0.2, and a best-performing NER setup with cross-entropy loss and a CRF layer (Murphy et al., 25 Jul 2025).

The primary training and evaluation corpus comprised 102 richly annotated ICU clinical progress notes, all containing at least one drug-related acute kidney injury ADE. BERTje was evaluated on NER, relation classification, and document-level ADE detection. For relation classification, the study did not train a separate transformer for RC; instead, it reused embeddings from the NER model to build RC features and used a 4-layer MLP for prediction. The broader evaluation included internal document-level validation on an ICU AKI corpus of 411 ICU progress notes and external validation on the WINGS corpus of 100 discharge letters, 36 of which contained at least one annotated ADE (Murphy et al., 25 Jul 2025).

On NER, BERTje was competent but not leading. Under strict matching, overall precision was $_{850k}$ 8, recall $_{850k}$ 9, and F1 $_{850k}$ 0; drug F1 was $_{850k}$ 1 and disorder F1 $_{850k}$ 2. Under lenient matching, overall precision was $_{850k}$ 3, recall $_{850k}$ 4, and F1 $_{850k}$ 5. MedRoBERTa.nl outperformed all models on NER, and RobBERT and NuNER also ranked above BERTje overall (Murphy et al., 25 Jul 2025).

On relation classification, the paper stresses that micro-F1 is misleading because ADEs were only 0.64% of entity pairs, making macro-F1 the more realistic comparison. In the two-step task with gold-standard entities, BERTje obtained ADE micro-F1 $_{850k}$ 6 and macro-F1 $_{850k}$ 7, while Indication relation classification reached micro-F1 $_{850k}$ 8 and macro-F1 $_{850k}$ 9. In the end-to-end task with predicted entities, ADE micro-F1 was $_{850k}$ 0 and macro-F1 $_{850k}$ 1, while Indication relation classification reached micro-F1 $_{850k}$ 2 and macro-F1 $_{850k}$ 3. These results made BERTje competitive, but not the top-performing transformer (Murphy et al., 25 Jul 2025).

At document level, BERTje again remained useful but secondary. On the ICU AKI corpus, using the F1 threshold, precision was $_{850k}$ 4, recall $_{850k}$ 5, F1 $_{850k}$ 6, and specificity $_{850k}$ 7; using the F2 threshold, precision was $_{850k}$ 8, recall $_{850k}$ 9, F1 $_{850k}$ 0, and specificity $_{850k}$ 1. On the external WINGS corpus, using the F1 threshold, precision was $_{850k}$ 2, recall $_{850k}$ 3, F1 $_{850k}$ 4, and specificity $_{850k}$ 5; using the F2 threshold, precision was $_{850k}$ 6, recall $_{850k}$ 7, F1 $_{850k}$ 8, and specificity $_{850k}$ 9. The study’s overall conclusion is that BERTje is a reasonable general-purpose Dutch transformer baseline for clinical NLP, but clinically pretrained MedRoBERTa.nl is preferable when available (Murphy et al., 25 Jul 2025).

BERTje has also been evaluated as a Dutch masked LLM for stereotype preference measurement. In the Dutch CrowS-Pairs study, the benchmark was adapted from the English CrowS-Pairs dataset by translating the sentence pairs with Google Translate, having two native speakers review the translations, adjusting stereotypes and fluency, localizing U.S.-specific references to Dutch culture, and removing culturally irrelevant or incoherent items. The final Dutch dataset contains 1463 sentence pairs across nine bias categories: Race/Color, Gender, Nationality, Socioeconomic status, Religion, Age, Sexual orientation, Physical appearance, and Disability (Strazda et al., 22 Jul 2025).

The evaluation follows the standard pseudo-log-likelihood scoring method for masked LLMs. Each sentence is scored by masking one token at a time, computing the log-likelihood of the original token given the rest of the sentence, and averaging across tokens. A model is counted as biased on a pair if it assigns a higher score to the more stereotypical sentence. Under this protocol, 50% is neutral, values above 50% indicate measurable preference for the stereotypical sentence, and values below 50% indicate preference for the less stereotypical sentence (Strazda et al., 22 Jul 2025).

BERTje’s overall bias score is 54.82. In the cross-model comparison, RobBERT also scored 54.82, multilingual BERT 52.43, English BERT 61.45, RoBERTa 65.14, FlauBERT 55.02, and CamemBERT 58.30. The paper therefore characterizes Dutch models, including BERTje, as the least biased overall among the model groups tested, while also noting that multilingual BERT is even closer to neutrality than BERTje in this benchmark (Strazda et al., 22 Jul 2025).

The category-wise profile is uneven. BERTje scores 51.79 on Race/Color, 51.53 on Gender, 50.87 on Nationality, 50.88 on Socioeconomic status, 65.35 on Religion, 67.07 on Age, 53.85 on Sexual orientation, 68.25 on Physical appearance, and 68.97 on Disability. The model is thus near-neutral in Nationality and Socioeconomic status, but shows clear stereotypical preference in Disability, Physical appearance, Age, and Religion. Persona prompting was not applied to BERTje in this study, because that experiment was reserved for autoregressive LLMs rather than masked LLMs (Strazda et al., 22 Jul 2025).

The bias results qualify two opposite simplifications. They do not support the claim that BERTje is bias-free, since its overall score is above neutrality and several category scores are substantially above 50. At the same time, they do not support the claim that Dutch monolingual pretraining intrinsically produces strong bias relative to other language settings, since BERTje remains below the English and French baselines tested. The paper further suggests that differences in corpus scale and curation may matter, noting that RobBERT was trained on 39GB of Dutch OSCAR data while BERTje was trained on a smaller, more curated 12GB corpus (Strazda et al., 22 Jul 2025).