---
title: 'RobBERT: Dutch NLP Pre-trained Model'
url: https://www.emergentmind.com/topics/robbert
type: topic
---

# RobBERT: Dutch NLP Pre-trained Model

Searching arXiv for papers on RobBERT and closely related Dutch NLP work.
RobBERT is a Dutch monolingual pre-trained language model based on RoBERTa, introduced by Delobelle et al. as a Dutch alternative to multilingual BERT and earlier Dutch BERT variants. In its original form, it is a RoBERTa-base model with 12 transformer encoder layers, 12 attention heads, about 117M trainable parameters, masked language modeling as its only pretraining objective, and pretraining on the Dutch section of OSCAR. Subsequent work has treated RobBERT as a general-purpose Dutch base model, as a teacher for distillation, as a starting point for temporal and domain adaptation, and as a benchmark in downstream tasks and bias evaluation [2001.06286][2211.08192].

## 1. Origin, architecture, and pretraining design

RobBERT was developed from the premise that monolingual models usually outperform multilingual ones on a specific language, and that Dutch NLP would benefit from a strong monolingual model trained with RoBERTa’s pretraining regime rather than earlier BERT-style setups. The original training corpus was the Dutch OSCAR section, described as about 39GB, 6.6 billion words, and 126 million lines. Training used Adam, polynomial decay, an initial learning rate reported as \(l_r = 10^{-6}\), a warm-up of 1000 iterations, \(\beta_1 = 0.9\), \(\beta_2 = 0.98\), weight decay \(= 0.1\), dropout \(= 0.1\), a batch size fixed at 8192 sentences, 2 epochs, and over 16k batches; training took about 3 days on a cluster with 4 Nvidia P100 GPUs per node and up to 20 nodes or 80 GPUs [2001.06286].

A central design distinction in the original release is between RobBERT v1 and RobBERT v2. RobBERT v1 used RoBERTa’s original tokenizer, whereas RobBERT v2 used a Dutch byte-level BPE tokenizer trained on Dutch OSCAR text. The vocabulary of v2 was capped at 40k tokens, which is 10k fewer than v1 because many tokens in v1 were non-Dutch or Unicode artifacts from noisy OSCAR data. This tokenizer split is methodologically important because later evaluations repeatedly attribute part of RobBERT’s performance profile to tokenization quality [2001.06286].

The original paper also framed RobBERT as a RoBERTa-based Dutch BERT model rather than a novel architecture. It follows the RoBERTa recipe by omitting next sentence prediction and retaining masked language modeling only. In later derivative work, this architectural stability remained intact: CoNTACT is explicitly described as “RobBERT plus COVID-19 tweet pretraining,” not a new architecture, while RobBERT-2022 retains the base model and updates tokenizer and corpus coverage to address concept drift [2203.07362][2211.08192].

## 2. Benchmark performance and low-resource behavior

The original evaluation emphasized both standard Dutch benchmarks and the effect of fine-tuning data size. On sentiment analysis with DBRD, RobBERT v1 reached 86.73% accuracy on a 10k subset and 94.42% on the full dataset, while RobBERT v2 reached 94.38% on the 10k subset and 95.14% on the full dataset. The paper compared these results with prior figures of 93.8% for ULMFiT, 93.0% for BERTje, and 84.0% \(F_1\) for BERT-NL, and interpreted the v2 result as statistically significant relative to the prior state of the art [2001.06286].

On the die/dat disambiguation task proposed in the same paper, RobBERT produced especially strong results in both zero-shot and fine-tuned settings. Zero-shot evaluation gave 98.03% for RobBERT v1 and 98.75% for RobBERT v2, compared with 90.21% for mBERT and 94.94% for BERTje. Fine-tuned evaluation on Europarl yielded 98.406% for v1 and 99.232% for v2, compared with 98.285% for mBERT and 98.268% for BERTje. The paper states that the tokenizer roughly halves the error rate when fine-tuned on the full dataset [2001.06286].

Part-of-speech tagging and named entity recognition show a more qualified picture. On full-data POS tagging with Lassy UD, mBERT reached 96.5%, BERTje 96.3%, and both RobBERT variants 96.4%; the original study notes that mBERT was slightly higher, but not significantly so. On CoNLL-2002 Dutch NER, RobBERT v2 reached 89.08 \(F_1\), above BERTje at 88.3 and RobBERT v1 at 87.53, competitive with BERT-NL at 89.7, but below a previously reported mBERT result of 90.94 from another study. The strongest general claim made in the paper is therefore not universal dominance across all tasks, but especially strong behavior in smaller-data settings: on POS, RobBERT v2 is best when training data are 1,000 examples or fewer, and on sentiment the gap between v1 and v2 is especially large on the 10k subset [2001.06286].

## 3. Temporal updating, domain adaptation, and compression

Later work turned RobBERT into a platform for model maintenance and specialization rather than treating the 2020 release as static.

| Variant | Modification | Reported effect |
|---|---|---|
| RobBERT-2022 | Tokenizer update plus continued pretraining on newer Dutch OSCAR | Preserves most older-task performance and improves newer COVID-related tasks |
| CoNTACT | Continued pretraining of RobBERT on Dutch COVID-19 tweets | Statistically significant gains over RobBERT on vaccine hesitancy detection |
| RobBERTje | Distilled student models from RobBERT v2 | Lower parameter count, faster fine-tuning, reduced gender-stereotypical bias |

RobBERT-2022 addresses evolving language use and concept drift. Its tokenizer update starts from the original tokenizer \(T_O\), trains a new tokenizer \(T_N\) on the updated corpus, adds tokens that appear in \(T_N\) but not in \(T_O\), and updates the BPE merge structure accordingly to form a merged tokenizer \(T_M\). This yields a new vocabulary size of 42,774, with 2,774 tokens added. The updated model is then further pretrained on the latest Dutch OSCAR corpus, released with data up to January 2022, containing 77 GB and 122,364,485 documents. Training enlarges the embedding matrix for the 2,774 new tokens and continues with the same pretraining regime as original RobBERT and RoBERTa, using gradient accumulation over 128 batches, effective batch size 1024, one 3080 Ti GPU, ADAM, learning rate \(10^{-6}\), 1k batch warmup, and early stopping based on validation performance [2211.08192].

The evaluation criteria for RobBERT-2022 are framed as plug-in replacement criteria: minimal concept drift on existing tokens and useful alignment for novel tokens. On the original benchmark suite, RobBERT-2022 reports SA 95.1 versus 94.4 for RobBERT, CR 97.8 versus 99.2, NER 87.0 versus 89.1, POS 96.1 versus 96.4, and NLI 84.9 versus 84.2. Pseudo-perplexity on the original 2019 corpus is 7.76 for RobBERT and 9.40 for RobBERT-2022. The clearest gains appear on recent COVID-related tasks: on a COVID-19 tweet topic classification task, original RobBERT and multilingual BERT both score 73%, while RobBERT-2022 scores 75%; on VaccinChat, RobBERT v2 reaches ACC 74.9% and \(F_1\) 77.2%, while RobBERT-2022 reaches ACC 76.3% and \(F_1\) 79.3% [2211.08192].

CoNTACT is a domain-adapted derivative built by continuing the pre-training phase of RobBERT using 2.8M Dutch COVID-19 related tweets from 2021, totaling 66.8 million whitespace-separated tokens. The continued pretraining runs for four epochs with masked language modeling, default learning rate, batch size 32, and a 20% validation split, producing validation loss 1.702. The downstream tasks are binary vaccine hesitancy detection and argumentation detection for vaccine hesitancy, evaluated on both Twitter and Facebook. CoNTACT shows statistically significant gains over RobBERT in all stance-detection settings: for example, on same-genre Twitter evaluation RobBERT gets 75.1 \(F_1\) on the hesitant class and CoNTACT 77.1; on Facebook, RobBERT gets 62.7 and CoNTACT 69.3; in cross-genre Facebook-to-Twitter evaluation, RobBERT gets 65.1 and CoNTACT 76.3. On argumentation detection, micro-\(F_1\) rises from 64 to 67 on Twitter-Twitter and from 50 to 58 on Facebook-Facebook [2203.07362].

RobBERTje compresses RobBERT v2 into several student models. DistilBERT-style students have 74M parameters, while the Bort variant has 46M; the teacher RobBERT v2 has 116M parameters. The paper studies shuffled, non-shuffled, and merged transfer corpora, and compares DistilBERT-style students against Bort. DistilBERT-style students are generally better than Bort on all tasks, while Bort is much smaller and much faster; the paper gives one concrete speed comparison in which fine-tuning on SICK-NL is 4× faster than RobBERT and 2.2× faster than the merged-sequence distillation. Distilled models remain fairly close to teacher performance on several tasks, but are much worse than RobBERT on NER and PPPL [2204.13511].

## 4. Downstream deployments and empirical use cases

RobBERT has been used as a general-purpose Dutch encoder in multiple downstream studies. In Dutch humor detection, it served as the main transformer baseline against Naive Bayes, an LSTM, and a CNN. The study constructed three datasets: jokes versus news, jokes versus proverbs, and jokes versus generated non-jokes produced by the dynamic template algorithm. RobBERT achieved ACC 98.8 and \(F_1\) 98.8 on the news dataset, ACC 99.6 and \(F_1\) 99.6 on the proverbs dataset, ACC 89.2 and \(F_1\) 89.1 on the dynamic-template single-text setting, and ACC 82.51 with \(F_1\) 82.5 on the dynamic-template pairwise setting. The authors state that RobBERT was the only model that was able to distinguish jokes from generated negative examples, and interpret this as an advantage of pre-trained language models for detecting semantic coherence in jokes [2010.13652].

In punctuation restoration and segmentation prediction for Dutch ASR output, RobBERT was fine-tuned as a token-classification model that predicts the punctuation mark following each word. The system operates with a sliding window of 200 words and a thresholding scheme over overlapping windows, with the best threshold reported as \(\theta = 0.1\). A Dutch model based on BERTje was also trained, but RobBERT slightly outperformed BERTje by 0.75% \(F_1\). On out-of-domain segmentation with \(S=\{.\}\), the machine-translation baseline achieved \(F_1 = 0.4219\), while FullStop EP achieved 0.8221, FullStop SoNaR 0.8742, FullStop Multilingual 0.8314, and FullStop Multilingual EP+SoNaR 0.8424. With \(S=\{.,?\}\), the best result was the monolingual SoNaR RobBERT model at \(F_1 = 0.9053\) [2301.03319].

RobBERT has also been used as a baseline in psychiatric text classification. In the belabBERT study, RobBERT was compared with a newly introduced Dutch RoBERTa-based model on transcripts from semi-structured interviews with 339 participants. Under 220-token chunks, RobBERT reached 69.64% validation accuracy and 69.06% test accuracy; under 505-token chunks, it reached 68.93% validation accuracy and 65.69% test accuracy. belabBERT outperformed these results with 75.68% test accuracy at 220 tokens and 73.91% at 505 tokens. In that paper’s framing, RobBERT is described as using the RoBERTa base tokenizer and a shuffled and pre-cleaned Common Crawl Dutch corpus, and the authors argue that these choices may be suboptimal for Dutch psychiatric text and long-range semantic dependencies [2106.01091].

## 5. Fairness, stereotyping, and social-bias measurement

Fairness and stereotyping are persistent themes in the RobBERT literature. The original RobBERT paper evaluated gender stereotyping in occupations by prompting the masked language model to predict pronouns in occupational templates. It reports that in the control template there is no correlation with profession gender, but the model still ranks the male pronoun higher in all cases; when profession and pronoun refer to the same entity, rankings correlate somewhat with profession gender, yet the male pronoun remains more likely in almost all cases, with “actrice” as the notable exception. The same paper also analyzed unequal predictive performance by gender in sentiment analysis using demographic parity ratio and equal opportunity, reporting an EO difference of 0.028 and DPR of 70.2% for positive review prediction, and noting that fairness conclusions depend strongly on the metric [2001.06286].

RobBERTje extended this line by explicitly measuring gender bias using the log probability bias score from Kurita et al. The reported scores are 1.10 for the teacher RobBERT, \(-0.52\) for the non-shuffled student, \(-0.50\) for the shuffled student, \(-0.67\) for the merged student, and 0.04 for Bort. The paper interprets these results as showing that all distilled models reduce gender stereotyping relative to the teacher, and that all except Bort overcompensate, consistent with knowledge distillation acting as a form of regularization [2204.13511].

A broader bias profile appears in Dutch CrowS-Pairs, a Dutch adaptation of the CrowS-Pairs benchmark for measuring social bias in masked language models. The dataset contains 1463 sentence pairs across 9 categories: race/color, nationality, physical appearance, age, disability, gender, sexual orientation, socioeconomic status, and religion. For MLMs such as RobBERT, the evaluation uses pseudo-log-likelihood scoring of minimally contrasting sentence pairs and records whether the model prefers the more stereotypical sentence; a score of 50% indicates neutrality and scores above 50 indicate preference for stereotypical sentences. RobBERT’s overall bias score is 54.82%, identical to BERTje and above multilingual BERT at 52.43%, but below English BERT at 61.45% and RoBERTa at 65.14%. Category-wise, RobBERT scores 45.26 for race/color, 53.44 for gender, 55.49 for nationality, 60.23 for socioeconomic status, 67.33 for religion, 59.76 for age, 58.97 for sexual orientation, 60.32 for physical appearance, and 81.03 for disability. The paper identifies disability at 81.03% as the highest single category score reported for any model in the paper. It also notes that persona prompting is not reported for RobBERT, because persona prompting is used only for the autoregressive language models GEITje and Mistral-7B [2507.16442].

## 6. Position within Dutch NLP and recurring limitations

Across the cited work, RobBERT occupies a specific position in Dutch NLP: it is repeatedly used as the strong monolingual baseline against which newer Dutch models, update strategies, and specialized systems are evaluated. The original paper argues that RobBERT improves state-of-the-art results for various tasks and especially significantly outperforms other models when dealing with smaller datasets. Later papers retain that framing even when they surpass it in a specific domain: belabBERT evaluates itself against “the current Dutch state-of-the-art model RobBERT,” CoNTACT treats RobBERT as the starting point for domain adaptation, and RobBERT-2022 treats it as a reusable base model that can be updated rather than retrained from scratch [2001.06286][2106.01091][2203.07362][2211.08192].

At the same time, the literature does not present RobBERT as uniformly best in every setting. The original evaluation already notes that mBERT is slightly higher on POS, though not significantly so, and that RobBERT does not clearly beat the best previously reported mBERT result on NER. Dutch CrowS-Pairs finds multilingual BERT somewhat less biased overall than the monolingual Dutch models, though not consistently the lowest in every category. On VaccinChat, RobBERT-2022 improves over RobBERT but is still outperformed by the domain-adapted models BERTje+ and CoNTACT+ [2001.06286][2507.16442][2211.08192].

Several limitations recur. The original paper notes that RobBERT does not exploit newer pretraining objectives such as sentence order prediction, that OSCAR is shuffled and line-based, that Dutch compound words might benefit from morpheme-based tokenization, and that fairness mitigation was not incorporated during training. The belabBERT study argues that tokenizer choice and shuffled pretraining may reduce long-range semantic capture for conversational psychiatric text. The RobBERT-2022 paper shows that temporal updating is beneficial because language changes over time, particularly around events such as the COVID-19 pandemic. This suggests that RobBERT is best understood not as a fixed endpoint, but as a robust Dutch RoBERTa base model whose later significance lies equally in its reuse: as a benchmark, as a teacher, and as a substrate for continual adaptation under changing data distributions [2001.06286][2106.01091][2211.08192].

Source: https://www.emergentmind.com/topics/robbert