Dutch CrowS-Pairs Dataset

• Dutch CrowS-Pairs is a benchmark that adapts the original CrowS-Pairs to the Dutch context with culturally relevant, minimally contrasting sentence pairs.
• It comprises 1,463 sentence pairs spanning nine bias categories, enabling detailed evaluation of social bias in language models.
• The dataset evaluation employs methods like pseudo-log-likelihood and prompt-based choice testing to reveal context-sensitive bias variations.

Searching arXiv for the original CrowS-Pairs dataset and French adaptation to support the article with current paper references. Dutch CrowS-Pairs is a Dutch adaptation of the original CrowS-Pairs challenge set for quantifying social biases in LLMs through minimally different sentence pairs. It localizes the benchmark to Dutch culture and language, because the original resource is English and US-specific and many stereotypes, group labels, and cultural referents do not transfer directly to the Netherlands. The resulting dataset contains 1,463 sentence pairs across nine bias categories, with each pair contrasting a sentence that targets a disadvantaged group against a corresponding sentence that targets an advantaged group in the Dutch context; a biased model is expected to prefer the stereotyped sentence (Strazda et al., 22 Jul 2025).

1. Origins and conceptual scope

Dutch CrowS-Pairs was introduced to measure social bias in Dutch LLMs under the same general paradigm as CrowS-Pairs: minimally contrasting sentences are used to reveal whether a model assigns higher plausibility to a stereotyped variant than to a less stereotyped or reversed-target variant. The benchmark preserves the original emphasis on direct pairwise comparison, but adapts the target groups, lexical choices, and cultural references to Dutch public discourse (Strazda et al., 22 Jul 2025).

The dataset covers the same nine demographic categories as the original challenge set: race/ethnicity (Race/Color), nationality, physical appearance, age, disability, gender, sexual orientation, socioeconomic status/occupation, and religion. In each pair, one sentence concerns a disadvantaged group and the other an advantaged group. Illustrative contrasts include “Marokkanen” versus “Belgen” for nationality, “moslim” versus “christelijke” for religion, “arme” versus “rijke” for socioeconomic status, and “mensen met autisme” versus “mensen zonder autisme” for disability. The paper presents these as bias-measurement stimuli and explicitly warns that the dataset contains offensive stereotypes (Strazda et al., 22 Jul 2025).

The underlying premise is that bias is culturally and linguistically situated. For Dutch, this means that a direct translation of US-centric examples is insufficient. The benchmark therefore operationalizes bias measurement not as a language-independent property of sentence templates, but as a context-dependent property of target-group contrasts embedded in Dutch sociocultural hierarchies. A plausible implication is that cross-lingual comparisons are meaningful only when the benchmark itself is culturally localized, rather than merely translated.

2. Construction, localization, and curation

The dataset was constructed through initial automatic translation with Google Translate, followed by human review by two native Dutch speakers. Their review focused on correctness, fluency, and especially the stereotyped trigger words. Localization involved replacing US-specific content with Dutch-relevant material, including changes such as “Mexican” to “Marokkaan” when appropriate, dollars to euros, pounds to kilograms, and names such as “Aaron” to “André” (Strazda et al., 22 Jul 2025).

Curation also included removal and repair. Forty-five pairs were removed for cultural irrelevance or incoherence, including Amish references and problematic constructions such as “As a Jew …”. The authors further applied systematic fixes for known CrowS-Pairs issues. Non-minimal pairs were rewritten so that only the target token differed. “Double switch” cases were repaired to avoid unintended meaning shifts. “Bias mismatch” cases were corrected so that both sentences belonged to the same category. Some items were recoded at the category level, for example changing “Italian” to “Catholic” to align with the religion category (Strazda et al., 22 Jul 2025).

The selection of disadvantaged and advantaged groups reflects Dutch sociocultural hierarchies recognized in public discourse and prior bias resources. Disadvantaged examples include Moroccan, Muslim, women, gay, seniors, overweight people, people with autism or other disabilities, and poor. Advantaged examples include White, Christian, men, straight, teenagers depending on context, thin, people without disabilities, and rich. Quality control was conducted by the two native speakers through discussion and cross-checking. No inter-annotator agreement figures are reported, and the authors note that some limitations inherited from CrowS-Pairs remain even after extensive repair (Strazda et al., 22 Jul 2025).

3. Category composition and representational structure

The 1,463 sentence pairs are unevenly distributed across the nine categories. This distribution inherits the original emphasis on race/ethnicity and gender, while disability and physical appearance remain relatively underrepresented (Strazda et al., 22 Jul 2025).

Category	Share of 1,463 pairs
Race/Color	32.47%
Gender	17.91%
Nationality	11.83%
Socioeconomic status	11.69%
Religion	6.90%
Age	5.61%
Sexual orientation	5.33%
Physical appearance	4.30%
Disability	3.96%

The dataset follows the CrowS-Pairs schema. Core fields include sent_more, the more stereotyping sentence in Dutch; sent_less, the less stereotyping sentence; and bias_type, one of the nine categories. Additional metadata may include source identifiers and notes about adaptations. The repository named “Dutch-CrowS-Pairs” hosts the data and the ready-to-use scripts used for masked-language-model and autoregressive-language-model evaluation; the paper advises consulting the repository for licensing terms, since licensing is not specified in the paper itself (Strazda et al., 22 Jul 2025).

This structure makes the benchmark suitable both for aggregate bias measurement and for category-wise analysis. At the same time, the category imbalance matters statistically: small categories such as disability yield wider uncertainty intervals, so category-level results require more caution than overall scores.

4. Evaluation protocol and scoring methodology

For masked LLMs, Dutch CrowS-Pairs uses pseudo-log-likelihood. For a tokenized sentence $x_{1:n}$, the paper gives

$$\mathrm{PLL}(x_{1:n}) = \sum_{i=1}^{n} \log p(x_i \mid x_{1:n} \setminus \{x_i\}).$$

Operationally, the Dutch CrowS-Pairs scoring follows Nangia et al. by conditioning on the modified tokens and masking one unmodified token at a time. The model’s own tokenizer is used, including WordPiece or BPE segmentation. If a content word splits into multiple subwords, the masking step is applied per subword and the log-probabilities are summed. For each pair, the model is counted as preferring the stereotyped sentence when $$\mathrm{score}(\text{stereotype}) > \mathrm{score}(\text{anti-stereotype})$$. The aggregate metric is

$$\mathrm{BiasScore}(\%) = 100 \times \frac{N_{\text{more}}}{N_{\text{total}}},$$

with 50% corresponding to stereotype-neutral behavior (Strazda et al., 22 Jul 2025).

For autoregressive models, the paper notes the standard left-to-right likelihood

$$\mathrm{LL}(x_{1:n}) = \sum_{i=1}^{n} \log p(x_i \mid x_{1:i-1}),$$

but in practice adopts a prompt-based two-option choice task following the BLOOM evaluation paradigm. The baseline Dutch prompt asks the model to choose which sentence is “waarschijnlijker,” with Option 1 corresponding to the more stereotyped sentence and Option 2 to the less stereotyped sentence. Two persona variants are also used: a “bad, mean person” instruction and a “good, kind person” instruction, both prepended to the same two-option question. Bias is again reported as the percentage of pairs for which the model selects the more stereotyped sentence (Strazda et al., 22 Jul 2025).

The paper also outlines recommended statistical practice beyond its reported point estimates. For overall and per-category proportions, it recommends testing against the null $p = 0.5$ with a two-sided binomial test, reporting 95% confidence intervals using Wilson or Clopper–Pearson intervals, and using bootstrap resampling over sentence pairs to quantify uncertainty. This recommendation is especially relevant for smaller categories, where point estimates are less stable (Strazda et al., 22 Jul 2025).

5. Empirical findings across model families

The paper evaluates masked LLMs in Dutch, English, and French. For Dutch, BERTje and RobBERT each obtain an overall bias score of 54.82, while multilingual BERT reaches 52.43. For English, BERT scores 61.45 and RoBERTa 65.14. For French, FlauBERT scores 55.02 and CamemBERT 58.30 (Strazda et al., 22 Jul 2025).

Language	Model	Overall bias score
Dutch	BERTje (Base)	54.82
Dutch	RobBERT (Base)	54.82
Dutch	multilingual BERT (Base)	52.43
English	BERT (Base)	61.45
English	RoBERTa (Large)	65.14
French	FlauBERT (Base)	55.02
French	CamemBERT (Base, RoBERTa-style)	58.30

Several comparative patterns are explicit. English models exhibit the most bias, French models are intermediate, and Dutch models the least. Within English and French, RoBERTa-style architectures show more bias than BERT-style architectures. High-bias categories across masked models include religion, disability, physical appearance, and socioeconomic status, whereas race/color and gender are often lower but still above 50%. The highest single-category score observed is RobBERT’s 81.03 on disability. Other category maxima reported across all masked models include RoBERTa 74.39 on age, RoBERTa 74.60 on physical appearance, BERT 74.75 on religion, RoBERTa 68.64 on socioeconomic status, CamemBERT 68.25 on nationality, RoBERTa 62.65 on race/color, BERT 60.23 on gender, and BERT 68.75 on sexual orientation, with RoBERTa close behind at 65.00. Within Dutch models specifically, RobBERT exceeds BERTje on gender, socioeconomic status, and disability, whereas BERTje exceeds RobBERT on race/color, age, and physical appearance; multilingual BERT trends lower overall and is counter-stereotypical in some categories, including nationality and socioeconomic status (Strazda et al., 22 Jul 2025).

For autoregressive models, the paper evaluates GEITje and Mistral-7B using the Dutch choice-prompt protocol.

Model	Prompt condition	Overall bias score
GEITje	Baseline	85.03
Mistral-7B	Baseline	59.67
GEITje	Bad persona	90.98
Mistral-7B	Bad persona	94.46
GEITje	Good persona	53.11
Mistral-7B	Good persona	22.21

Persona effects are large. Under the “bad persona,” both models show markedly higher preference for stereotyped sentences; category-level standouts include sexual orientation at 96.15 for GEITje and 98.72 for Mistral, socioeconomic status at 90.64 for GEITje and 95.91 for Mistral, and religion at 91.09 for GEITje and 97.03 for Mistral. Under the “good persona,” measured bias drops sharply and in some cases reverses below 50%, most notably for Mistral at 22.21 overall. The paper interprets this as evidence that bias expression in generative models is highly context- and instruction-sensitive, while also cautioning that prompt-based mitigation is brittle and does not remove underlying associations (Strazda et al., 22 Jul 2025).

A common misconception is that the comparatively lower Dutch masked-language-model scores imply absence of bias. The reported values do not support that interpretation: Dutch models are lower than the English and French models in this comparison, but they remain above the 50% stereotype-neutral point.

6. Relation to multilingual debiasing research

Before the release of Dutch CrowS-Pairs as a dedicated Dutch benchmark, multilingual work had already used Dutch translations of CrowS-Pairs in a narrower evaluation setting. One such study restricted the task to gender, race/ethnicity, and religion; randomly sampled $N \in \{20, 30, 40, 50\}$ items; selected $N = 40$ because this yielded more than 75% correlation with the full dataset for both BERT and mBERT; and evaluated three random seeds per language, so the Dutch evaluation set comprised three samples of 40 Dutch sentence pairs, or 120 pairs in total (Reusens et al., 2023).

That earlier study translated the sampled English items into Dutch and did not describe a dedicated human verification protocol for the sentence-pair translations. It reported mBERT base (uncased, 12 layers, hidden size 768) results using PLL-based CrowS-Pairs scoring implemented as in Meade et al. Its Dutch baseline bias rate was 67.66%, with category-level values of 56.32% for gender, 65.83% for race, and 80.83% for religion. Across debiasing configurations, the strongest Dutch improvement reported was obtained by INLP debiased in German and evaluated in Dutch, reducing the overall Dutch bias rate to 61.16%. SentenceDebias was reported as the most consistent technique across languages, reducing bias in mBERT on average by about 13% (Reusens et al., 2023).

These results are not directly comparable to those of Dutch CrowS-Pairs in its 2025 form, because the earlier study used translated subsets restricted to three categories, whereas Dutch CrowS-Pairs contains 1,463 culturally adapted pairs across nine categories. The contrast is nonetheless informative. The multilingual study explicitly identified the lack of a comprehensive, peer-reviewed Dutch CrowS-Pairs as a limitation and recommended developing a native Dutch bias benchmark that captures Dutch-relevant stereotypes and linguistic phenomena. Dutch CrowS-Pairs occupies precisely that benchmark role (Reusens et al., 2023).

7. Limitations, interpretation, and ethical handling

The benchmark has several explicit limitations. Coverage is imbalanced: race/color and gender dominate the dataset, while disability and physical appearance have fewer items. Cultural specificity is also central. Choices such as “Marokkanen,” “moslim,” “rijk/arm,” and “blank/christelijk” are Dutch-contextual and may not generalize to other Dutch-speaking communities or to different time periods. Even after repair, some minimal-pair constraints and category assignments remain debatable, and the authors acknowledge inherited design issues from CrowS-Pairs. Quality control relied on two native speakers, but no inter-annotator agreement is reported (Strazda et al., 22 Jul 2025).

Methodological limitations also affect interpretation. PLL is an approximation and is sensitive to tokenization. For autoregressive models, two-option prompting conflates instruction-following with underlying likelihoods. Prompt context can substantially modulate results, as shown by the persona experiments, but the paper explicitly cautions that this does not amount to removal of underlying associations. A plausible implication is that benchmark scores should be read as measurements of elicited behavior under a specific protocol, not as exhaustive characterizations of representational content.

The paper therefore recommends careful reporting: overall and per-category scores with 95% confidence intervals and category counts, sensitivity checks for casing and punctuation, and robust parsing and deterministic decoding where possible for autoregressive evaluation. It also recommends caution with tokenization mismatches, subword handling, and category imbalance (Strazda et al., 22 Jul 2025).

Ethically, the dataset contains explicit offensive stereotypes and may be distressing. Its intended use is research on bias measurement and mitigation. The paper recommends warning annotators and users beforehand, restricting access when appropriate, avoiding deployment of examples or derived generations to end users, using outputs strictly for analysis, removing them from logs when no longer needed, and following institutional ethics guidance. Within those constraints, Dutch CrowS-Pairs functions as a culturally localized instrument for systematic bias auditing of Dutch LLMs and for comparative work on multilingual bias and debiasing (Strazda et al., 22 Jul 2025).