Dutch Retrieval Datasets Overview

• Dutch retrieval datasets are structured resources for IR and NLP in Dutch, featuring diverse annotation methods such as translation, manual labeling, and synthetic data generation.
• They integrate traditional and LLM-driven techniques to adapt underrepresented Dutch corpora for applications in legal, financial, and conversational domains.
• Robust evaluation using metrics like recall@k, nDCG, MAP, and MRR ensures high-fidelity benchmarking of retrieval systems tailored for domain-specific challenges.

Dutch retrieval datasets are structured resources created to enable the training, evaluation, and benchmarking of information retrieval (IR) systems and other NLP models for Dutch-language content. These datasets encompass a broad spectrum of domains, annotation styles, and task definitions, ranging from traditional keyword-based retrieval and dense ranking to specialized domains such as legal, financial, conversational, and embedding-focused retrieval. Historically, the Dutch language has been underrepresented in IR benchmarks compared to English, necessitating both the creation of original datasets and the adaptation of existing multilingual or translated resources. This has spurred systematic developments, including translation pipelines, domain-specific annotation efforts, and synthetic data generation using LLMs.

1. Dataset Construction: Sources, Domains, and Annotation Strategies

Dutch retrieval datasets derive from three principal approaches:

• Translation or adaptation of pre-existing benchmarks (e.g., BEIR-NL, SICK-NL, bBSARD, MTEB-NL) (Banar et al., 11 Dec 2024, Wijnholds et al., 2021, Lotfi et al., 10 Dec 2024, Banar et al., 15 Sep 2025)
• Manual or semi-automatic annotation of native Dutch corpora (e.g., PAL project children’s diary entries, Dutch restaurant SemEval reviews) (Haanstra et al., 2019)
• Synthetic data generation via LLMs (e.g., E5-NL task triplets, GEITje conversational corpora, financial instructions for FinGEITje) (Banar et al., 15 Sep 2025, Vanroy, 5 Dec 2024, Noels et al., 3 Oct 2024)

Primary sources used include Wikipedia dumps, government portals, literature (SoNaR-500, TwNC), legal statutes (Justel), news, question answering corpora, and crawled web data (mC4). Annotation strategies range from BIO token labeling for aspect/opinion extraction (Haanstra et al., 2019), sentence-level entailment pairs (Wijnholds et al., 2021), and typological semantic parses (Kogkalidis et al., 2019), to parallel alignment for translation-based datasets (Lotfi et al., 10 Dec 2024).

Table: Exemplary Dutch Retrieval Dataset Properties

Dataset	Annotation	Source Domain
BEIR-NL	Machine translation, IR labels	Biomedical, QA, Fact checking
bBSARD	Parallel (legal), manual check	Statutory law
SICK-NL	Manual translation, NLI labels	Image captions, NLI
FinGEITje	LLM synthetic, instruction labels	Financial news, tweets, tables
Children’s diary	Manual BIO, aspect/opinion	Diary entries (PAL)
MTEB-NL	Mix, IR/STS/classification labels	Multidomain

Annotation methodology critically affects downstream retrieval performance, domain adaptation, and robustness, with domain-specific fine-tuning (e.g., financial, legal) and careful translation quality control emerging as key technical challenges.

2. Model Evaluation Protocols and Metrics

Dutch retrieval datasets are evaluated using standard IR metrics—recall@k, nDCG@10, MAP, MRR—as well as tailored metrics where task designs require. For instance, BEIR-NL employs nDCG@10 and Recall@100 for dense ranking and reranking models (Banar et al., 11 Dec 2024), bBSARD evaluates recall, MAP/MRR, and nDCG for lexical, dense, and fine-tuned bi-encoder models (Lotfi et al., 10 Dec 2024). DUMB applies Relative Error Reduction (RER) to compare model improvements against baselines (Vries et al., 2023).

LaTeX formulas for metrics are explicitly reported in several works:

$\text{RER} = \frac{p_M - p_B}{1 - p_B}$

where $p_M$ is the model’s accuracy and $p_B$ the baseline (Vries et al., 2023). For contrastive retrieval learning:

$$L = -\log \left( \frac{\exp(\cos(Q_i, A_i)/\tau)}{\sum_{j=1}^B \exp(\cos(Q_i, A_j)/\tau)} \right)$$

with batch negatives and temperature $\tau$ (Lotfi et al., 10 Dec 2024).

These metrics facilitate robust cross-model and cross-dataset comparison, highlighting relative strengths of BM25, dense, cross-encoder, and language-specific encoder architectures.

3. Translation Methodologies and Their Impact

Translation-based construction techniques are prominent in Dutch retrieval dataset development. BEIR-NL was created by translating 14 English BEIR datasets using Gemini-1.5-flash, GPT-4o-mini, or Google Translate, with domain-specific prompts and manual native review yielding a 2.2% major error rate (Banar et al., 11 Dec 2024). SICK-NL used semi-automatic translation followed by manual alignment to preserve meaning and lexical coherence (Wijnholds et al., 2021). bBSARD legal questions were translated via GPT-4o (temperature 0) and subsequently checked by human annotators (Lotfi et al., 10 Dec 2024).

Translation artifacts can introduce semantic drift, inconsistencies, or domain-specific lexical mismatches. Back-translation experiments with BEIR-NL revealed a 1.9–2.6 point drop in nDCG@10 for mean model retrieval scores, signaling inherent limitations of relying strictly on automated translation for benchmark generation (Banar et al., 11 Dec 2024). This suggests that rigorous evaluation practices and additional native annotation are required to maintain retrieval fidelity.

4. Domain-Specific and Synthetic Data Resources

Recent advances have emphasized the creation of Dutch domain-specific retrieval sets. FinGEITje introduced a pipeline for generating and filtering over 147k Dutch financial instruction samples using LLM-based translation and de-duplication techniques (Noels et al., 3 Oct 2024). bBSARD offers the first statutory article retrieval dataset for Dutch, with legal questions and articles aligned at scale (Lotfi et al., 10 Dec 2024). MTEB-NL includes previously unseen retrieval datasets (ArguAna-NL, NFCorpus-NL, SCIDOCS-NL, SciFact-NL), selected to avoid overexposure during model fine-tuning (Banar et al., 15 Sep 2025).

Synthetic triplet generation using LLMs has expanded coverage for retrieval embeddings. E5-NL used synthetic data (approx. 350k hard negative triplets) layered with human-annotated retrieval sets and stringent filtering—via topic sampling and re-ranking constraints—to construct a robust Dutch embedding training corpus (Banar et al., 15 Sep 2025).

Such resources enable zero-shot, dense, and contrastive retrieval approaches that better represent Dutch linguistic peculiarities, domain context, and task diversity.

5. Current Benchmarks, Open Evaluation, and Model Trends

Multiple benchmarks and leaderboards now provide standardized, open resources for Dutch retrieval evaluation. DUMB covers nine tasks including QA, NLI, WSD, CR, and offers a leaderboard at dumbench.nl (Vries et al., 2023). BEIR-NL is released on the Hugging Face hub, facilitating rapid benchmarking of IR systems including BM25, multilingual-e5, LaBSE, BGE-M3, mContriever, and cross-encoder rerankers, as well as legal and financial tasks (Banar et al., 11 Dec 2024, Lotfi et al., 10 Dec 2024, Noels et al., 3 Oct 2024).

MTEB-NL, integrating legacy and curated Dutch datasets, supports zero-shot and fine-tuning evaluation for embedding models, with performance gains shown for E5-NL models employing vocabulary trimming and cross-tokenizer mapping (Banar et al., 15 Sep 2025).

Table: Domain-Specific Dutch Retrieval Benchmarks

Benchmark	Domains Covered	Notable Release
BEIR-NL	Biomedical, QA, IR	Hugging Face, translation pipeline
DUMB	NLI, QA, WSD, CR	dumbench.nl, RER metric
bBSARD	Legal statutes	Hugging Face, bilingual alignment
FinGEITje	Financial tasks	Financial QA, NER, HC, RE
MTEB-NL	Multitask, embedding	Open, legacy + new retrieval sets

State-of-the-art results have been achieved by large dense models (multilingual-e5, gte-multilingual-base, DeBERTaV3 variants) and fine-tuned language-specific models (RobBERT‑2023, Tik-to-Tok, E5-NL). Lexical baselines like BM25 continue to be highly competitive, especially when paired with reranking modules.

6. Challenges, Limitations, and Future Directions

Dutch retrieval datasets confront specific challenges:

• Translation artifacts and semantic drift can limit benchmark fidelity, even with high-quality machine translation and manual review (Banar et al., 11 Dec 2024).
• Domain-specific pretraining for underrepresented domains such as legal and financial texts requires sustained annotation and data acquisition efforts (Lotfi et al., 10 Dec 2024, Noels et al., 3 Oct 2024).
• Synthetic data generation leverages LLMs but may introduce bias or inadequate linguistic coverage if not stringently filtered (Banar et al., 15 Sep 2025, Vanroy, 5 Dec 2024).
• Benchmark overexposure remains a concern for realistic zero-shot evaluations, mandating dynamic integration of lesser-used resources (Banar et al., 15 Sep 2025).

Ongoing efforts to expand Dutch benchmarks cover instructional and conversational tasks (GEITje, Language Resources for Dutch LLMling), as well as fine-grained embedding-based evaluation (MTEB-NL, E5-NL) (Vanroy, 5 Dec 2024, Vanroy, 2023, Banar et al., 15 Sep 2025). The shift toward open leaderboards and modular evaluation pipelines (e.g., on Hugging Face and dumbench.nl) is accelerating reproducible research and enabling granular tracking of model trends across domains.

A plausible implication is that future Dutch retrieval datasets will require more dynamic updating, continued data acquisition from native sources, and multi-domain expansion including legal, financial, health, and conversational contexts. Addressing translation artifacts and refining synthetic data pipelines are expected to remain research priorities, alongside the development of domain-adaptive, efficiency-focused Dutch retrieval models that leverage emerging architectures and larger LLMs.

---

Dutch retrieval datasets have become increasingly sophisticated, spanning manual annotation, high-quality translation, domain specialization, and synthetic expansion. Through innovations in benchmark design, evaluation metrics, and modeling strategies, these resources now underpin a robust and fast-evolving ecosystem for Dutch IR and NLP, while also revealing active challenges and opportunities for methodological refinement and domain-specific adaptation.