Dynaword Approach for Reproducible NLP Corpora
- Dynaword approach is a framework for constructing NLP pretraining corpora that emphasizes continuous updates, open licensing, and reproducible curation.
- It enforces legal traceability, comprehensive documentation, and community-driven extensibility to overcome limitations of static, one-shot releases.
- Empirical validation using Danish Dynaword demonstrates measurable improvements in language model performance versus traditional datasets.
The Dynaword approach is a framework for constructing NLP pretraining corpora as continuously developed, openly licensed, reproducible, documented, and extensible datasets rather than as one-shot static releases. It was introduced together with Danish Dynaword, a concrete implementation described as the largest openly licensed Danish corpus, and is motivated by three recurring weaknesses in prior large-scale dataset practice: ambiguous licensing, static releases, and quality assurance restricted to the original publishing team (Enevoldsen et al., 4 Aug 2025).
1. Conceptual basis
Dynaword begins from a critique of prevailing corpus publication norms. The framework argues that many technically useful datasets are weak as long-term public infrastructure because they may be downloadable without clearly permitting redistribution or derivative works, they are commonly released once without an update path, and their quality assurance remains centralized in the originating group. In this formulation, the central problem is not only dataset size or model utility, but also whether a corpus can be legally reused, maintained after publication, audited, corrected, and extended over time (Enevoldsen et al., 4 Aug 2025).
A key distinction is drawn between three openness tiers: replicable, which enables reproduction; open access, which enables download; and openly licensed, which enables “resharing, reuse, and modification.” This distinction is operational rather than rhetorical. The framework stresses that in some releases the published license may apply only to “the packaging, metadata, and annotations, but not to the data itself,” which creates a mismatch between technical availability and legally robust reuse. Dynaword therefore elevates licensing provenance to a first-class design constraint rather than treating it as secondary documentation (Enevoldsen et al., 4 Aug 2025).
The approach also treats dataset stasis as a technical liability. Static releases cannot easily absorb newly available data, improvements in OCR or extraction, bug fixes, source removals prompted by legal concerns, or revised cleaning methods. This suggests that Dynaword redefines the dataset from a finished artifact into a maintained resource whose composition can change transparently through versioned updates.
2. Core principles
The framework is organized around four explicit principles.
| Principle | Stated requirement | Immediate implication |
|---|---|---|
| “Traceable and open licensing” | All component datasets “must be openly licensed and maintain a traceable license” | Redistribution and derivative use have a documented basis |
| “Reproducibility” | “It should be possible to derive a substantially similar dataset” | Collection and processing must be reconstructible |
| “Documented” | The dataset should be documented under best practice, specifically citing Datasheets for Datasets | Source provenance and curation decisions are inspectable |
| “Extensible” | It should be possible to improve and expand the corpus, and the process for doing so should itself be documented | Community updates become part of the method |
These principles are narrower than a general claim of openness. “Traceable and open licensing” requires not just permissive labeling, but source-level license references and review of whether the source owner actually has the right to license the content. “Reproducibility” is defined as the ability to derive a “substantially similar dataset,” with the paper explicitly linking this to the notion of a “substantially equivalent system.” “Documented” is tied to datasheet-style reporting, and “Extensible” requires that expansion itself be proceduralized rather than left informal (Enevoldsen et al., 4 Aug 2025).
The paper explicitly compares this aspiration to continuously developed open-source software and community datasets such as NumPy and Universal Dependencies. This suggests that the framework borrows governance intuitions from software maintenance: versioning, changelogs, contribution pathways, and reviewable updates become part of corpus design rather than post-publication conveniences.
3. Workflow, curation, and governance
The operational workflow is source-granular. Potentially suitable datasets are identified, their license and content are reviewed, the data are collected, light quality checks are run, datasheets are produced, and sources are either included or rejected with reasons recorded. Source discovery for Danish Dynaword drew on the Danish Foundation Models project, the Hugging Face Hub, Sprogteknologi.dk, social media outreach, personal communication, and issues opened in the Danish Dynaword repository (Enevoldsen et al., 4 Aug 2025).
The inclusion screen is intentionally conservative. Straightforward exclusions occur when data are not Danish, not openly licensed, or when “the redistributor lacks permission to license or re-license it.” For more difficult cases, maintainers “may request legal advice from faculty services.” Quality assurance is described as minimal and conservative because the goal is a reusable infrastructure layer rather than a heavily opinionated filtered corpus. The checks include verifying that text is Danish, coherent, and readable; all sources are deduplicated and short documents are removed; OCR-derived sources receive additional quality inspection (Enevoldsen et al., 4 Aug 2025).
A concrete rejection example illustrates the process. The Danish subsection of Common Corpus was excluded because OCR quality was insufficient, with “alpha ratio generally below 0.7,” and most text unreadable. Rejections are documented and the issue is closed. This matters because Dynaword’s governance model does not merely accept contributions; it records exclusion rationales as part of corpus provenance.
Community collaboration is part of the method rather than an auxiliary feature. The paper reports contributions “from companies, government institutions, individuals and universities” and from people spanning “cultural heritage to NLP.” Review remains maintainer-led, but the governance mechanisms are concrete: explicit inclusion principles, source-level documentation, maintainers’ licensing review, escalation for uncertain legal cases, repository-mediated discussion, versioned releases, changelog transparency, and “light-weight tests” for data formatting, data quality, and documentation quality (Enevoldsen et al., 4 Aug 2025).
4. Danish Dynaword as the reference implementation
Danish Dynaword is the concrete validation case for the framework. It begins from the public segments of Danish Gigaword but excludes components that violate Dynaword principles. The paper lists four such exclusions: Twitter/social media data at approximately 32M tokens, copyrighted samples from OpenSubtitles at less than 1M tokens, Common Crawl segments at approximately 100M tokens, and DanAvis at approximately 30M tokens because of lack of coherence due to scrambling (Enevoldsen et al., 4 Aug 2025).
The resulting corpus spans many domains, including legal text, social media/forum material, spoken or transcribed resources, public-sector web text, medical sources, encyclopedic and parallel corpora, books and novels, news, dialect resources, and other specialized collections. Its total size in Appendix Table $\ref{tab:dataset_overview}$ is 4.80B tokens, measured in Llama 3 tokens. Some of the largest listed sources are Cellar at 1.15B, retsinformation.dk at 818.25M, NCC Books at 531.97M, Heste-nettet.dk at 389.32M, NCC Parliament at 338.87M, and OpenSubtitles at 271.60M (Enevoldsen et al., 4 Aug 2025).
The paper emphasizes that this scale is paired with an unusual openness profile in the Danish setting.
| Dataset | Tokens | Noted properties |
|---|---|---|
| Danish Dynaword (v1.2.7) | 4.8B | Contributions Yes; replicable, open access, openly licensed |
| Danish Gigaword | ~1B | Contributions No; open access, openly licensed; not replicable |
| Common Corpus (dan) | ~0.3B | Contributions No; open access, openly licensed; not replicable |
| SnakModel | ~13.6B | Replicable and open access; not openly licensed |
| Fineweb (dan) | ~26B | Replicable and open access; not openly licensed |
This comparison underwrites the paper’s main infrastructural claim: Dynaword is not the largest Danish corpus overall, but it is the largest openly licensed Danish corpus and uniquely combines openness with reproducibility and a contribution process. The authors also state that Danish Dynaword contains “over four times as many tokens as comparable prior openly licensed Danish releases” (Enevoldsen et al., 4 Aug 2025).
5. Empirical validation
The framework is validated not only by corpus assembly but also by language-model utility. Section “Training Experiments” compares Danish Dynaword and Danish Gigaword using Gemma-1B models, either continually pre-trained from gemma-3-1b-pt or trained from scratch. The stated setup uses max sequence length 6144, effective batch size 32, learning rate for pretrained models, learning rate for scratch models, a cosine scheduler in both cases, Danish Dynaword v1.2.0, and code in a GitHub repository with commit 76b546e (Enevoldsen et al., 4 Aug 2025).
Held-out validation datasets are DDT, JVJ, Synnejysk.dk, and Nordjyllands News, with additional contemporary evaluation on DR news articles and Danish Wikipedia articles published after January 1, 2025. The paper reports that, relative to the Gigaword baseline, Dynaword yields an average relative improvement of 5.9% for continual pre-training and 26% improvement for from-scratch training. Even under size-matched comparison, the reported gains are 2.6% for continual pre-training and 18% for scratch training (Enevoldsen et al., 4 Aug 2025).
The paper also reports downstream evaluation in the appendix: continual pretraining on Danish Dynaword improves performance on 7 out of 9 Danish EuroEval tasks. Examples against the Gigaword continual-pretraining baseline include angry-tweets sentiment (38.80 vs 36.58), dansk NER (17.97 vs 15.19), da-talemaader knowledge (35.31 vs 25.47), and hellaswag-da (26.84 vs 24.77). Because some tasks do not improve, the paper presents the downstream picture as mixed rather than uniform (Enevoldsen et al., 4 Aug 2025).
These results are significant within the paper’s own argument because they support two claims simultaneously: first, that a corpus constructed under stricter licensing and governance constraints can still be competitively useful for language modeling, and second, that the benefits are not reducible only to added size, since the size-matched setting also improves.
6. Policies, limitations, and subsequent use
The framework includes explicit policies for derived data and benchmark contamination. The current stated policy is that Dynaword does not, to the authors’ knowledge, contain synthetic data, machine-translated data, or automatically transcribed data. It does include human-annotated audio transcriptions, translations by expert translators, and OCR’d documents. The repository also marks datasets contained in benchmarks so that model developers can exclude them during training; listed Danish Dynaword examples include the Danish dependency treebank, with annotation excluded, and Nordjyllands News (Enevoldsen et al., 4 Aug 2025).
The paper is also explicit about trade-offs. Dynaword remains “an order of magnitude smaller than non-openly licensed sources,” especially Common Crawl-derived corpora such as FineWeb and OSCAR. Because only clearly licensed material is admissible, the corpus is biased toward domains with transparent rights, especially legal and public-sector text, and has limited social media coverage. Continuous maintenance creates ongoing review, legal, and repository burdens. The authors also state that reviewing small data changes proved difficult, that there is limited support for reviewing large data diffs, and that dataset poisoning is an open challenge (Enevoldsen et al., 4 Aug 2025). A plausible implication is that the framework exchanges maximal scale for stronger provenance, auditability, and maintenance structure.
Subsequent work uses Danish Dynaword not merely as a released corpus but as an indexed and operational dataset. In a propensity-aware memorization study, Dynaword appears as the Danish corpus used both for model training and as the reference corpus for tracing generations, with reported corpus-scale statistics of 5.66M samples, 6.83B Llama 3 tokens, and 10.5 GB. In that study, an infini-gram index is built over Dynaword, and SimpleTrace validation on Dynaword achieves perfect retrieval and exact-match results on the reported validation protocol (Barmina et al., 4 Jun 2026). This suggests that the Dynaword framework’s emphasis on openly licensed, inspectable, versioned corpora also enables downstream auditing workflows that depend on full corpus access.
In sum, the Dynaword approach defines dataset construction as an ongoing infrastructural process governed by source-level licensing traceability, reproducible collection, datasheet-style documentation, extensibility, and lightweight automated checks. Its Danish implementation demonstrates that such a process can produce a 4.8B-token openly licensed corpus with cross-sector contributions and measurable language-model utility, while preserving explicit acknowledgement of domain skew, maintenance costs, and scale limits relative to less legally constrained web-scale alternatives (Enevoldsen et al., 4 Aug 2025).