German Commons: Open German Text Corpus

• German Commons is a large, openly licensed German text corpus with over 154 billion tokens from nearly 36 million documents across seven domains.
• It employs a rigorous pipeline featuring quality filtering, deduplication, and PII sanitization to ensure high-quality, reproducible data for LLM training.
• The corpus supports transparent expansion and cross-domain analysis, making it a crucial resource for robust German language model development and historical studies.

The German Commons is the largest collection of openly licensed German-language text to date, purpose-built to facilitate the training of LLMs with full legal clarity. Comprising over 154 billion tokens and nearly 36 million documents, the corpus consolidates data from 41 established providers across seven thematic domains: web, political, legal, news, economic, cultural, and scientific texts. Each document is stringently verified for openness, with licenses of at least Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA 4.0) or equivalent, guaranteeing that derivative models and redistributions remain within well-defined legal standards. The German Commons incorporates a sophisticated data processing pipeline with quality filtering, deduplication, formatting normalization, and personal data removal, ensuring both consistency and compliance. Accompanying code and resources enable reproducible construction and extensibility, thereby establishing a new ecosystem standard for robust, transparent German LLM pretraining data (Gienapp et al., 15 Oct 2025).

1. Corpus Architecture and Domain Coverage

The German Commons corpus contains 35.78 million documents amounting to 154.56 billion tokens, processed using the GPT-2 tokenizer. The data is partitioned into seven thematic domains, each systematically sourced and audited for license clarity. The distribution is summarized in the table below.

Domain	Documents (M)	Tokens (B)	Representative Sources
Web Commons	15.48 (43.3%)	19.89 (12.9%)	Wikipedia (DWDS TEI), Wikivoyage, Wikipedia Discussions, YouTube-Commons subtitles, One Million Posts, The Stack
News Commons	13.27 (37.1%)	72.67 (47.0%)	Deutsches Zeitungsportal, Europeana Newspapers, ANNO, Wikinews
Cultural Commons	6.11 (17.1%)	54.49 (35.3%)	DiBiLit/DiBiPhil, Wikisource, German-PD, BLBooks, MOSEL, SBB Fulltexts, Wikiquote
Political Commons	0.26 (0.7%)	3.57 (2.3%)	Reichstagsprotokolle, Bundestags-Drucksachen, Plenarprotokolle, political speeches, EuroVoc
Legal Commons	0.51 (1.4%)	2.99 (1.9%)	Bundesrecht (CC0), OpenLegalData, BFH, BGH, BVerfG, EUR-Lex
Economics Commons	0.057 (0.16%)	0.11 (0.07%)	TEDEUTenders (procurement notices)
Scientific Commons	0.094 (0.26%)	0.84 (0.54%)	Wikibooks, Polytechnisches Journal, DOAB, arXiv, Wikiversity, OpenAlex

This architecture enables both domain specialization and cross-domain robustness, supporting nuanced modeling of legal, political, journalistic, technical, and cultural registers (Gienapp et al., 15 Oct 2025).

2. Licensing Framework and Legal Compliance

A strict open-licensing policy governs corpus composition. Requirements mandate a minimum of CC-BY-SA 4.0 or equivalent, including EUPL 1.2, Artistic 2.0, and public-domain equivalents (CC0, Unlicense, MIT-0, 0BSD), as well as attribution licenses (MIT, BSD-variants, Apache 2.0, CC-BY-2.0/3.0/4.0). Non-commercial, research-only, or use-limiting licenses are categorically excluded.

Each document is tagged with its SPDX-canonical license URL, providing unambiguous license provenance. The verification process relies exclusively on provider-supplied metadata from trusted institutional sources (national libraries, GLAM institutions, Wikimedia, Zenodo, Hugging Face), mapping all identifiers to recognized SPDX standards. Any document lacking an accepted license is discarded at ingest. Copyleft licenses (like share-alike) ensure that downstream uses remain compliant, enabling open redistribution and derivative use for both research and commercial purposes (Gienapp et al., 15 Oct 2025).

3. Data Processing and Quality Control Pipeline

The corpus is constructed via the “llmdata” library, orchestrating a pipeline with the following key stages:

3.1 Text Extraction and Formatting:

PDFs are processed via Grobid (scholarly) or OlmOCR (general), while TEI and wiki-marked texts have editorial, pagination, and bibliography metadata stripped. Markdown remains intact. Canonicalization includes UTF-8 encoding fixes, Unicode NFC normalization, ligature decomposition, whitespace normalization, and correction of hyphenation artifacts.

3.2 Language and Length Filtering:

Paragraphs are analyzed using FastText’s 176-language classifier, ensuring only German text (with probability $p ≥ 0.65$) progresses. Sequences shorter than 32 GPT-2 tokens are excluded.

3.3 Quality Filtering (Percentile Heuristics):

On a per-document basis, a series of heuristics reject outliers in the 5th–95th percentile for measures such as alphabetic-word ratio, bullet-line density, ellipsis ratio, hash-char density, stopword count, repeated n-gram proportion, top 4-gram fraction, and average word length. Additional OCR-specific filters target case and punctuation anomalies.

3.4 Deduplication:

Paragraph-level deduplication is accomplished with Dolma’s LSH-based Bloom filter (shingle size: 20-grams; duplicate threshold: 80% overlap; false positive rate: $10^{-4}$). The Jaccard similarity $$J(A,B) = \frac{|shingles(A) \cap shingles(B)|}{|shingles(A) \cup shingles(B)|} ≥ 0.8$$ marks duplicates for exclusion.

3.5 PII Sanitization:

Regexes and Microsoft Presidio identify emails, phone/IP/credit-card/IBAN numbers, and URLs, replacing detected spans with non-sensitive placeholders (e.g., “[email protected]”).

3.6 License Mapping and Final Compliance:

Every document’s license is mapped to an SPDX URL, and only those with every license meeting the open list are retained. These procedures collectively ensure consistent, legally reusable, noise-reduced data across domains (Gienapp et al., 15 Oct 2025).

4. Reproducibility and Community Extensibility

The German Commons is engineered for full transparency and ongoing extension. All pipeline code (“llmdata”), assembly scripts, and schema are available in public repositories. The final corpus is distributed in Parquet format, partitioned by domain and source, facilitating targeted sampling.

The paragraph-level deduplication Bloom filter is published alongside the data, enabling external users to LSH-dedupe novel contributions against the main corpus. Addition of new sources is governed by explicit guidelines: confirm open-license metadata, develop or specify a compatible extractor, process via the standard pipeline, and contribute via a source/auth table pull request. This infrastructure supports continuous corpus growth while enforcing methodological consistency (Gienapp et al., 15 Oct 2025).

5. Impact for German LLM Development

The German Commons directly addresses the critical deficit of fully open, high-quality pretraining text for German LLM research. Prior to its release, most German corpora were either web-derived with ambiguous licensing or comprised small, fragmented, and legally opaque datasets. The corpus allows open-licensed model weights and pretraining recipes, supporting unencumbered redistribution and commercial use.

Rich cross-domain diversity (legal, news, politics, science, economics, culture, web) enables development and evaluation of models robust across stylistic, lexical, and topical registers. The intensive filtering and deduplication procedures significantly elevate the signal-to-noise ratio of model training data. Built-in capabilities for sub-sampling by domain or document length (as in Table 7) permit controlled experimentation on, for instance, domain adaptation or context length sensitivity.

By open-sourcing the construction code, schemas, and deduplication resources, the German Commons establishes a methodological and legal standard for future multilingual and monolingual LLM data collection. It enables reproducible pretraining pipelines and transparent expansion, reducing the technical and legal barriers to high-quality, open model releases for the German language (Gienapp et al., 15 Oct 2025).

6. Historical and Sociolinguistic Context: The Parliamentary Subset

Within the German Commons “Political Commons” partition, the lower house parliamentary texts (Reichstag, Bundestag) afford a high-resolution semantic record of German ideological currents from 1867 to 2020 (Walter et al., 2021). The DeuPARL corpus, a major component, encompasses over 9,700 parliamentary sessions and nine historical slices, with protocols aligned to party dominance and regime changes.

Diachronic word embeddings (skip-gram, $d=200$, window 5), trained and aligned via orthogonal Procrustes, reveal that antisemitic bias peaks in the Weimar period and collapses post-1945, while anti-communist bias is prominent in early CDU-led periods, declines in SPD-governed eras, and resurges after reunification. Bias quantification leverages WEAT effect sizes, ECT correlations, and co-occurrence-based t-statistics in a Procrustes-aligned embedding space, showing marked shifts correlating with historical events: the rise of antisemitic rhetoric from Kaiserreich to Weimar, its repudiation after the Holocaust, and the Cold War’s anti-communist polarization. Combined, dense semantic spaces and graph-based propagation chart the waxing and waning of these ideologies, establishing the parliamentary section of the German Commons as an empirical resource for both computational linguistics and modern historical inquiry (Walter et al., 2021).

7. Methodological and Community Significance

The German Commons exemplifies comprehensive open-corpus construction through licensing rigor, context-sensitive filtering, and reproducible workflows. It is methodologically distinguished by:

• Verified, SPDX-tagged licensing from institutional data providers, excluding ambiguous or research-only material;
• Granular domain tagging, enabling targeted downstream sampling and analytic stratification;
• Explicit, percentile-based noise and outlier filters, improving corpus homogeneity;
• Transparent data provenance and pipeline code, fostering reproducibility and community trust.

A plausible implication is the establishment of analogous open-commons infrastructures in other under-resourced languages, leveraging the German Commons as both technical and legal template. By aligning corpus construction with strict open-licensing principles and advanced normalization heuristics, the German Commons provides a scalable model for transparent, reproducible, and legally unencumbered LLM pretraining pipelines.