KletterMix: German Pretraining Corpus

• KletterMix is a monolingual German corpus of approximately 725 billion GPT-2 tokens, translated 1:1 from ClimbMix with full preservation of document boundaries and metadata.
• It employs a robust translation pipeline using Qwen3.5-397B-A17B along with COMETKiwi-based filtering to ensure high translation quality and corpus integrity.
• The corpus supports controlled pretraining and annealing evaluations, showing significant improvements over existing German datasets on benchmarks like HellaSwag and ARC-Challenge.

KletterMix is a high-quality German corpus for LLM pretraining and annealing, designed as a reusable dataset artifact for the natural language processing and modeling community (Kraus et al., 2 Jun 2026). It is constructed by translating the English pretraining mixture ClimbMix into German while preserving document boundaries, metadata, source structure, and topical diversity, thereby yielding a monolingual German corpus that remains aligned 1:1 with its English source (Kraus et al., 2 Jun 2026). The dataset is positioned as a response to the relative underdevelopment of German-language pretraining resources, which are described as smaller, less carefully curated, weakly documented, and rarely validated through controlled training experiments. Within that framing, KletterMix combines large scale, preserved provenance, corpus-level analysis, translation-quality diagnostics, and controlled downstream evaluation in a single dataset artifact.

1. Corpus definition and scale

KletterMix comprises approximately 725 billion GPT-2 tokens of German text, spread over millions of documents; the exact document count matches the English source ClimbMix (Kraus et al., 2 Jun 2026). All documents are German translations of an English pretraining mixture, so the corpus is monolingual in German but aligned 1:1 with its English source. This alignment is central to the dataset’s design because it permits direct corpus-level comparison between the English and German sides.

The English source is stored as sharded JSONL files. Each record contains a document_id, English text, and metadata including source cluster, source location, approximate source length, and license. In KletterMix, document boundaries and identifiers are preserved, all original metadata fields are retained, and the mixture’s cluster-and-source structure and topical diversity remain intact. This design enables direct corpus-level comparisons such as length ratios and topic coverage.

A plausible implication is that KletterMix is not only a German pretraining resource but also an aligned analytical substrate for studying cross-lingual corpus transfer under fixed source composition. That implication follows from the explicit preservation of document-level structure and metadata together with the 1:1 alignment to ClimbMix.

2. Translation architecture and execution

The translation pipeline uses Qwen3.5-397B-A17B in 8-bit float (FP8), served with speculative decoding (MTP-2) on NVIDIA B200 GPUs (Kraus et al., 2 Jun 2026). Documents are assigned through length-aware routing to one of eight source-length buckets: 4 k, 8 k, 16 k, 18 k, 20 k, 32 k, 64 k, and $> 64$ k tokens. The pipeline applies bucket-specific batching and timeouts.

Documents that exceed a 20 000-token source budget are split into sentence chunks of at most 20 000 tokens, with a 2 000-token “previous-translation” window for continuity. The decoder cap for each source chunk of length $\ell_s$ is set dynamically as

$$\ell_t^{max} = \min(L_{max}, \lceil \alpha \cdot \ell_s + \beta \rceil),$$

with $L_{max}=32\,768$, $\alpha=2.0$, $\beta=1\,024$, and a minimum cap of 2 048. The execution is described as robust and shard-wise: workers preserve intermediate outputs, monitor endpoint health, and resume interrupted shards without reprocessing completed records.

These details matter because the dataset is built from a modern pretraining mixture rather than from isolated benchmark translation. The combination of sharded execution, length-aware routing, chunking, and continuity windows indicates an explicit attempt to operationalize high-volume document translation while preserving the structural properties needed for pretraining.

3. Filtering, proxy modeling, and release variants

Post-processing begins with a stratified pilot sample of approximately 20 000 documents, with 100 documents per source cluster, translated and scored using COMETKiwi in a reference-free quality-estimation setup (Kraus et al., 2 Jun 2026). The pilot is used to reveal cluster-level quality variation. A gradient-boosted, target-only proxy model is then trained on those pilot COMETKiwi scores using features such as language ID signals, token- and character-shape statistics, and token-repetition and lexical-diversity metrics.

The language ID signal is specified as GlotLID probability for German. The token- and character-shape statistics include length and $\alpha$-, digit-, and punctuation-ratio features. Proxy validation on 18 275 held-out documents yields Pearson $\approx 0.73$, Spearman $\approx 0.73$, and MAE $\approx 0.048$. The release includes the full “unfiltered” corpus and three fixed-budget 12 B-token training splits filtered by proxy score thresholds of at least 0.50, 0.55, and 0.60.

This filtering strategy is notable because it separates full-corpus release from fixed-budget training subsets. A plausible implication is that the authors treat quality control and token-budget matching as distinct experimental concerns: the first concerns the corpus as an artifact, while the second concerns controlled model comparison.

4. Corpus analyses and inherited structure

The corpus exhibits a heavy-tailed document-length profile: most documents are shorter than 10 000 tokens, with a long tail beyond 20 000 (Kraus et al., 2 Jun 2026). Within each source-length bucket, German targets generally match the bucket rank, but long-bucket tails reveal some unusually short translations, which are reported as indicating possible truncation or failure cases.

Topical structure is inherited through cluster_id values from 1 to 20, which track broad topical domains. Examples given include Cluster 1 for math/statistics, Cluster 9 for astronomy, and Cluster 16 for environmental sustainability. Token share by cluster is uneven; Clusters 6, 7, and 12 each contribute more than 10% of tokens. All major English-source content types, including web pages, news, books, Q&A, code, and education, are represented in roughly the same proportions as in ClimbMix.

KletterMix also retains all original source-location fields such as crawl domain and publication locale. Although a full country-level histogram is not plotted, every record’s provenance is preserved for downstream geographic analyses. This provenance retention is technically consequential because it supports geographic or source-stratified auditing without reconstructing metadata after translation.

On the pilot set, COMETKiwi scores average approximately 0.83, with source-cluster means reported in the appendix. Proxy-estimated quality across clusters, measured through 10th–90th and 25th–75th quantiles, is reported as consistent between the 12 B-token subset and the full release, demonstrating that fixed-budget sampling preserves the corpus quality profile. This suggests that the filtered training subsets are intended to remain distributionally representative of the broader corpus rather than to become narrowly optimized slices.

5. Pretraining and annealing protocol

Controlled model training uses a Qwen3-0.6B decoder-only Transformer with 28 layers, hidden size 1 024, FFN size 3 072, Grouped-Query Attention with 16 query heads and 8 KV heads, SwiGLU activations, RMSNorm, RoPE with base $\ell_s$0, vocabulary size 151 936, and maximum context 4 096 (Kraus et al., 2 Jun 2026). The training budget is 12 billion tokens, described as Chinchilla scaling for the 0.6 B-parameter model.

The training recipe uses batches of 512 sequences of 4 096 tokens, equivalent to 2.1 M tokens per step, with microbatch=8 and grad-accum=8, giving 5 722 steps. Optimization uses Fused Adam with $\ell_s$1, $\ell_s$2, $\ell_s$3, weight_decay=0.1, and gradient clipping at 1.0. The learning-rate schedule is cosine annealing from 0 to $\ell_s$4 over 5% warmup, equal to 286 steps, decaying to $\ell_s$5. Precision is BF16 macro plus FP8 micro.

The annealing setup continues models after 5 100 steps on FineWeb2-DE using either GermanWeb or KletterMix with matched 12 B-token subsets, while keeping training hyperparameters identical. The paper explicitly states that there is no explicit loss-weight $\ell_s$6 beyond the cosine learning-rate schedule; “annealing” here refers to a continued-training curriculum. This clarification addresses a likely misconception: in this context, annealing is not a mixture-weighting mechanism but a staged continuation protocol.

The baselines are FineWeb2-DE, described as a cleaned German web crawl from FineWeb2, and GermanWeb, described as a curated German-only web corpus with quality filtering and provenance tracking.

6. Evaluation results, trade-offs, and limitations

Downstream evaluation uses 5-shot accuracy with lm-eval-harness on four German-language benchmarks: MMLU, PIQA, HellaSwag, and ARC-Challenge (Kraus et al., 2 Jun 2026). The reported aggregate metric is

$\ell_s$7

with propagated standard error. The benchmark suite mixes broad academic and professional knowledge, physical commonsense, event-continuation commonsense, and science question answering.

The principal results are as follows:

Setting	Core Avg.	Notes
GermanWeb	$\ell_s$8	Independent pretraining
FineWeb2-DE	$\ell_s$9	Independent pretraining
FineWeb2-DE $$\ell_t^{max} = \min(L_{max}, \lceil \alpha \cdot \ell_s + \beta \rceil),$$0 GermanWeb	$$\ell_t^{max} = \min(L_{max}, \lceil \alpha \cdot \ell_s + \beta \rceil),$$1	Annealing
FineWeb2-DE $$\ell_t^{max} = \min(L_{max}, \lceil \alpha \cdot \ell_s + \beta \rceil),$$2 KletterMix	$$\ell_t^{max} = \min(L_{max}, \lceil \alpha \cdot \ell_s + \beta \rceil),$$3	$$\ell_t^{max} = \min(L_{max}, \lceil \alpha \cdot \ell_s + \beta \rceil),$$4 vs. GermanWeb, $$\ell_t^{max} = \min(L_{max}, \lceil \alpha \cdot \ell_s + \beta \rceil),$$5 vs. start
Unfiltered KletterMix	$$\ell_t^{max} = \min(L_{max}, \lceil \alpha \cdot \ell_s + \beta \rceil),$$6	KletterMix variant
KletterMix-Filt$$\ell_t^{max} = \min(L_{max}, \lceil \alpha \cdot \ell_s + \beta \rceil),$$7	$$\ell_t^{max} = \min(L_{max}, \lceil \alpha \cdot \ell_s + \beta \rceil),$$8	Best point estimate

For annealing, gains are concentrated in HellaSwag at $$\ell_t^{max} = \min(L_{max}, \lceil \alpha \cdot \ell_s + \beta \rceil),$$9 and ARC-C at $L_{max}=32\,768$0, while PIQA is stable and MMLU is roughly constant. Among KletterMix variants, stricter filtering improves reasoning-heavy tasks, specifically HellaSwag and ARC-C, but can slightly hurt MMLU. The paper interprets this as a trade-off that motivates validation-selected thresholds rather than a single universal cutoff.

The statistical treatment is deliberately narrow. All 5-shot accuracies report evaluation-set standard errors, and the comparisons are single-run, matched-token-budget experiments. No multi-seed significance tests are reported. This is an important limitation: the standard errors quantify benchmark sampling noise, but they do not establish run-to-run stability. Even so, the matched-token-budget design makes the reported improvements directly comparable within the stated experimental frame.

Taken together, the reported evidence supports the claim that carefully curated translated data can substantially strengthen the German pretraining data ecosystem (Kraus et al., 2 Jun 2026). More specifically, KletterMix combines scale, preserved source structure, quality diagnostics based on COMETKiwi and proxy modeling, and controlled pretraining and annealing results showing measurable gains over established German corpora. A plausible implication is that translated corpora with preserved provenance and source composition may serve not merely as stopgap resources for lower-resourced languages, but as systematically analyzable pretraining substrates in their own right.