KletterMix: Climbing Toward High-Quality German Pretraining Data

Abstract: High-quality pretraining data is a central ingredient in modern LLMs, but German-language resources remain far less developed than their English counterparts: they are often smaller, less carefully curated, weakly documented, and rarely validated through controlled training experiments. We introduce KletterMix, a high-quality German corpus for LLM pretraining and annealing, designed as a reusable dataset artifact for the natural language processing and modeling community. KletterMix is built by translating a state-of-the-art English pretraining corpus into German while preserving document boundaries, metadata, source structure, and topical diversity. This construction yields a German corpus with the scale and diversity of a modern pretraining dataset, while enabling direct comparison to its English source. We document the dataset through a broad set of corpus-level analyses, including translation quality, document length distributions, topic coverage, source composition, and geographic metadata. Using COMETKiwi, we show that the translated documents achieve strong quality across diverse domains, suggesting that careful translation can preserve much of the semantic and stylistic richness of the original corpus. Beyond dataset construction, we evaluate KletterMix as training data. Through controlled pretraining and annealing ablations against established German corpora, we show that models trained on KletterMix achieve measurable improvements on German-language downstream evaluations. These results demonstrate that carefully curated translated data can substantially strengthen the German pretraining data ecosystem.

• The paper demonstrates that high-quality translation and proxy-based filtering significantly enhance German LLM pretraining compared to noisy web crawls.
• It employs a detailed pipeline that preserves document boundaries and metadata, using length-aware routing and context-aware chunking to ensure translation integrity.
• Evaluation shows that KletterMix improves training efficiency and downstream reasoning performance, establishing a robust foundation for non-English LLMs.

KletterMix: Advancing German LLM Pretraining through High-Quality Translation and Corpus Engineering

Motivation and Context

Pretraining data strongly dictates LLM performance. While English corpora benefit from extensive curation, documentation, and downstream validation, German-language pretraining resources lag behind, often suffering from raw web crawl noise, limited diversity, insufficient filtering, and weak documentation. KletterMix directly addresses this disparity by transferring the mixture design, source diversity, and topical structure of high-quality English pretraining data (ClimbMix) into German via advanced machine translation, preserving document boundaries and rich metadata. This initiative moves beyond naive web crawling, testing the hypothesis that carefully engineered translation pipelines and rigorous quality controls can establish German pretraining datasets of comparable strength to those for English.

Pipeline Architecture

KletterMix’s pipeline is orchestrated across several stages: source normalization, length-aware routing, document-preserving translation, scalable parallel execution, and post-hoc quality estimation. Documents are bucketed by length to maximize translation efficiency across heterogeneous inputs. Translation is executed using Qwen3.5-397B-A17B-FP8, employing context-aware chunking such that lengthy documents retain cross-chunk discourse continuity.

Proxy-based quality annotation is central: a pilot subset stratified by source cluster is scored using COMETKiwi, and a gradient-boosted regression proxy is trained to generate corpus-wide quality signals using only the target text and derived cheap features (GlotLID signals and text-shape metrics). The proxy shows high Pearson correlation ($r \approx 0.73$) and low MAE ($\approx 0.048$) relative to COMETKiwi, enabling practical large-scale filtering.

Corpus Diagnostics and Quality Analysis

Document-level and corpus-level diagnostics are performed, highlighting KletterMix’s heavy-tailed length profile and the preservation of source mixture structure. Most translations are under 10k tokens, with a long tail extending beyond 20k, mimicking heterogeneous web mixtures typical in English datasets.

Figure 1: Translated document lengths in KletterMix exhibit a heavy-tailed distribution, with most documents under 10k tokens, validating the pipeline’s length-aware batching.

Stricter filtering (proxy-score thresholds at 0.50, 0.55, 0.60) is applied to produce ablation splits for controlled training studies.

Figure 2: Proxy-score distribution and filtering thresholds for 12B-token ablations, marking retained and removed documents to evaluate translation quality impact.

Cluster-level diagnostics confirm consistency in quality profiles across inherited source clusters, enabling systematic audit and targeted curation.

German token share by cluster and corresponding cluster-level topics are mapped to ensure topical diversity, as visualized in the cluster composition plots.

Figure 3: German token share by inherited source-cluster metadata, quantifying the proportional contributions of different source clusters.

Translation Quality and Failure Modalities

Qualitative inspection of translated documents uncovers both high-fidelity renderings and typical MT failure cases: refusal behaviors (content rejection), wrong regional varieties (e.g., Swiss/Alemannic output), inappropriate translation of code identifiers and technical syntax, and partial truncations. Advanced prompt engineering and length-aware chunking minimize these artifacts but do not eliminate them altogether; continued auditing and manual review remain necessary.

Pretraining Dynamics and Benchmark Evaluation

Controlled ablations are performed using Qwen3-0.6B across matched 12B-token splits, comparing KletterMix against FineWeb2-DE and GermanWeb baselines. KletterMix consistently delivers lower training and validation loss, evidencing higher sample efficiency and more effective pretraining signal.

Figure 4: Qwen3-0.6B training loss curves for matched 12B-token German subsets, illustrating superior convergence for KletterMix relative to baselines.

Held-out German evaluations (MMLU, PIQA, HellaSwag, ARC-Challenge) show that KletterMix delivers the best aggregate accuracy (Core Avg. up to 40.2), especially when stricter quality filtering is applied. Gains are concentrated in HellaSwag and ARC-Challenge, which depend on event-level coherence and compositional reasoning—areas less well-covered by raw web data.

Annealing experiments confirm that KletterMix serves as an effective late-stage steering corpus: models previously trained on native German data benefit from further training on the translated mixture, again primarily in reasoning-heavy downstream tasks.

Practical and Theoretical Implications

KletterMix demonstrates that translated and document-preserving pretraining corpora can transfer more than linguistic surface forms—the mixture structure, document boundaries, metadata, and topical diversity imported from English substantially augment German LLMs. Proxy-score filtering delivers measurable improvements, highlighting the necessity of scalable quality diagnostics in massive synthetic data construction. These results emphasize that curated translation pipelines are not simply data augmentation tools, but critical components of rigorous LLM training mixtures.

On the theoretical side, KletterMix motivates new research directions in cross-lingual transfer, mixture engineering, translationese detection, and proxy-based quality control. The preservation of source cluster identifiers and metadata enables aligned comparative studies between English and German corpora, facilitating studies on semantic drift, translation artifacts, and reasoning transfer.

Practically, KletterMix enables the development of German-specific LLMs suited for both instruction-tuning and zero-shot tasks, addressing domains historically underserved by native-language crawls. The dataset's composition is well-documented, openly licensed, and aligned with modern transparency standards, supporting reproducibility and downstream auditability.

Limitations and Future Directions

Despite strong numerical results, KletterMix is limited by inherited source biases, translation artifact risks (e.g., translationese, dropped content, terminology drift), and scope restrictions (fixed parameter/model budget, limited downstream task set). Quality proxies are useful for ranking but cannot fully replace source-aware evaluation or manual inspection. Future work should broaden seed coverage, increase model and token budgets, apply the pipeline to other languages (e.g., French, Italian, Spanish), and expand downstream benchmarks.

Manual audit integration, improved error analysis (on content type, technical domains, naturalness, factual accuracy), and domain-specific filtering of problematic translations (e.g., boilerplate, URLs, duplicated data) would further increase corpus utility and reliability.

Conclusion

KletterMix establishes a practical, empirically validated path toward high-quality non-English pretraining data. Through robust machine translation, document boundary preservation, rich metadata retention, and scalable proxy-based filtering, the corpus enables superior German LLM training and downstream performance, particularly in reasoning- and event-centric benchmarks. The approach is not a substitute for native-language web crawls, but a powerful complement that should be extended, audited, and systematically evaluated in broader multilingual contexts.