Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Data Curation Pipeline

Updated 4 July 2025
  • Data Curation Pipeline is a multi-stage system that selects, cleans, filters, augments, and integrates diverse data sources to create high-quality, purpose-specific datasets.
  • It employs a blend of heuristic filtering, machine learning classifiers, and synthetic augmentation to enhance data quality and tailor outputs for various tasks.
  • Advanced deduplication, quality scoring, and model-based filtering ensure that the curated data remains coherent, diverse, and domain-appropriate for effective downstream applications.

A data curation pipeline is a structured, multi-stage system designed to select, clean, filter, augment, and integrate heterogeneous data sources in order to produce high-quality, purpose-specific datasets suitable for downstream tasks such as machine learning training or scientific analysis. Modern data curation pipelines may combine rule-based heuristics, model-based (machine learning) filtering, and synthetic data generation to address complex requirements for quality, diversity, and domain appropriateness.

1. Pipeline Architecture and Stages

The Aleph-Alpha-GermanWeb curation pipeline exemplifies a comprehensive system integrating multiple filtering and augmentation steps to produce German-language LLM pre-training datasets. The pipeline employs a blend of heuristic logic, machine learning classifiers, and generative LLMs in the following stages:

  1. Heuristic Filtering:
    • Excludes documents from a blacklist of unwanted domains (e.g., 4.6M domains identified as fraud, adult, or low-quality), using both regular expressions and curated lists.
    • Extracts plain text from HTML using the resiliparse parser, which was empirically found to outperform the trafilatura tool used in baselines like FineWeb2 for LLM pre-training outcomes.
    • Performs strict language identification via fastText n-gram classifiers, retaining only documents with high German language probability.
  2. Text Quality Cleaning:
    • Applies repetition detection thresholds at multiple levels: line, paragraph, and n-gram character redundancy. For example, documents with >28.2% duplicate lines are rejected.
    • Document-level heuristics filter out texts with abnormal lengths, unnatural mean word lengths, excessive symbols, low fractions of alphabetic words, or high numeric content.
    • Line-level rules discard lines with excessive capitalization, high numeric ratios, too few words, or recognized boilerplate. Specific thresholds are tuned to optimize for German-language coherence.
  3. Deduplication:
    • Exact deduplication is performed at the document hash level; fuzzy deduplication uses MinHash signatures over 5-gram character windows and locality sensitive hashing (LSH) to cluster near-duplicates and remove redundant samples.
  4. Model-based Filtering:
    • Trained fastText and BERT classifiers predict document grammaticality and various aspects of quality (e.g., informativeness, orthographic correctness), using labeled data derived from LanguageTool (for grammar) and an LLM-judge (Mistral-Nemo-Instruct-2407) for style/formality.
    • Documents are assigned quality points via a ruleset based on classifier outputs, and sorted into five quality buckets, with only upper buckets feeding into final dataset versions.
    • This "stacked" approach allows for robust error minimization and improves sample diversity and relevance to downstream LLM tasks.
  5. Synthetic Data Generation:
    • Organic (web-scraped and filtered) German-language content is augmented by prompting an instruction-tuned LLM (Mistral-Nemo-Instruct-2407, 12B params) to generate synthetic paraphrases and factual expansions in five styles: Wikipedia-style rephrasing, summarization, pedagogical restatements, information extraction, and question–answer pair creation.
    • For long organic documents, segmentation is performed before LLM completion to fit model context windows and to maintain quality.
    • Synthetic variants per organic sample are strictly limited (≤5) to prevent quality degradation from overexposure (“epoching”) of repeated data.

2. Data Sources and Motivation

The Aleph-Alpha-GermanWeb dataset is composed of:

  • Recent Common Crawl German-language web data (September 2024 to February 2025): Filtered and de-duplicated via the above pipeline.
  • FineWeb2: A state-of-the-art, heuristically filtered multilingual web corpus, used both as direct input and as a base for synthetic data generation.
  • Synthetic Data: Generated in-context from actual, organic German documents using LLMs. Each synthetic document is conditioned on real German content, ensuring that the synthetic expansion remains linguistically, culturally, and topically accurate.

This composite design addresses two constraints: the comparative scarcity of high-quality German-language web content (German is ≤1.4% of FineWeb2), and the limits of translation-based corpus construction for LLMs. Conditioning synthetic data on native German sources harnesses authentic domain, genre, and style variance.

3. Filtering, Quality Control, and Model-based Selection

The pipeline applies multi-layered controls to enforce quality and relevance:

  • Grammar and Educational Quality Classification:
    • A fastText classifier is trained with silver-standard labels (from LanguageTool grammar rules) and gold-standard human/LLM labels (for style and coherence).
    • A BERT model further refines classification, especially at high-confidence thresholds.
  • Ensembling and Scoring:
    • Heuristics and model outputs are ensembled via a point-scoring system: clean, grammatical, educational documents get more points; documents are retained or bucketed according to point totals.
  • Deduplication:
    • Multi-stage deduplication ensures global uniqueness, even for near-identical samples resulting from different crawls or from paraphrasing in the synthetic stage.

This approach minimizes the presence of ungrammatical, off-topic, duplicated, or low-informational samples, which is critical for high-capacity LLM pre-training.

4. Synthetic Data Generation Methodology

Synthetic data is produced as follows:

  • Prompting: For each organic document (or chunk), a series of prompt templates (e.g., “Paraphrase in Wikipedia style”) is applied. For long texts, a semantic text splitter ensures contiguous, context-appropriate input to the LLM.
  • Generation: Each prompt generates one synthetic document. Output is cleaned to strip any unintended LLM artifacts (“Here is the new version: ...”).
  • Blending: Synthetic and organic datasets are combined at controlled ratios. To prevent overfitting or distributional skew, the number of synthetic variants per organic sample is capped—a practice directly supported by evidence that excessive repetition in synthetic data (“epoching” beyond 5 times) harms LLM downstream performance.

5. Evaluation and Benchmark Comparison

Empirical evaluation demonstrates the pipeline’s effectiveness:

  • Benchmarks:
    • MMMLU-DE (professional human German translation): Multidisciplinary knowledge and reasoning.
    • ARC-Easy-DE: Reasoning and commonsense multiple choice.
    • TruthfulQA-DE: Factuality and truthfulness in answer generation.
    • HellaSwag-DE: Commonsense completion.
  • Models: Both a 1B Llama-architecture and an 8B Hierarchical Autoregressive Transformer (tokenizer-free) are pre-trained on the resulting datasets.
  • Results: For the same token budget, Aleph-Alpha-GermanWeb (filtered CC, synthetic, or mixture) outperforms FineWeb2 alone on all benchmarks. Notably, even when FineWeb2 is hybridized with premium human-curated German sources (such as Wikipedia and the German National Library), Aleph-Alpha-GermanWeb maintains a significant performance lead.
  • Significance: High-quality, model-filtered, and LLM-augmented data gives stronger results than simply scaling up web-crawled data or enriching with “classic” curated sources, consistent at both the 1B and 8B scale and robust to tokenization schemes.

6. Implications for LLM Pre-training and Data-driven Research

The results support several important conclusions:

  • Model-based filtering and synthetic data generation provide substantial leverage in languages with limited web corpus size, bypassing diminishing returns from brute-force data scaling.
  • Conditioned synthetic generation (using authentic, filtered native language documents as prompt context) improves data quality and domain relevance compared to non-conditioned translation or random generation.
  • Deduplication, grammar and educational quality classification, and point-based ensembling are all essential to maintain high effective information density and limit overfitting/scaling law violations.
  • Pipeline extensibility allows adaptation to other resource-scarce languages and domains by substituting appropriate classifiers, LLM prompts, and filtering thresholds.

The Aleph-Alpha-GermanWeb curation pipeline illustrates the emergent consensus that, in LLM pre-training, quality-optimized, compositionally validated, and adaptively augmented data pipelines are essential for both model efficiency and final model capability, especially as language coverage and application diversity expand.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Upgrade)