OpenCoder Annealing Corpus

• The OpenCoder Annealing Corpus is a multicomponent dataset comprising original, algorithmic, synthetic, and textbook code samples for advanced code LLM training.
• It employs stringent deduplication and language-specific filtering to ensure data integrity, provenance, and reproducibility.
• Ablation studies confirm that the inclusion of synthetic and algorithmic subsets significantly boosts model performance in code reasoning and evaluation.

The OpenCoder Annealing Corpus is a rigorously curated, multicomponent dataset central to the final-stage (“annealing”) training of the OpenCoder code LLM, enabling advanced model performance in code generation, algorithmic reasoning, and scientific evaluation. It is distinguished by its open and fully reproducible design, comprehensive documentation of data provenance, cleaning heuristics, deduplication, and ablation-driven quantification of impact. The corpus is tightly integrated into the OpenCoder pipeline and establishes best practices for the scientific community on the construction of high-quality, open-access code LLM corpora (Huang et al., 2024).

1. Composition and Data Sources

The Annealing Corpus constitutes 100B tokens assembled from four primary categories, with source and mixture weights fixed at curriculum initiation:

Category	Sub-corpus	Tokens	Mixture Fraction
Original Distribution	RefineCode	83.94 B	84%
Algorithmic	Algorithmic Corpus	12.44 B	12%
Synthetic	High-Quality Snippets	2.71 B	2.7%
Synthetic	Code Textbooks	0.91 B	0.9%
Total	–	100.0 B	100%

• RefineCode: Cleaned and deduplicated code from the broader OpenCoder pretraining pipeline, covering 607 programming languages, with ~70% of tokens from the top 10 (Python, Java, C++, JavaScript, etc.).
• Algorithmic Corpus: Programmatically sampled from RefineCode by matching content/filename to algorithm task keywords (e.g., “leetcode”, “def solution”, “class Solution”).
• High-Quality Snippets: Synthetic, validated functions and tests generated by strong LLMs, seeded on Algorithmic Corpus code, with output gated by parsing, compilation, and automated unit test success.
• Code Textbooks: Synthetic, human-readable “lesson” and “concept point” summaries with algorithm discussion and stepwise walkthroughs, also LLM-generated over hqcode dataset seeds.

All four are deduplicated and quality-filtered by a unified pipeline.

2. Corpus Construction, Cleaning, and Deduplication

The construction process for annealing-phase data is characterized by strict filtering and deduplication, ensuring provenance traceability and minimizing data leakage:

• Heuristic Per-file and Language-specific Filters: Inheritance of RefineCode’s pipeline, which includes copyright-removal, PII-masking, and extensive code-formatting/correctness checks for every file.
• Deduplication Pipeline:
  • Exact (SHA256) Deduplication: Retain only the file from the repository with highest star count and most recent commit; removes ≈75% of raw GitHub code files.
  • Fuzzy (MinHash LSH) Deduplication: Character-level 5-gram shingling, 2,048-dimensional MinHash, and LSH (16 bands × 128 rows), with Jaccard similarity threshold ≥ 0.9. Keeps the highest-quality repo’s copy, eliminates ≈6% further.
• Deduplication Example (Python slice):
  • Raw files: 485,817,123
  • After deduplication: 30,488,834 files (~6.3% retained), ≈32.74B tokens
  • For comparison, repository-level dedup yields ≈99.47B tokens

Synthetic outputs are further filtered to require both successful parsing/compilation and passing code test harnesses.

3. Synthetic Data Generation Protocol

Synthetic annealing-stage data, comprising both high-quality code snippets and code textbooks, is generated using state-of-the-art LLMs with carefully documented prompt engineering and generation constraints:

• High-Quality Code Snippets:
  • Seed: Extracted functions from Algorithmic Corpus
  • Generator: Strong open-weight LLMs (e.g., Qwen2-72B-Instruct)
  • Prompting: Few-shot with two in-context “function + tests” examples
  • Post-filtering: Retain only if code compiles/parses and all generated pytest-style test cases pass
  • Scale: 2.71B tokens across multiple languages
• Code Textbooks:
  • Seed: hqcode dataset (multilingual algorithmic tasks, GPT-4-Mini)
  • Generation: Multi-turn conversations to produce lessons, complexity analysis, and stepwise walkthroughs
  • Scale: 0.91B tokens
• Hyperparameters:
  • Temperature = 0.7–1.0 (controls diversity)
  • Max tokens per example ≈ 512
  • Beam size = 1 (greedy decoding)

4. Training Curriculum, Sampling, and Scheduling

Annealing-stage training uses a fixed mixture and explicit learning-rate schedule to optimize model adaptation:

• Data Sampling: Sub-corpus $i$ with mixture fraction $w_i$ is sampled with probability

$p_i = \frac{w_i}{\sum_j w_j}$

with $w = [0.84, 0.12, 0.027, 0.009]$ and $\alpha = 1$—no reweighting.

• Context Window: 4,096 tokens, 50% chunk overlap, yielding ≈25 million training chunks.
• Learning-rate Schedule: WSD (“Warmup-Sharp-Decay”, from MiniCPM):
  • Warmup to $\eta_{peak} = 3\times10^{-4}$ for first 8B tokens (2 000 steps)
  • Exponential decay to $\eta_{min} = 10^{-5}$ at 100B tokens:

  $$\eta(t) = \eta_{\max}\times\exp\!\Bigl(-\,\frac{t - T_{\rm warmup}}{T_{\rm total} - T_{\rm warmup}}\ln\frac{\eta_{\max}}{\eta_{\min}}\Bigr)$$

• Mixture Fixing: Ratios are set at the curriculum start and never modified to avoid overfitting and catastrophic forgetting.

5. Quantitative Impact and Ablation Studies

Experimental ablations in the OpenCoder work validate the criticality of the Algorithmic and Synthetic subsets:

Model Variant	HumanEval Pass@1	MBPP Pass@1
Without Synthetic & Algorithmic	50.1	65.2
With Full Annealing	54.3	70.6

• Exclusion of Algorithmic and Synthetic data causes a 4–5 point drop in pass@1 accuracy on HumanEval and MBPP.
• The final 100B annealing tokens (the corpus described here) are where the largest jumps in model code reasoning and algorithmic performance occur.
• This demonstrates that high-quality, self-contained, and test-validated examples possess markedly higher marginal utility than further scaling base corpus volume.

6. Best Practices and Reproducibility Guidelines

The OpenCoder release is accompanied by explicit recommendations for reconstructing or extending the Annealing Corpus:

1. Begin with a completely cleaned and deduplicated base corpus (RefineCode pipeline), ensuring strict copyright and ethical compliance.
2. Maintain ≥80% “original-distribution” tokens throughout annealing to mitigate catastrophic forgetting.
3. Use keyword-driven sampling plus repeated code-quality filters to extract a high-signal Algorithmic Corpus subset.
4. Generate all synthetic examples from these seeds with strong LLMs and enforce stringent pass-through compilation and testing.
5. Fix sub-corpus mixture ratios at annealing onset; focus on quality, not sheer synthetic volume.
6. Provide full documentation for prompt templates, sampling, and LLM hyperparameters.
7. Pilot all modifications on small-scale models (e.g., 1.5B parameters) with ablation validation before scaling.

This explicit methodology enables both strict reproducibility and principled extension, supporting new mixture explorations (e.g., heavier algorithmic or novel synthetic content) while maintaining robust learning anchors.

7. Significance in Open LLM Research

The OpenCoder Annealing Corpus sets a comprehensive standard for transparent, scientifically valid LLM dataset construction in code AI research. Its practices in deduplication, synthetic data generation, mixture scheduling, ablation-based validation, and open-sourcing of all steps render it a model for future code LLM and foundation-model efforts. The corpus provides both a benchmark and reference implementation for the community, emphasizing that curation, not mere volume, underlies state-of-the-art code reasoning performance (Huang et al., 2024).