- The paper introduces CodeAlchemy, a framework that transforms public code into five data streams—including enhanced code, grounded developer tasks, dialogue, and execution traces—yielding more than 500B synthetic tokens across 15 languages.
- The paper finds that diverse mixtures outperform repeated training on homogeneous data, while reasoning traces improve results substantially, including Java MultiPL-E performance rising from 0.0 to 30.5.
- The paper shows that a 3B model trained with CodeAlchemy data reaches 83.5% HumanEval and 15.36 ROUGE-2 on TraceEval, outperforming models roughly ten times larger on several code reasoning measures.
CodeAlchemy is a synthetic data generation framework that transforms publicly sourced code into diverse, semantically rich pretraining data. The authors, from the MIT-IBM Watson AI Lab and IBM Research, construct five complementary data streams from three seed corpora (stack-edu, the-stack-v2-train-smol-ids, RefineCode) spanning 15 languages, producing over 500B tokens of synthetic data plus 350B reasoning tokens — an order of magnitude beyond prior efforts such as SwallowCode (50B, Python-only) or Nemotron-Pretraining-Code-v2 (~480B) (2606.10087). The framework also contributes two new evaluation benchmarks, DevEval and TraceEval, which expose substantial weaknesses in frontier models on practical developer tasks and execution prediction.
Motivation and design
The paper identifies three limitations of pretraining on raw code: high-quality code is scarce; raw code provides sparse signal for realistic user interactions such as debugging, refactoring, and explanation; and next-token prediction teaches syntax but not semantics — a model learns the pattern of x = foo(y) without learning what values x takes or how state evolves. Prior synthetic code efforts address only parts of this space: SwallowCode restricts quality rewriting to Python (Fujii et al., 5 May 2025), while Nemotron-Pretraining-Code-v2 limits rewriting to Python and cross-language tasks to Python→C++.
CodeAlchemy's pipeline proceeds in stages. First, gpt-oss-20b scores stack-edu files 0–10 for quality; a distilled SmolLM2-360M scorer accelerates this at scale. The score distribution is striking: most languages average below 5, with a large mass at score 0, indicating publicly sourced code is far from production quality. Five synthesis strategies then operate on scored files:
- CodeEnhance rewrites low-quality code (bins 4–6) to add unit tests, documentation, error handling, and style-guide adherence, yielding 112B tokens across 15 languages. Judged by gpt-oss-120b, rewritten code reaches a quality of ~8 regardless of original score.
- CodeQA uses five hand-crafted templates inspired by benchmark formats (basic programming, competitive programming, function execution, completion, trace prediction), grounding generation in actual source files to avoid prompt-only repetitiveness. Sandbox execution validates correctness, eliminating ~28% of samples; the result is 22M QA pairs across 7 languages.
- CodeDev generates grounded developer tasks (debugging, refactoring, porting, feature extension) referencing concrete code elements, with difficulty evolution via mutation/crossover/hybrid/invention strategies and three-axis quality filtering. After filtering, 62M pairs span 207B tokens plus 68B reasoning tokens.
- CodeDialogue extends CodeDev into multi-turn conversations averaging 3.6 rounds, producing 31M conversations and 150B tokens plus 271B reasoning tokens.
- CodeTrace instruments 4M files across 14 languages to emit structured trace events (IN/OUT/VAR/BRANCH/LOOP/ERR/TRANSFORM), generates test inputs, executes in isolated bubblewrap-based sandboxes with whitelisted package registries, and filters non-determinism by triple execution (~75% filtered). The result is 1.3M (code, trace) pairs covering 5,430 libraries — the largest execution-trace dataset reported, well beyond prior work restricted to toy grammars or single Python functions.
New benchmarks
Two benchmarks target gaps in existing evaluation. DevEval contains 1,488 developer tasks across 12 languages, clustered for diversity via embeddinggemma-300m embeddings and difficulty-calibrated sampling; models are judged against Claude Sonnet 4.5 reference responses via pairwise preference scoring with order-switching. TraceEval contains 1,050 execution-prediction tasks across 14 languages, filtered for predictability and calibrated so that no model scores near zero, evaluated by exact match and line-level ROUGE-2 F1.
The headline finding is that frontier models perform poorly on both. On TraceEval, Claude Sonnet 4.5 achieves only 5.6% exact match and 30.8% line bigram F1; gemma-3-27b-it achieves 0.4% exact match. On DevEval, gpt-oss-120b wins 61% against the Claude reference, while granite-4.0-h-small wins only 2%. These results imply that current LLMs have substantial deficits in mental program simulation and in satisfying concrete developer goals given a source file — capabilities not measured by HumanEval-style function synthesis.
Experimental findings
All experiments use continual pretraining of a 3B base checkpoint, varying only data mixtures. Ten research questions structure the analysis:
| Question |
Key result |
| RQ1: Is enhanced data sufficient alone? |
CodeEnhance alone: HumanEval 40.9 but MBPP collapses to 5.6; mixing restores MBPP |
| RQ2: Are constituents complementary? |
CodeTrace alone yields 13.55 ROUGE-2 on TraceEval; CodeDev/Dialogue lead DevEval |
| RQ3: Do mixtures balance performance? |
Mix3 (20% raw) gives best balance: HE 40.2, MBPP 52.9 |
| RQ4: Repetition vs. diversity? |
1 epoch over Mix1 (600B) beats 5 epochs over homogeneous data on nearly all metrics |
| RQ5: Reasoning traces help? |
+3.7 HumanEval, Java MultiPL-E from 0.0 to 30.5 |
| RQ6: Better SFT backbone? |
Mix1 backbone after SFT: 83.5 HumanEval vs 76.8 raw (+6.7) |
| RQ7: Generator model quality? |
Smaller gemma-3-4b generator outperforms larger ones on MBPP+ (52.1 vs 46.6 Nemotron) |
| RQ8: Seed quality? |
Matters for raw code, largely irrelevant post-rewrite |
| RQ9–10: vs. SwallowCode/Nemotron? |
Combined mixtures beat baselines; 90%+ judge win rates |
Several results deserve emphasis. First, the MBPP collapse under CodeEnhance is a bold and counterintuitive claim: style-standardized rewrites (e.g., converting is_Power_Of_Two to snake_case per PEP 8) break benchmark harnesses that call original names, and more broadly reduce the stylistic diversity MBPP rewards. The authors show this is a convention mismatch rather than a capability gap — their best model recovers to 63.2 MBPP / 53.4 MBPP+ once diversity is restored. Second, diversity beats repetition: one epoch over a diverse 590B-token mixture outperforms five epochs over any homogeneous subset, echoing Kimi K2's observation in the text domain. Third, smaller generators can be better: gemma-3-4b-it rewrites preserve beneficial variation that stronger models over-standardize away, even though LLM judges rate the weaker model's output lower on correctness — a tension the paper resolves by noting judges reward technical depth while MBPP rewards naming variation. Fourth, reasoning traces during pretraining yield consistent gains, most dramatically on Java MultiPL-E (0.0 → 30.5).
The final 3B model, trained on the Mix1 mixture and fine-tuned on OpenCodeInstruct, achieves 83.5% HumanEval, 63.2% MBPP, 8.09% DevEval win rate, and 15.36 ROUGE-2 on TraceEval — outperforming models roughly 10× its size including Gemma-3-27B and Granite-4.0-32B on these axes. A 10B-token CodeTrace-only anneal already exceeds gemma-3-27b-it on TraceEval, implying execution traces are an unusually efficient training signal for semantic understanding.
Limitations and open questions
The paper concedes several constraints. CodeTrace covers only a fraction of the software ecosystem; ~75% of instrumented files are discarded as non-deterministic or unexecutable, and coverage of web development and ML frameworks remains thin. All experiments are Python-centric in ablations (RQ6–RQ9 are explicitly Python-only), leaving cross-language generalization of the smaller-generator finding open. DevEval and TraceEval rely on LLM-generated references and LLM judging, inheriting judge bias risks despite order-switching mitigation. Most significantly, CodeAlchemy addresses single-file tasks only; repository-level agentic workflows (multi-file refactoring, dependency management, test-driven debugging as in SWE-bench) are left entirely unaddressed, and whether the observed gains transfer to such settings is an open question. Whether the MBPP naming-convention artifact generalizes to other benchmarks sensitive to stylistic conventions also merits further study.
Conclusion
CodeAlchemy demonstrates that large-scale, semantically grounded synthetic data — quality-aware rewriting, template-grounded QA, developer tasks, multi-turn dialogue, and executed traces — substantially outperforms scaling on raw code repositories, enabling a 3B model to match or exceed models ten times larger on code generation and execution-prediction benchmarks. Its two new benchmarks quantify real deficiencies of frontier models in semantic understanding, and its ablations establish that stylistic diversity, not merely quality, governs robust downstream performance.