SynthSmith Framework for Code LLMs

• SynthSmith is a synthetic data-generation framework that procedurally constructs novel problem statements, solutions, and test suites for competitive programming.
• It employs a feature-driven pipeline and dual verification to ensure high-quality, diverse datasets that enhance code reasoning model performance.
• Empirical results show that its SFT and RL datasets enable state-of-the-art performance, outperforming models trained on larger real-world corpora.

SynthSmith is a fully synthetic data-generation framework designed to train LLMs for code reasoning in competitive programming, independently of human-written corpora. Its architecture procedurally constructs novel problem statements, synthesizes matching solutions, and generates rigorous test suites through an explicitly feature-driven pipeline. SynthSmith demonstrates that high-capacity code LLMs can be developed using exclusively synthetic supervised fine-tuning (SFT) and reinforcement learning (RL) datasets—enabling greater scaling, diversity, and verifiability than possible with real-world competitive programming data. Empirical results show these synthetic datasets are sufficient to achieve state-of-the-art performance on standard code reasoning benchmarks, surpassing models trained on larger real-world corpora (Wu et al., 11 Jan 2026).

1. Motivation and Design Rationale

Competitive programming tasks require deep algorithmic reasoning, complex solution chains, and robustness to adversarial cases. Real-world benchmarks such as APPS, CodeContests, and TACO are constrained by scale and locked to historic contest distributions, inhibiting further generalization gains. SynthSmith addresses these limitations by:

• Generating novel tasks via explicit feature evolution, rather than repurposing existing problems.
• Ensuring solution quality with a dual-verification protocol that cross-examines LLM-generated code against automatically constructed and semantically weighted test suites.
• Enabling dataset scaling by generating modular SFT and RL corpora whose properties obey empirical scaling laws, and whose scale/diversity can be systematically tuned (Wu et al., 11 Jan 2026).

This approach unbinds code reasoning LLMs from the finite pool of real competitive programming tasks, creating an extensible foundation for continual progress.

2. Feature-Based Synthesis Pipeline

SynthSmith operates in four primary stages, detailed in the following table and expanded upon subsequently:

Stage	Key Technique	Output
Task Generation	Feature tree sampling & prompt-based mapping	Code-style statements
Test-Case Generation	Prompting-based and tool-based generation	Reproducible test inputs
Solution Sampling	Long-CoT via multiple LLMs	Candidate code solutions
Dual Verification	Majority voting & weighted evaluation	Golden solution, test suite

Task Generation via Feature Trees

The feature tree is synthesized by parsing 10,000 TACO competition snippets, with GPT-4-labeled leaves representing concepts such as “Dijkstra’s algorithm,” “segment tree,” or “boundary conditions.” The tree is evolved in breadth (e.g., adding “heap sort” alongside “merge sort”) and depth (e.g., “prefix sum” ↦ “Fenwick tree”), resulting in approximately 100,000 discrete features. A consistent subtree is extracted—one whose features can realistically co-occur in a task—and mapped, through a prompt-driven process, to a structured JSON with:

• feature_roles_tree—feature leaves tagged as “potential_use,”
• selected_features_tree—a minimal coherent subset,
• integration_strategy—a natural language formulation for task design.

This representation is rendered to Codeforces, LeetCode, or AtCoder styles via templates. Decoupling feature selection from narrative formulation demonstrably increases task difficulty by ~5 avg@4 points, measured via ablation.

Test-Case Generation

Without ground-truth test sets, SynthSmith synthesizes $n$ input cases $x_1, ..., x_n$ by:

• Prompting-based generation: LLMs suggest normal, edge, and stress cases.
• Tool-based generation: Deterministic generators from the CYaRon library (graphs, trees, polygons) are used with fixed seeds for reproducibility.

Solution Sampling

For each task $q$, $m$ advanced code-reasoning models (e.g., DeepSeek-R1, Qwen3-Thorough) generate candidate chain-of-thought solutions $A_1, ..., A_m$. Static filtering rejects any with AST check failures or incomplete explanations.

Dual Verification

SynthSmith employs a two-step cross-verification protocol:

• Test-output labeling by voting: For test input $x_i$, each solution $A_j$ is executed to yield outputs $y_{ij}$. The provisional ground truth $y_i$ is assigned via plurality vote. Each $(x_i, y_i)$ receives a semantic weight $x_1, ..., x_n$0 based on input difficulty. The candidate test suite $x_1, ..., x_n$1 is thus established.
• Golden solution selection: $x_1, ..., x_n$2 is split into weighted ($x_1, ..., x_n$3) and hold-out ($x_1, ..., x_n$4) sets. Each $x_1, ..., x_n$5’s score is $x_1, ..., x_n$6. The highest-scoring solution on $x_1, ..., x_n$7 is verified for maximal accuracy on $x_1, ..., x_n$8; if confirmed, it is retained as the golden solution $x_1, ..., x_n$9 along with $q$0. Otherwise the task $q$1 is discarded.

Empirically, using $q$2 candidates, the false-positive rate of test-output labeling is $q$3, and the error rate of golden solutions on held-out real tasks is $q$4.

3. Dataset Construction and Training Protocols

Post-verification, SynthSmith materializes:

• SFT Corpus: $q$5 pairs, encompassing 200,000 tasks ($q$6B tokens), trained for 8 epochs.
• RL Corpus: $q$7 instances on 40,000 tasks, supporting $q$8 rollouts per task with 270 PPO/GRPO optimization updates.

Supervised fine-tuning minimizes

$q$9

while reinforcement learning employs Group Relative Policy Optimization (GRPO) with

$m$0

where $m$1 is the importance ratio, $m$2 is the estimated advantage, and $m$3 (rollout reward) is continuous and defined as:

• $m$4 if no code or compile error,
• $m$5 if compiles but fails all tests,
• $m$6 for partial correctness.

4. Scaling Laws and Ablative Analysis

SynthSmith-driven training reveals scaling phenomena along two SFT dimensions: number of unique tasks $m$7 and solutions per task $m$8. For a fixed compute budget:

• Larger $m$9 (e.g., $A_1, ..., A_m$0K to $A_1, ..., A_m$1K tasks $A_1, ..., A_m$2 solution) improves avg@8 from $A_1, ..., A_m$3 on LiveCodeBench.
• For equivalent size, increased task diversity ($A_1, ..., A_m$4K$A_1, ..., A_m$51) outperforms increased solution count per task ($A_1, ..., A_m$6K$A_1, ..., A_m$74 or $A_1, ..., A_m$8K$A_1, ..., A_m$98), indicating that generalization is more strongly driven by problem diversity.

Ablation studies identify:

• Dual verification yields $x_i$0 points over naïve use of raw solutions.
• Long chain-of-thought (Long-CoT) completions (DeepSeek-R1) give $x_i$1 points at v5 compared to Short-CoT.
• Codeforces-style task rendering (70% of data) yields up to $x_i$2-point advantage over alternative formats.
• Tool-based CYaRon test inputs result in $x_i$3 pass rate for golden solutions, versus $x_i$4 for prompting-based inputs.

5. Training Configurations and Empirical Performance

Key SFT hyperparameters: learning rate $x_i$5, batch size $x_i$6, $x_i$7 epochs, training on $x_i$8 GPUs ($x_i$9GB each) over $A_j$0 hours.

RL fine-tuning: batch size $A_j$1, learning rate $A_j$2, rollout temperature $A_j$3, $A_j$4 rollouts, $A_j$5 GRPO/PPO updates on $A_j$6 GPUs for $A_j$7 days.

On LiveCodeBench v5 and v6,

• SFT only: $A_j$8 avg@8 (v5), $A_j$9 (v6)
• SFT+RL: $y_{ij}$0 avg@8 (v5), $y_{ij}$1 (v6)

This surpasses larger real-data models such as DeepCoder‐14B and AReal-boba2-14B (avg@32$y_{ij}$2) while requiring fewer rollouts at test time (8$y_{ij}$3 fewer than Qwen3-8B for equivalent pass@$y_{ij}$4).

6. Implications and Significance

SynthSmith establishes a reproducible methodology for generating high-quality, challenging, and verifiable synthetic data for competitive programming code LLMs. Its feature-based, dual-verified pipeline supplies datasets with tunable scale and granularity, facilitating the systematic study of scaling laws and dataset composition effects in code reasoning. When used to train models such as X-Coder-7B, SynthSmith enables code reasoning performance on par with, or exceeding, substantially larger models trained on real or mixed datasets—without any reliance on human-written programming tasks or tests (Wu et al., 11 Jan 2026).

A plausible implication is that fully synthetic pipelines such as SynthSmith may become foundational for benchmarking and advancing code-centric LLMs, especially as the utility of existing real-world datasets plateaus or as new competitive programming paradigms emerge. The ability to procedurally generate and validate vast, diverse, and high-difficulty task corpora will likely remain essential for future developments in code reasoning research.