--- title: SynthSmith Framework for Code LLMs url: https://www.emergentmind.com/topics/synthsmith-framework type: topic --- # SynthSmith Framework for Code LLMs SynthSmith is a fully synthetic data-generation framework designed to train large language models (LLMs) for code reasoning in competitive programming, independently of human-written corpora. Its architecture procedurally constructs novel problem statements, synthesizes matching solutions, and generates rigorous test suites through an explicitly feature-driven pipeline. SynthSmith demonstrates that high-capacity code LLMs can be developed using exclusively synthetic supervised fine-tuning (SFT) and reinforcement learning (RL) datasets—enabling greater scaling, diversity, and verifiability than possible with real-world competitive programming data. Empirical results show these synthetic datasets are sufficient to achieve state-of-the-art performance on standard code reasoning benchmarks, surpassing models trained on larger real-world corpora [2601.06953]. ## 1. Motivation and Design Rationale Competitive programming tasks require deep algorithmic reasoning, complex solution chains, and robustness to adversarial cases. Real-world benchmarks such as APPS, CodeContests, and TACO are constrained by scale and locked to historic contest distributions, inhibiting further generalization gains. SynthSmith addresses these limitations by: - **Generating novel tasks** via explicit feature evolution, rather than repurposing existing problems. - **Ensuring solution quality** with a dual-verification protocol that cross-examines LLM-generated code against automatically constructed and semantically weighted test suites. - **Enabling dataset scaling** by generating modular SFT and RL corpora whose properties obey empirical scaling laws, and whose scale/diversity can be systematically tuned [2601.06953]. This approach unbinds code reasoning LLMs from the finite pool of real competitive programming tasks, creating an extensible foundation for continual progress. ## 2. Feature-Based Synthesis Pipeline SynthSmith operates in four primary stages, detailed in the following table and expanded upon subsequently: | Stage | Key Technique | Output | |------------------------|----------------------------------------------|-----------------------------| | Task Generation | Feature tree sampling & prompt-based mapping | Code-style statements | | Test-Case Generation | Prompting-based and tool-based generation | Reproducible test inputs | | Solution Sampling | Long-CoT via multiple LLMs | Candidate code solutions | | Dual Verification | Majority voting & weighted evaluation | Golden solution, test suite | ### Task Generation via Feature Trees The feature tree is synthesized by parsing 10,000 TACO competition snippets, with GPT-4-labeled leaves representing concepts such as “Dijkstra’s algorithm,” “segment tree,” or “boundary conditions.” The tree is evolved in breadth (e.g., adding “heap sort” alongside “merge sort”) and depth (e.g., “prefix sum” ↦ “Fenwick tree”), resulting in approximately 100,000 discrete features. A consistent subtree is extracted—one whose features can realistically co-occur in a task—and mapped, through a prompt-driven process, to a structured JSON with: - `feature_roles_tree`—feature leaves tagged as “potential_use,” - `selected_features_tree`—a minimal coherent subset, - `integration_strategy`—a natural language formulation for task design. This representation is rendered to Codeforces, LeetCode, or AtCoder styles via templates. Decoupling feature selection from narrative formulation demonstrably increases task difficulty by ~5 avg@4 points, measured via ablation. ### Test-Case Generation Without ground-truth test sets, SynthSmith synthesizes $n$ input cases $x_1, ..., x_n$ by: - **Prompting-based generation:** LLMs suggest normal, edge, and stress cases. - **Tool-based generation:** Deterministic generators from the CYaRon library (graphs, trees, polygons) are used with fixed seeds for reproducibility. ### Solution Sampling For each task $q$, $m$ advanced code-reasoning models (e.g., DeepSeek-R1, Qwen3-Thorough) generate candidate chain-of-thought solutions $A_1, ..., A_m$. Static filtering rejects any with AST check failures or incomplete explanations. ### Dual Verification SynthSmith employs a two-step cross-verification protocol: - **Test-output labeling by voting:** For test input $x_i$, each solution $A_j$ is executed to yield outputs $y_{ij}$. The provisional ground truth $y_i$ is assigned via plurality vote. Each $(x_i, y_i)$ receives a semantic weight $w_i \in \{1,\ldots,4\}$ based on input difficulty. The candidate test suite $T_\mathrm{cand} = \{(x_i, y_i, w_i)\}$ is thus established. - **Golden solution selection:** $T_\mathrm{cand}$ is split into weighted ($T_\mathrm{gold}$) and hold-out ($T_\mathrm{val}$) sets. Each $A_j$’s score is $S_j = \sum_{(x,y,w) \in T_\mathrm{gold}} w \cdot 1(A_j(x) = y)$. The highest-scoring solution on $T_\mathrm{gold}$ is verified for maximal accuracy on $T_\mathrm{val}$; if confirmed, it is retained as the golden solution $A_\mathrm{golden}$ along with $T_\mathrm{gold}$. Otherwise the task $q$ is discarded. Empirically, using $m=8$ candidates, the false-positive rate of test-output labeling is $<5.3\%$, and the error rate of golden solutions on held-out real tasks is $\approx7.9\%$. ## 3. Dataset Construction and Training Protocols Post-verification, SynthSmith materializes: - **SFT Corpus:** $(q, A_\mathrm{golden})$ pairs, encompassing 200,000 tasks ($3.38$B tokens), trained for 8 epochs. - **RL Corpus:** $(q, T_\mathrm{gold})$ instances on 40,000 tasks, supporting $n=8$ rollouts per task with 270 PPO/GRPO optimization updates. Supervised fine-tuning minimizes $$ J_{\mathrm{SFT}}(\theta) = -\mathbb{E}_{(x, y) \sim D} \sum_{t=1}^{|y|} \log p_\theta(y_t \mid y_{