FrontierSmith: Open-Ended Coding Synthesis

• FrontierSmith is an automated system that converts closed-ended coding challenges into open-ended tasks for advanced LLM evaluation.
• It uses a multi-stage pipeline with evolutionary mutation and divergence metrics to ensure algorithmic diversity and challenge depth.
• The system significantly improves LLM performance by providing scalable, human-like coding benchmarks that highlight continuous optimization.

FrontierSmith is an automated system for synthesizing open-ended coding problems at scale, targeting the development and training of advanced LLM coders. By transforming existing closed-ended competitive programming problems into open-ended variants, FrontierSmith systematically generates problem instances that require solution strategies exhibiting genuine algorithmic diversity. This procedural generation of long-horizon, optimization-based problem data addresses the scarcity of open-ended benchmarks crucial for both training and evaluating LLMs in nuanced, human-like coding tasks (He et al., 14 May 2026).

1. Motivation, Background, and Problem Formulation

The distinction between closed-ended and open-ended coding tasks forms the foundation for FrontierSmith's design. Closed-ended tasks, prevalent in benchmarks like LeetCode and Codeforces, possess discrete pass/fail criteria and a tractable certificate of optimality. They dominate most current datasets, accounting for hundreds of thousands of curated problems. Open-ended tasks instead define continuous optimization objectives (e.g., minimizing makespan, maximizing compression ratio) and allow for a spectrum of solution qualities without a singular optimal answer. Such tasks are notably scarce—FrontierCS contains approximately 240 open-ended problems, and ALE-bench around 40—due to their reliance on expert-defined objectives, verifiers, test case design, and the challenge of guaranteeing no single dominating solution strategy.

This scarcity directly impedes LLM progress on open-ended code generation: while SOTA LLMs achieve over 80% pass rates on closed-ended problems, their performance on open-ended tasks remains substantially below human expert baselines (e.g., Gemini 3.0 Pro at 29.4 vs. human 95.4 on FrontierCS).

2. System Pipeline and Evolutionary Mutation Process

FrontierSmith operates through an iterative, multi-stage pipeline, outlined as Algorithm 1 in the source. This pipeline begins with a large pool of closed-ended programming problems (e.g., 47,136 from HardTests). At each iteration, batches of problems are sampled and mutated into open-ended candidate formulations. The mutation mechanisms encompass:

• Change Goal: Converts decision or exact-answer questions into optimizations (e.g., 2-SAT to Min-True 2-SAT).
• Restrict Outputs: Imposes additional constraints raising the problem's computational hardness (e.g., degree bounds in spanning tree problems).
• Generalize Inputs: Increases problem generality and complexity (e.g., relaxing bipartiteness in maximum independent set).

These mutations alter the problem tuple $(\mathcal{O}, \mathcal{C}_I, \mathcal{C}_O)$—goal and input/output constraints—guaranteeing removal of existing polynomial-time solutions and introduction of continuous objectives.

Candidates are first filtered by LLM-based prompts to ensure they define valid open-ended tasks: unbounded optima, non-trivial heuristic space, and a continuous scoring function. Candidates passing this filter advance to quantitative assessment.

3. Quantitative Idea Divergence Metrics

FrontierSmith introduces the concept of "idea divergence" to evaluate whether newly synthesized problems elicit diverse solution strategies. For candidate problem $c$, the true divergence is: $$d(c) = \Pr_{s_i,s_j\sim\mathrm{Solver}(c)}[\mathrm{strategy}(s_i)\neq\mathrm{strategy}(s_j)].$$ Here, $s_i, s_j$ are solutions sampled from the LLM policy, and $\mathrm{strategy}(\cdot)$ maps a solution to its core algorithmic approach.

There are two primary estimators:

• LLM-Based Divergence: $n$ solutions are paired and classified by an LLM-judge as employing different strategies (1) or not (0), producing: $$\hat{d}_\text{LLM}(c) = \frac{1}{\binom{n}{2}}\sum_{1\le i<j\le n}\mathrm{LLM\text{-}Judge}(s_i,s_j).$$
• Execution-Grounded Divergence: With test instances $T_c$ and verifier $V_c$, each solution is scored to obtain behavioral vectors $\mathbf{q}_i$, leading to: $c$0 This captures behavioral (as opposed to purely syntactic) diversity.

Selection proceeds in two stages: initially by $c$1 (coarse LLM-based filter), then after test/verifier synthesis, by ranking with $c$2 to select the final problems.

4. Test Suite and Verifier Generation

The reliability and validity of synthesized open-ended problems depend on rigorous test suite and verifier construction using two specialized agents:

• Test Case Agent: Generates diverse input classes that stress different strategies (e.g., varying graph densities).
• Verifier Agent: Implements an objective-specific, continuous scoring function $c$3, typically: $c$4 where $c$5 is a baseline solution.

A cross-validation loop ensures test case quality and verifier correctness: solutions are executed through the system, crashes or inconsistencies prompt regeneration or revision, and only candidates that survive this process (approximately 10%) contribute to the final dataset.

5. Dataset Construction, Experimental Setup, and Evaluation

FrontierSmith synthesizes open-ended problems in four rounds. Each round samples $c$6 seeds, produces multiple candidates, filters to the top 100 by $c$7, then to 50 by $c$8, culminating in a set of 200 high-quality problems per experiment. Problem statements average 150–250 words, with input sizes and time/memory limits mirroring existing human benchmarks. The pass rate on validation for test case and verifier infrastructure is approximately 10%.

The experimental protocol evaluates LLMs (Qwen3.5-9B and Qwen3.5-27B) on open-ended benchmarks—FrontierCS (172 algorithmic tasks, continuous scores) and ALE-bench (10 heuristic tasks, Elo-rating). Training employs veRL with GRPO, group size and rollout batch set to 8, over 100 training iterations. Hardware resources are specified per model scale.

Baselines cover both human-curated open-ended problem sets and controls: closed-ended HardTests and random reward assignments.

6. Quantitative Results and Analysis

The synthesized problems from FrontierSmith yield substantial performance gains in LLM training relative to closed-ended data and random controls. For Qwen3.5-9B, the Avg@5 score on FrontierCS improves from 1.80 (base) to 10.62, closely tracking the human-curated set (11.17). On ALE-bench, Elo rating improves by over 300 points (327.22 to 633.58). The 27B model exhibits even greater absolute improvement, surpassing human-curated tasks on several benchmarks.

Ablation studies confirm the critical role of divergence filters; removing them reduces performance (FrontierCS 10.62 → 8.57; ALE-bench 633.6 → 564.4). Both the LLM-based and execution-based divergence metrics cleanly separate open-ended from closed-ended problems (mean $c$9 vs. $$d(c) = \Pr_{s_i,s_j\sim\mathrm{Solver}(c)}[\mathrm{strategy}(s_i)\neq\mathrm{strategy}(s_j)].$$0), validating their efficacy as quality signals.

Agentic code solvers (Claude Sonnet 4.6, GPT-5.5, K2.6) manifest long-horizon behaviors—over 100 interaction turns, and multi-million token use—on FrontierSmith problems, mimicking the complexity and engagement of human-curated open-ended tasks.

7. Limitations, Discussion, and Future Prospects

Current iteration of FrontierSmith focuses on self-contained, single-file algorithmic tasks and does not yet address multi-file project structures, GPU kernel-level optimization, or system-level benchmarks. RL training is conducted with single-turn policies, and agentic RL (multi-turn tool-use and interaction) remains to be explored. Approximately 90% of mutated candidates fail at the test/verifier infrastructure step, indicating opportunities for more robust environment construction.

Potential future directions include synthesis of more complex multi-file or system environments (e.g., Docker-like), integration of agentic RL for multi-turn planning and exploration, improvement of test/verifier generation via ensemble or program analysis techniques, and curriculum-based bootstrapping from earlier generations to foster a self-improving ecosystem of open-ended code challenges.

FrontierSmith demonstrates that principled, automated transformation and selection can convert closed-ended programming seeds into high-quality, open-ended tasks suitable for scalable LLM training and evaluation, reducing the dependency on costly human curation while advancing the development of capable, generalist coding agents (He et al., 14 May 2026).