CYANEA: Learning-based Program Synthesis

• CYANEA is an online learning-based system for program synthesis that dynamically selects between symbolic solvers and LLM-prompt pairs.
• It frames solver selection as a contextual multi-armed bandit problem, leveraging past query outcomes to optimize under time and cost constraints.
• Evaluated on SyGuS benchmarks, CYANEA solves 37.2% more queries than the best single solver and operates within 4% of the virtual best solver.

CYANEA is an online, learning-based system for program synthesis that selects, on the fly, which symbolic solver, LLM, and prompt combination to deploy for a given synthesis query under explicit time and cost budgets. It frames solver and prompt selection as a contextual multi-armed bandit problem in which each arm is either a symbolic solver or an LLM-prompt pair, and it learns from prior queries which configurations are most effective for which classes of tasks. The system is evaluated on SyGuS competition benchmarks, ranking function synthesis benchmarks, and SMT-derived synthesis queries, all represented in SyGuS-IF; on these workloads, it solves 37.2% more queries than the best single solver and achieves results within 4% of the virtual best solver (Li et al., 9 Jan 2025).

1. Motivation and scope

CYANEA is motivated by heterogeneity in program synthesis performance across solver families, model choices, and prompting styles. In the setting studied, practitioners may choose among symbolic solvers, multiple LLMs, and multiple prompting styles, including natural-language prompting, few-shot prompting, role prompting, emotional stimuli, and multi-stage prompting through a Lisp intermediate. Different combinations dominate on different problem families: on some benchmarks, a symbolic enumerative solver is markedly better than LLMs; on others, a particular LLM combined with a particular prompt template is superior. These choices also induce different API costs, token usage, and latency (Li et al., 9 Jan 2025).

The target setting is a stream of synthesis queries presented over time. A non-expert user must decide whether to invoke an LLM or a symbolic solver, which LLM to call, and how to prompt it. Poor choices reduce the number of solved queries and increase elapsed time and financial cost. CYANEA addresses this by learning a query-conditioned selection policy rather than relying on a single fixed solver or a single fixed prompt. The system is applied to three sources of synthesis queries: classic syntax-guided synthesis competition problems in SyGuS-IF, ranking function synthesis benchmarks, and fresh synthesis tasks generated from SMT problems. In all three cases, the common representation is a SyGuS-IF specification with an SMT background theory, a function to synthesize, and a logical correctness constraint (Li et al., 9 Jan 2025).

2. Formalization as contextual bandit optimization

A synthesis query is defined as a tuple

$q = \langle \tau, f, \phi \rangle,$

where $\tau$ is the background theory, $f$ is the function to synthesize, and $\phi$ is a quantifier-free correctness formula. A valid solution is a body for $f$ such that

$\tau \models \forall x.\ \phi(f).$

For a candidate $f$, correctness is checked by SMT through the unsatisfiability of

$\exists x.\ \neg \phi(f).$

The user presents a sequence of queries

$Q = \{q_1, \ldots, q_m\}$

and a set of solvers

$S = \{s_1, \ldots, s_n\},$

where each $\tau$0 is either a symbolic solver or an LLM-prompt pair. Given per-query budgets for time $\tau$1 and cost $\tau$2, the objective is to produce correct solutions for as many queries as possible while respecting those budgets and maximizing a reward function (Li et al., 9 Jan 2025).

Each arm in the multi-armed bandit corresponds to a solver configuration. In the single-layer variant, the arms are all symbolic-solver and LLM-prompt combinations. In the multi-layer variant, a first-layer bandit chooses among base models, and a second-layer bandit associated with each LLM chooses among that model’s prompt styles. For each query, the system selects an ordered list of arms and allocates time and token budgets to them.

Three reward functions are instantiated. The time-focused reward is

$\tau$3

where $\tau$4 is the total time used on the query. The cost-focused reward is

$\tau$5

where $\tau$6 is the estimated token cost, with

$\tau$7

for each LLM call, and the enumerative solver assigned a fixed small cost $\tau$8. The binary reward is

$\tau$9

These reward definitions encode different priorities: success rate, solve time, or API cost (Li et al., 9 Jan 2025).

3. System architecture and online learning procedure

For each query, CYANEA executes a fixed pipeline: it featurizes the SyGuS-IF input, predicts a ranking of solvers using a contextual bandit, allocates cost and time budgets across the ranked solvers, deploys the solvers sequentially, validates candidate solutions with SMT, and updates its internal state with the observed outcome. The main components are an input parser, a featurizer, a predictor, a cost allocator, a time allocator, and a deploy module. The deploy module runs each solver under the allocated budgets, stops when a correct solution is found or the budgets are exhausted, and records whether the solver succeeded, how much time it used, how much cost it incurred, and the resulting reward (Li et al., 9 Jan 2025).

The predictor is a $f$0-nearest-neighbor contextual multi-armed bandit rather than a linear contextual bandit such as LinUCB. The justification given is that solver performance is not expected to vary linearly with the feature vector, whereas $f$1-NN provides a simple nonparametric alternative. The system maintains a database of past queries with their feature vectors, the solvers that solved them, and the observed rewards. For a new query with feature vector $f$2, it finds the $f$3 nearest neighbors in Euclidean feature space. For each solver $f$4, the score is

$f$5

where $f$6 is the reward obtained when $f$7 solved neighbor $f$8. Solvers are ranked in descending order of this score, and any solver absent from the relevant neighbors is appended in random order to preserve exploration. When an LLM-prompt pair solves a new query, that query and its reward are added to the database (Li et al., 9 Jan 2025).

Time and cost allocation are modeled probabilistically. For a given solver, observed costs $f$9 are assumed to follow an exponential distribution with rate $\phi$0, giving log-likelihood

$\phi$1

and maximum-likelihood estimate

$\phi$2

Using the exponential cumulative distribution function $\phi$3, the system chooses a per-solver allocation $\phi$4 such that

$\phi$5

which yields

$\phi$6

An analogous procedure is used for runtime to obtain $\phi$7. These estimates are contextual because they are computed from the $\phi$8 nearest neighbors for each solver. Budgets are assigned greedily down the ranked list until the total reaches the global limits; lower-ranked solvers may therefore receive zero budget (Li et al., 9 Jan 2025).

The implementation uses a total time budget $\phi$9 seconds, a total cost budget $f$0 in the specified token units, $f$1, up to 16 LLM attempts per prompt style, and cvc5 for SMT checking. For LLM solvers, after each incorrect attempt, the SMT counterexample or error is reported back to the model in the next prompt iteration (Li et al., 9 Jan 2025).

4. Solver portfolio and prompt design

The solver portfolio consists of two LLMs and one symbolic enumerative solver. The LLMs are GPT-3.5-Turbo-0125, referred to as gpt, and Meta-LLaMA-3-70B, referred to as llama. The symbolic component is a CEGIS-style enumerator with A*-based search over a grammar covering the full logic. For each LLM, six distinct prompt styles are defined, producing arms such as gpt-p1 through gpt-p6 and llama-p1 through llama-p6; together with the enumerator, these constitute the bandit’s action set (Li et al., 9 Jan 2025).

The prompt library is constructed from several commonly studied strategies. One strategy translates the SyGuS-IF logical constraints into natural language. Another uses few-shot prompting with three previously solved synthesis examples placed before the new problem. A third uses a higher-resource programming-language intermediate: because SyGuS-IF is relatively rare in training data, the system first asks the LLM to solve the function in Lisp and then to convert the result into SMT-LIB or SyGuS-IF. The reported prompts are: “Solve the following function 'solution' with Lisp… write one Lisp-like defun…” and “Please convert the Lisp function you generated into SMT-LIB format… (define-fun …) …”. The paper reports that Lisp $f$2 SyGuS is more robust than Python $f$3 SyGuS. Additional prompt dimensions include role prompting via the prefix You are a good program synthesizer and emotional stimuli via an appended instruction emphasizing the importance of not failing (Li et al., 9 Jan 2025).

Rather than exploring all possible combinations of these dimensions, the system fixes six prompt styles per LLM, each corresponding to a specific on/off combination of natural language versus direct logical specification, Lisp intermediate versus direct SyGuS, role prompt, emotional tail, and few-shot examples. The arm set is fixed throughout execution; dynamic addition or removal of arms at runtime is not implemented. This fixed-portfolio design makes solver selection the central adaptive mechanism rather than prompt generation or portfolio management.

5. Empirical evaluation

The evaluation uses 1269 synthesis queries drawn from three sources: SyGuS competition benchmarks, ranking function synthesis benchmarks, and SMT-derived synthesis queries. These tasks span linear integer arithmetic, bitvectors, programming-by-example, and invariant or ranking-function synthesis, and all are represented in SyGuS-IF syntax. Learning is strictly online: there is no offline train/test split. For fairness, the 1269 queries are shuffled uniformly at random and the experiment is repeated for 20 runs, with the bandit and budget models learning sequentially within each run (Li et al., 9 Jan 2025).

Baselines include each individual LLM-prompt arm, the enumerative solver alone, a virtual best solver that selects the highest-reward solver for each query with oracle knowledge, CYANEA with equal time and cost allocation across solvers, and both single-layer and double-layer $f$4-NN architectures under each reward function. Reported metrics are percentage solved, number solved, Par-2, total reward for $f$5 and $f$6, average time per query, and average cost per query. Par-2 is defined as

$f$7

so lower values are better (Li et al., 9 Jan 2025).

The virtual best solver solves 91.8% of queries, corresponding to 1165 of 1269, with Par-2 equal to 23,596. The best single solver is llama-p4, which solves 64.3% of queries, or 816 of 1269, with Par-2 approximately 95,251. The best CYANEA configuration is the single-layer $f$8-NN system with cost-based reward $f$9, denoted “Single k-NN ($\tau \models \forall x.\ \phi(f).$0)”, which solves 88.3% of queries on average, or $\tau \models \forall x.\ \phi(f).$1 out of 1269, with Par-2 equal to 37,636, total $\tau \models \forall x.\ \phi(f).$2 reward approximately 1008.7, total $\tau \models \forall x.\ \phi(f).$3 reward approximately 904.4, average time approximately $\tau \models \forall x.\ \phi(f).$4 seconds per query, and average cost approximately $\tau \models \forall x.\ \phi(f).$5 (Li et al., 9 Jan 2025).

These results quantify two headline comparisons. Relative to the best single solver, the gain is

$\tau \models \forall x.\ \phi(f).$6

which is the reported 37.2% increase in solved queries. Relative to the virtual best, the ratio

$\tau \models \forall x.\ \phi(f).$7

corresponds to 96.1% of oracle solved-count performance, which is the basis for the statement that CYANEA is within 4% of the virtual best solver. CYANEA also improves Par-2 substantially over the best single solver, from approximately 95,251 to 37,636 (Li et al., 9 Jan 2025).

Across configurations, single-layer $\tau \models \forall x.\ \phi(f).$8-NN consistently outperforms the double-layer variant. For example, single-layer $\tau \models \forall x.\ \phi(f).$9 solves 88.3% of queries, whereas double-layer $f$0 solves 84.5%, or $f$1, and exhibits significantly larger variance. The paper attributes this to data sparsity in the lower-layer prompt bandits, which only receive examples after the upper layer selects the corresponding base model. The MLE-based budget allocator also improves over equal allocation. With linear budgets, single-layer $f$2-NN solves about 87.0% of queries, or 1104, rather than 88.3%, and yields Par-2 around 39,072 rather than 37,636. Double-layer with linear budgets drops to 71.8% solved, or $f$3. Among individual arms, llama-p4 is the strongest at 64.3% solved, gpt-p4 solves 54.3%, llama-p2 solves 35.0%, and the enumerative solver alone solves 52.2% with cost $f$4 and average time $f$5 seconds (Li et al., 9 Jan 2025).

6. Limitations, interpretation, and extensions

Several limitations are explicit. CYANEA assumes a fixed portfolio of solvers and prompts, without autonomous addition, pruning, or adaptation to newly introduced models. The contextual bandit also assumes an approximately stationary environment over the query stream. Its context representation is described as mostly syntactic, including keyword frequencies, query length, constant counts, and logic type, rather than richer structural or semantic representations. The time and cost models rely on exponential distributions, which are convenient but may be mismatched to heavier-tailed or multimodal empirical behavior. The $f$6-NN bandit is presented as a heuristic method without explicit regret guarantees. The LLM set is limited to GPT-3.5 and LLaMA-3, and the experiments use comparatively generous budgets of 100 seconds and 100,000 token-units per query (Li et al., 9 Jan 2025).

The reported ablations suggest several technical interpretations. The superiority of the single-layer selector over the multi-layer alternative suggests that, for a portfolio of two LLMs, six prompts per LLM, and one enumerator, direct joint ranking of all arms is more data-efficient than hierarchical decomposition. The performance gap between MLE-based and linear allocation suggests that budget estimation is not merely a secondary engineering detail but materially affects solved count and Par-2. The individual-arm results also indicate that prompt choice is a substantive variable rather than a marginal perturbation: differences among $f$7 through $f$8 are large enough that automatic prompt selection becomes worthwhile. This suggests that program synthesis performance depends not only on model identity but also on the interaction between model, prompt template, and query family (Li et al., 9 Jan 2025).

Future directions stated or implied include richer contextual bandits, better featurization through semantic features or learned embeddings of specifications, dynamic portfolio management, and application of the same framework beyond program synthesis to domains such as theorem proving, code repair, and static analysis. The system could also be integrated with additional SyGuS solvers, including bottom-up enumeration or hybrid learning-plus-enumeration methods, provided they accept SyGuS-IF. More refined cost models based on real API pricing, energy consumption, or cluster load are also natural extensions. In that broader perspective, CYANEA functions as a meta-solver: it does not replace symbolic synthesis or LLM-based synthesis, but learns how to orchestrate their complementary strengths under explicit resource constraints.