LLM-Based Synthesis Methods

• LLM-Based Synthesis Approach is a method that integrates pre-trained language models with symbolic and probabilistic search to generate and refine structured digital artifacts.
• The approach employs techniques like A* search and weighted PCFGs to achieve significant performance improvements, such as an 80.1% solve rate in program synthesis benchmarks.
• Applications span various fields including program synthesis, hardware design, retrosynthesis in chemistry, mathematical problem generation, and automated knowledge extraction.

LLM-Based Synthesis Approach refers to a broad class of methods in which the synthesis of formal programs, hardware, scientific knowledge, mathematical problems, or other structured digital artifacts is performed by, or in collaboration with, LLMs. In contrast to purely symbolic or domain-specialized enumeration techniques, LLM-based synthesis leverages pre-trained or fine-tuned LLMs to generate, guide, or optimize the construction process. This approach spans foundational program synthesis, knowledge extraction, chip design, program compositionality, chemistry, and more. The defining feature is the integration (at various levels) of the generative capacities, priors, and reasoning capabilities of LLMs into the synthesis pipeline.

1. Enumerative and Probabilistic Program Synthesis with LLM Guidance

LLM-based synthesis frameworks in program synthesis often combine classical enumerative search with probabilistic guidance derived from LLM output. In "Guiding Enumerative Program Synthesis with LLMs" (Li et al., 6 Mar 2024), the synthesis process begins by prompting a pre-trained LLM (e.g., GPT-3.5) for candidate solution programs given a formal specification, typically in a DSL such as those used in SyGuS benchmarks. If the LLM’s one-shot solution is incorrect, the derivation trees (production rules) of these candidates are harvested to build a weighted probabilistic context-free grammar (pCFG). Rule weights are calculated as

$$w[r_i] = \sum_{\text{prog}_i} |r_i| \text{ in } D_{\text{prog}_i},$$

where $|r_i|$ is the count of rule $r_i$ in the left-most derivation of candidate $\text{prog}_i$. After normalization, rule probabilities drive an enumeration using either categorical sampling or A*-guided search with cumulative and heuristic costs:

$$c(x) = \sum_{r \in D_x} -\log_2 P[r] \qquad g(x) = \begin{cases} 0, & x \text{ is terminal} \ \text{heuristic}, & \text{otherwise} \end{cases}$$

with search traced using

$f(x) = c(x) + g(x).$

Syntactic feedback loops allow the enumerator to periodically prompt the LLM with partial solutions and counterexamples, updating the grammar and refining the search over time. This hybrid approach achieves a benchmark-solving rate of 80.1% (LLM ∪ A*-) on SyGuS, outstripping both stand-alone LLM (49%) and state-of-the-art enumerators (e.g., cvc5 at 68.1%).

2. Hybrid Surrogate-Guided Synthesis and Context-Free Approximation

The HySynth approach (Barke et al., 24 May 2024) combines LLM-generated completion samples—not necessarily fully correct but rich in domain-relevant fragments—with bottom-up enumerative synthesis over a weighted PCFG surrogate. For a PBE task, an LLM is prompted to generate many candidate programs using DSL grammar, IO examples, and in-context demos. All derivations are parsed; each rule’s relative frequency

$$p(R) = (count(R) + \alpha) / \sum_{R'\in R(N)}(count(R') + \alpha)$$

yields the PCFG. The search then proceeds via dynamic programming, organizing programs by cost $-\log p(P)$. Empirically, the PCFG guidance focuses the search on promising regions, producing program enumeration speedups (e.g., 4× on ARC puzzles, 5× over pure search for SyGuS strings) and markedly higher task success rates compared to both direct LLM completions and symbolic methods without guidance.

3. LLM-Driven Compositional and Tool-Augmented Synthesis Strategies

For challenging program synthesis tasks beyond direct end-to-end completion, LLMs have been deployed for compositional and repair strategies (Khan et al., 12 Mar 2025). When self-reflection fails on a PBE problem, partial correctness of an LLM-generated first candidate $F_1$ can be leveraged. Systematic forward or backward decomposition identifies reusable prefixes/suffixes or partitions IO examples, with subtasks defined accordingly:

• Forward1: Salvage prefix/executed intermediates and recursively synthesize the remainder.
• Backward1: Invert the last operation, synthesize code that would produce observed intermediates.
• IfThenElse: Partition examples, synthesize branches, and recombine with synthesized conditional logic.

The approach formally models program composition using

$F = F_1 \circ F_2,$

with semantics $[F](I) = F_2(F_1(I))$, and parallel composition for conditional branches:

$$[F_1 \parallel_{c} F_2](I) = \text{if } c(F_1(I)) \text{ then } F_1(I) \text{ else } F_2(I).$$

These methods improve solution rates for hard benchmarks beyond iterative error-feedback schemes.

4. LLM Synthesis Applications: Hardware, Chemistry, Mathematics, and Knowledge Domains

LLM-based synthesis has proven effective across hardware design, scientific literature, chemistry, mathematical reasoning, and more:

• HDL/RTL Synthesis and Hardware Design: Hierarchical prompting and agentic pipelines, exemplified by ROME (Nakkab et al., 23 Jul 2024), automate decomposition, HDL code generation, and integration, reducing latency and cost versus flat prompting. Closed-loop frameworks like MCP4EDA (Wang et al., 25 Jul 2025) leverage LLMs to adapt TCL scripting via backend-aware feedback, closing the gap between pre-layout synthesis estimates and post-layout realities.
• Retrosynthesis and Molecule Design: In LLM-Augmented retrosynthesis (Wang et al., 11 May 2025), LLMs generate route-encoded, multi-step synthesis plans. Population-based evolutionary search accepts, mutates, and corrects complete retrosynthetic routes based on molecule- and reaction-level validity, leveraging partial rewards tied to synthetic complexity. Empirically, this holistic strategy outperforms both single-step LLM predictors and combinatorial search.
• Mathematical Dataset Generation: Synthesis by Design (Xu et al., 9 Jun 2025) structures mathematical reasoning as code with explicit step annotations. Structural interventions at the graph level (i.e., intervention $t_i' = f(t_i)$) yield new harder problems with longer reasoning chains. Fine-tuning on the resulting dataset improves LLM handling of multi-step reasoning.
• Automated Knowledge Extraction: Pipelines such as KEP (Silva et al., 5 Nov 2024) and domain-specific workflows in materials (Shi et al., 6 Aug 2024, Okabe et al., 28 Oct 2024) use LLMs for paragraph/relevance categorization and precise extraction of synthesis details, often tuned by human-AI interactive curation or prompt-based data selection.
• Conversational Data Synthesis for Recommender Systems: Active data augmentation (Surana et al., 21 Apr 2025) synthesizes domain-specific conversational training samples by selecting informative seeds using Jensen-Shannon (JS) divergence and Fisher information, with LLMs generating realistic dialogues and recommendations for fine-tuning smaller CRS models in data-scarce settings.

5. Technical Foundations and Probabilistic Modeling

LLM-based synthesis systems heavily exploit probabilistic modeling and search:

• Production rule weighting and probabilistic context-free grammars (PCFGs) learned from LLM output guide candidate enumeration and search cost computation.
• Smoothing, e.g.,

$$w'[\alpha \rightarrow \beta] = 10 \left(\frac{w[\alpha \rightarrow \beta]+1}{10}\right)^\gamma$$

with $\gamma = 0.4$, mitigates zero probabilities in sparse domains.

• A* search for symbolic synthesis and dynamic programming in bottom-up searches are rigorously adapted to leverage LLM-derived priors.

For chemical and mathematical synthesis, vectorized representations and similarity measures such as generalized Tanimoto similarity,

$$T_s(S_1, S_2) = \frac{1}{2}\left[T_s(R_1, R_2) + T_s(P_1, P_2)\right],$$

allow for semantically meaningful evaluation of LLM-generated outputs, especially in multi-component or symmetrically structured domains.

6. Limitations and Active Disambiguation

Despite substantial advances, LLM-based synthesis suffers from persistent limitations:

• Standalone LLMs often generate “plausible but incorrect” artifacts, particularly for rare constructs or when semantic requirements are strict.
• Non-determinism and training-data gaps introduce reproducibility and generalization challenges.
• For incremental or configuration-based synthesis (e.g., network ACLs), ambiguity in user intent regarding update integration (e.g., rule ordering) can lead to drastic semantic shifts. Systems such as Clarify (Mondal et al., 16 Jul 2025) augment LLMs with automated disambiguators, using binary search over candidate global behaviors with user feedback to ensure correct integration.

7. Impact, Applications, and Future Research Directions

LLM-based synthesis approaches have substantial impact across domains:

• Compelling performance gains in formal program synthesis (SyGuS, API, DSLs), complex hardware design, and scientific and biomedicine knowledge integration.
• New paradigms, including closed-loop design space exploration (MCP4EDA) and agentic graph synthesis (GraphMaster (Du et al., 1 Apr 2025)), extend LLM capabilities to manage complex toolchains or collaborative agent settings.
• Future research aims for tighter semantic coupling between LLM and symbolic search, improved feedback loops, scaling smoothing and rule weighting strategies, and richer cross-domain applications (multimodal synthesis, high-stakes scientific reasoning, privacy-aware conversational data generation).

A concise comparison of representative synthesis frameworks and their primary characteristics is shown below:

Synthesis Domain	Key LLM Integration	Performance Gains / Highlights
Program Synthesis (SyGuS)	Weighted pCFG, A* search, CEGIS	80.1% solve rate, outperforms cvc5 and LLM-alone
DSL/Structured Synthesis	PCFG from LLM, bottom-up DP search	4×–5× enumeration speedup, higher success rates
Hardware (HDL/RTL)	Hierarchical prompting, closed-loop	Pass@k↑, 15–30% timing reduction (MCP4EDA)
Retrosynthesis (Chemistry)	Route-level LLM, evolutionary search	90–100% solve rate on benchmark datasets
Math Reasoning Dataset	Code as structure, interventions	Improved reasoning on problems with long solutions
Config Synthesis (Networking)	LLM with disambiguator module	Resolution of integration ambiguities, formal safety

LLM-based synthesis thus represents a convergence of generative modeling, probabilistic symbolic search, and task-specific orchestration, establishing new foundations for automated solution generation in formally and semantically rich digital domains.