Distribution-Aware Algorithm Design with LLM Agents

Abstract: We study learning when the learned object is executable solver code rather than a predictor. In this setting, correctness is not enough: two solvers may both return valid solutions on the deployment distribution while differing substantially in runtime. Given samples from an unknown task distribution, the learner returns code evaluated on fresh instances by both solution quality and execution time. Our central abstraction is a \emph{solver hint}: reusable structure inferred from samples and compiled into specialized solver code. We prove that the empirically fastest sample-consistent solver from a fixed library generalizes in both correctness and runtime, and that statistically identifiable hints can be recovered and compiled from polynomially many samples. Empirically, we instantiate the framework with LLM code agents on (21) structured combinatorial-optimization target distributions across seven problem classes. The synthesized solvers reach mean normalized quality (0.971), improve by (+0.224) over the average heuristic pool and by (+0.098) over the highest-quality heuristic, and are (336.9\times), (342.8\times), and (16.1\times) faster than the quality-best heuristic, Gurobi, and the selected time-limited exact backend, respectively. On released PACE 2025 Dominating Set private instances, the synthesized solver is valid on all (100) graphs and runs about two orders of magnitude faster than top competition solvers, with a moderate quality gap. Inspection shows that many gains come from changing the computational scale: replacing ambient exponential search or general-purpose optimization with compiled distribution-specific computation.

• The paper introduces distribution-aware program learning that synthesizes specialized solver code, ensuring both correctness and computational efficiency.
• It develops a runtime-aware ERM framework and sample-efficient hint recovery, proving exponential runtime speedups for combinatorial optimization problems.
• LLM agents iteratively propose, analyze, and refine structural hints, achieving orders-of-magnitude improvements over standard heuristic solvers.

Distribution-Aware Program Synthesis with LLM Agents

Problem Formulation

The paper proposes a formal and empirical framework for distribution-aware program learning, in which the learning output is not a classifier or regressor, but executable solver code. The central focus is on combinatorial optimization and decision problems—typified by graph algorithms, constraint satisfiability, packing, and routing—where correctness alone does not suffice: solutions must also be optimized for computational efficiency on future, unseen instances drawn from the deployment distribution.

Crucially, two algorithms may be correct (i.e., always return valid solutions) but differ by orders of magnitude in runtime. Therefore, the central theoretical object is distribution-aware generalization in both correctness and computation. The proposed abstraction is a solver hint: a reusable, sample-inferred structural summary, which is compiled into specialized solver code. This structure-driven approach enables the generation of solvers with distribution-specific amortized efficiency.

Theoretical Framework

The authors provide a rigorous treatment of two regimes: selection from a fixed solver library and synthesis via learnable solver hints.

Runtime-aware Empirical Risk Minimization (ERM):

When restricted to a finite library of candidate solvers, the runtime-aware ERM rule selects the fastest correct solver on sampled data. The main theorem bounds generalization error and expected deployment runtime, showing that with high probability, the empirically fastest sample-consistent solver approaches the class-optimal deployment runtime up to standard statistical error terms in sample size and library cardinality:

• For any finite solver class $\mathcal{C}$, with $n$ samples, excess deployment runtime is $O(T_{\max}\sqrt{\frac{\log|\mathcal{C}|}{n}})$, where $T_{\max}$ bounds per-instance runtime.

Sample-Efficient Hint Recovery:

Beyond fixed libraries, the effective search is over a hint space $\mathcal{H}$ (structural abstractions), rather than over solvers directly. Under a realizability assumption (deployment instances generated under a true but unknown $h^*\in\mathcal{H}$), the paper proves that, given a score margin $\gamma>0$, the underlying hint is identifiable from $O(\log |\mathcal{H}|/\gamma^2)$ samples. The recovered hint is then compiled into code that amortizes structure-specific computational speedups.

A formal example is provided for SAT with planted backdoors; with sufficiently many samples, the structural shortcut (e.g., backdoor variable set) can be learned and compiled, yielding an exponential speedup in instance solution time, while fallback guarantees correctness.

Synthesis Methodology via LLM Agents

The framework is instantiated using LLMs as code agents. The synthesis loop is a non-trivial, iterative proposal-and-selection pipeline:

• Stage 1: LLM proposes a hypothesis describing suspected distributional structure (e.g., motif, backdoor, resource bottleneck) in natural language.
• Stage 2: LLM writes an analysis program to estimate and summarize evidence (statistics) supporting the hypothesis from the training set.
• Stage 3: LLM generates deployment solver code conditioned on the recovered evidence/hint.
• Selection: Candidates are evaluated and ranked on validation data by normalized solution quality, optimality rate, and empirical runtime.

A diversity-preserving beam search maintains multiple competing hypotheses and solver implementations throughout successive refinement and search, allowing recovery from spurious shortcuts and expansion of the hint search space.

Empirical Results

Benchmark and Evaluation Protocol

The benchmark comprises 21 distribution families across seven canonical NP-hard or combinatorial classes: Graph Coloring, MAXSAT, Maximum Independent Set (MIS), Minimum Dominating Set (MDS), Packing LP, Multidimensional Knapsack (MDKP), and Euclidean TSP. Each target distribution exhibits nontrivial, recurring cross-instance structure but is hidden from learners (no access to generation rules or optima during synthesis).

Baseline comparisons include:

• Domain heuristics (e.g., DSATUR, greedy covers, LKH, density sorting)
• Solver-backed (e.g., Gurobi, OR-Tools, PySAT, HiGHS)
• Time-limited exact methods

Metrics are normalized quality (relative to optimum), optimality fraction, and instance runtime.

Headline Quantitative Results

Synthesized LLM solvers demonstrate substantial improvements in the quality-runtime frontier:

• Mean normalized quality: 0.971 (on scale where 1.0 = optimum)
• Improvement over average heuristic: +0.224
• Improvement over best heuristic: +0.098
• Mean per-instance speedups:
  • Over best heuristic: 336.9x
  • Over Gurobi (10s budget): 342.8x
  • Over time-limited exact solver: 16.1x

On PACE 2025 Dominating Set challenge instances, synthesized solvers validated on all test cases and achieved 99x–109x lower runtime than the top competition solvers, with a moderate (~3.3%) solution size gap.

Mechanistic Findings

Inspection and ablation studies reveal that gains are not mere implementation artifacts but arise from computational scale reduction: the learned solvers replace ambient exponential or generic optimization procedures with distribution-specific computation (e.g., template checking, kernelization, motif decomposition, surrogate scoring, restricted DP, structured candidate generation).

Success is predicated on discovering structure that is truly reusable (not per-instance heuristic) and can trigger fast paths for most future instances from the same distribution.

Iterative Synthesis and Robustness

Runtime gains are often realized not in the initial LLM proposal, but cumulatively over multiple synthesis iterations, as the agent refines or replaces structural hypotheses based on validation performance and failure cases.

Ablation with zero-sample (no distributional access) synthesis shows that while normalized quality may saturate, sample-conditioned synthesis yields faster specialized solvers in most distributions, confirming the efficacy of explicit structure discovery from samples.

Graph relabeling perturbations suggest the main source of variance is presentational (node ordering etc.), with feasibility robust but some quality and speed artifacts traceable to brittle exploitation of presentation-specific clues in a minority of cases.

Implications and Future Directions

Practical Implications

The evidence supports the feasibility of sample-driven amortized algorithm design by LLM agents, at least for regimes where the task distribution supports recurring, discoverable structure. This paradigm can lower the data and expertise barrier for generating specialized solver code (e.g., for scientific, engineering, or logistics deployments), reduce long-term computational cost on repeated workloads, and accelerate solution pipelines.

However, the methodology is not robust to arbitrary input distributions; performance degrades under significant distributional shift, and one-time synthesis costs must be amortized over sufficient test instances. The stochastic nature of the proposal loop naturally yields variability—the system is best deployed with strong validation, fallback, and human oversight.

Theoretical Implications

This work operationalizes the notion that average-case tractability reflects not only correctness but data-induced reductions in deployed computation. Rather than worst-case guarantees, the synthesized solvers serve as constructive witnesses to distribution-specific complexity reductions—complementing classical analyses that rely on closed-form distributional characterizations.

Conclusion

The presented work formalizes and demonstrates distribution-aware algorithm synthesis as a tractable, sample-efficient, and empirically robust approach for producing deployment-specialized solvers with both high solution quality and substantial runtime improvements. By leveraging LLMs to generate, select, and compose code based on inferred reuse structure, the methodology achieves orders-of-magnitude speedups and approaches optimality on diverse structured optimization tasks.

The synthesis loop—sample to hint to specialized solver—embodies a shift toward programmatic amortization of algorithm design, with broad implications for automated reasoning, meta-algorithmics, and adaptive computation. Continued challenges include stability, robustness to perturbation and distribution shift, and principled control of inductive bias and diversity in the synthesis process.