A2DEPT: Large Language Model-Driven Automated Algorithm Design via Evolutionary Program Trees

Abstract: Designing heuristics for combinatorial optimization problems (COPs) is a fundamental yet challenging task that traditionally requires extensive domain expertise. Recently, LLM-based Automated Heuristic Design (AHD) has shown promise in autonomously generating heuristic components with minimal human intervention. However, most existing LLM-based AHD methods enforce fixed algorithmic templates to ensure executability, which confines the search to component-level tuning and limits system-level algorithmic expressiveness. To enable open-ended solver synthesis beyond rigid templates, we propose Automated Algorithm Design via Evolutionary Program Trees (A2DEPT), which treats LLMs as system-level algorithm architects. A2DEPT explores the vast program space via a tree-structured evolutionary search with hybrid selection and hierarchical operators, enabling iterative refinement of complete algorithms. To make open-ended generation practical, we enforce executability with a lightweight program-maintenance loop that performs feedback-driven repair. In experiments, A2DEPT consistently outperforms representative LLM-based baselines on both standard and highly constrained benchmarks. On the standard benchmarks, it reduces the mean normalized optimality gap by 9.8% relative to the strongest competing AHD baseline.

• The paper introduces A2DEPT, presenting complete solver design via evolutionary program trees to move beyond traditional heuristic tuning.
• It applies a tree-structured search with adaptive operator scheduling and a maintenance loop that ensures robustness and executability of synthesized algorithms.
• Empirical results reveal significant reductions in optimality gaps on benchmarks such as CVRP and MIS, outperforming existing LLM-based heuristic methods.

LLM-Driven Automated Algorithm Design via Evolutionary Program Trees

Motivation and Problem Statement

Automated algorithm design for combinatorial optimization problems (COPs) has historically relied on handcrafted heuristics, a process that is both labor-intensive and sensitive to domain intricacies. While recent research has successfully leveraged LLMs for Automated Heuristic Design (AHD), most prior approaches restrict LLMs to component-level tuning within fixed solver templates to guarantee executability, thus limiting system-level innovation and generalizability. The paper proposes A2DEPT (Automated Algorithm Design via Evolutionary Program Trees), a novel framework where LLMs synthesize complete solver programs, allowing a paradigm shift from template-bound AHD toward genuinely open-ended Automated Algorithm Design (AAD).

Figure 1: Framework dependence and executability challenges in scaling from heuristics to full solver logic for the Traveling Salesman Problem.

Methodological Innovations

A2DEPT introduces a tree-structured evolutionary search strategy in program space, combining hybrid selective pressure with hierarchical variation operators. The system maintains a global search tree, expanding frontier nodes in batches by coupling simulated annealing-based primary selection with Boltzmann sampling to preserve trajectory diversity. Three operator classes are dynamically scheduled: localized micro-tuning, macro-mutation for structural rewrites, and semantic crossover for recombination. To render open-ended synthesis practical, A2DEPT implements a lightweight program maintenance mechanism that performs dependency analysis, iterative repair, and code pruning to enforce run-time executability.

Figure 2: A2DEPT enables program-level evolutionary search and executability maintenance, departing from template-bound heuristic design.

Numerical Results and Empirical Analysis

A2DEPT is empirically validated across six NP-hard combinatorial optimization benchmarks, including classic and highly constrained problems such as Maximum Independent Set (MIS), Capacitated Vehicle Routing Problem (CVRP), Flexible Job-Shop Scheduling (FJSP), and Capacitated Facility Location (CFLP).

On standard benchmarks, A2DEPT consistently outperforms all tested LLM-based AHD baselines—including FunSearch, EoH, ReEvo, and MCTS-AHD—yielding significant reductions in mean normalized optimality gaps. Specifically, on CVRP under DeepSeek, it achieves an 8.79% gap versus 17.42% for the nearest competitor, and on MIS, a 4.20% gap compared to 11.35%. The average reduction with respect to the strongest baseline is 9.8% across tasks. In high-constraint environments, it demonstrates lower infeasibility rates and superior robustness—on CEVRPTW, A2DEPT achieves zero gap and only 2.56% infeasibility.

Detailed ablation studies confirm that A2DEPT's adaptive operator scheduler, diversity-aware supplementary selection, and program maintenance loop are all critical to its performance, with each component's removal causing measurable degradation.

Figure 3: Convergence trajectory on CVRP reveals structural breakthroughs and sustained improvement, contrasted with baseline plateauing.

Theoretical and Practical Implications

A2DEPT establishes the viability of using LLMs as algorithm architects over the entire solver logic, not merely heuristics slots. The evolutionary program tree enables modular edits, supporting both structural jumps and fine-grained optimization via feedback. Practical implications include accelerated algorithm discovery for NP-hard tasks, improved robustness to variant constraints, and potential for cross-domain transfer of solver architectures. Theoretically, the work demonstrates that LLM-driven search can escape template expressiveness bottlenecks and supports credit assignment via hierarchical operator scheduling. Risks of system-level migration—such as type consistency violations under data representation change—are observed, highlighting the need for semantic static analysis in future extensions.

Speculation on Future Developments

The AAD paradigm advocated by A2DEPT could catalyze advances in automated program synthesis for both discrete and continuous optimization, foster industrial-scale solver codebases with hierarchical module management, and enable LLMs to autonomously explore multi-objective or cross-domain algorithm spaces. Anticipated future directions include minimizing verification overhead via static checks, enhancing robustness against LLM reliability deficits, and extending evolutionary search to multi-modal or dynamic environments.

Conclusion

A2DEPT demonstrates that evolutionary program tree search, when coupled with executability maintenance and adaptive operator scheduling, allows LLMs to synthesize complete optimization algorithms surpassing heuristic-component-only baselines. The framework consistently yields robust, high-quality solvers across diverse combinatorial domains, setting a new operational paradigm for LLM-driven automated algorithm design (2604.24043).