LLM-Driven Algorithm Discovery

• LLM-driven algorithm discovery is a process where LLMs generate, evaluate, and refine algorithm variants using a closed-loop pipeline that integrates benchmarking and explainability.
• The approach employs prompt engineering, stochastic program synthesis, and sensitivity analysis to identify key components and optimize performance on benchmark problems.
• It accelerates algorithm innovation by producing interpretable, class-specific heuristics that adapt to problem landscapes through continuous feedback and refinement.

LLM-driven algorithm discovery refers to the automated, model-centric process in which LLMs are used to propose, refine, and optimize algorithms—primarily (but not exclusively) for optimization and scientific computing—by generating executable code or modular algorithmic variants in response to prompts, followed by systematic benchmarking and explainability-guided iteration. This paradigm represents a convergence of machine learning, evolutionary computation, symbolic program synthesis, and explainable AI, with the aim of both accelerating algorithmic innovation and increasing the interpretability of discovered solutions (Stein et al., 20 Nov 2025).

1. Formalization of the LLM-Driven Algorithm Discovery Pipeline

LLM-driven algorithm discovery is structured as a closed loop of generation, evaluation, attribution, and refined generation. The process can be modeled as a multi-stage stochastic program synthesis pipeline:

Stage 1: LLM-Driven Variant Generation

• The system defines a prompt template $\mathcal{P}$ (e.g., "Generate a new variant of CMA-ES that...").
• The LLM, parameterized by $\theta$, defines a conditional distribution over candidate algorithms $\mathcal{A}$ (code, pseudocode, or data structures):

$P_{\rm LLM}(\mathcal{A}\mid\mathcal{P};\theta)$

• Sampling is controlled by temperature $T$ and top-$k$ or nucleus $p$ strategies to balance creativity and adherence.

Stage 2: Program Evaluation

• Each variant $\mathcal{A}_i$ is instantiated and executed on a benchmark suite $\mathcal{F} = \{f_1, f_2, \ldots\}$.
• The benchmarking platform $\mathcal{B}$ returns summary metrics:

$$\mathbf{y}_i = \mathcal{B}(\mathcal{A}_i, \mathcal{F})$$

Stage 3: Explainable Attribution and Feedback

• An explainable-AI module computes component-level attributions (e.g., Shapley values) or hyperparameter importances for each variant:

$$\phi_j = \sum_{S\subseteq[M]\setminus\{j\}} \frac{|S|!\,(M-|S|-1)!}{M!}\left(\mathbb{E}[y|S\cup\{j\}] - \mathbb{E}[y|S]\right)$$

• Attribution results are used to adjust the prompt for the next generation or reweight the LLM’s prior.

This loop repeats until a convergence or budget stopping criterion is satisfied, yielding a dynamic, feedback-driven search process anchored in both generative modeling and empirical testing (Stein et al., 20 Nov 2025).

2. Foundations: Three Pillars—Discovery, Explanation, Problem-Class Quantification

2.1 LLM-Driven Generation

Algorithmic variant generation relies on prompt family engineering, with template classes indexed by problem structure (e.g., "ES-style," "DE-style"). Iterative sampling incorporates temperature and nucleus sampling adjustments to traverse the relevant algorithmic manifold.

Generation Probability:

$$P(\mathcal{A}_i\mid\mathcal{P}_t; \theta,T) = \prod_{k=1}^{L_i} P_\theta(w_k\mid w_{<k}, \mathcal{P}_t;T)$$

2.2 Explainable Benchmarking

Component importance is attributed using supervised surrogates (e.g., random forests) or direct sensitivity analysis, mapping binary (or real-valued) vectors encoding modules and hyperparameters to performance metrics. Attribution via Shapley values or gradients establishes which structural properties of $\mathcal{A}_i$ most affect outcomes.

Formal model:

$$y_i = g(\mathbf{h}(\mathcal{A}_i)) + \varepsilon, \quad \varepsilon\sim\mathcal{N}(0,\sigma^2)$$

2.3 Problem-Class Descriptors (Exploratory Landscape Analysis)

Quantification of problem structure is performed through computable descriptors:

• Modality: estimated from random-walk local optima counts.
• Ruggedness: via autocorrelation $\rho(k)$.
• Information content: $$\mathit{IC}(\epsilon) = -\sum_{i,j} p_{ij}(\epsilon)\log_2 p_{ij}(\epsilon)$$.
• Barrier tree height, basin count: via local clustering.

These features enable clustering of problems into classes $\mathcal{C}_1, \ldots, \mathcal{C}_K$ that empirically correlate with component attributions, supporting class-specific algorithm design rules (Stein et al., 20 Nov 2025).

3. Closed Knowledge Loop: Discovery ↔ Explanation ↔ Description

The integration of LLM-driven generation, explainable benchmarking, and landscape-based problem classification forms a closed "knowledge loop" characterized by:

1. Discovery: Generate algorithm variants $\{\mathcal{A}_i\}$ from current prompt $\mathcal{P}_t$.
2. Benchmark & Explain: Evaluate $\mathbf{y}_i$, compute attributions $\{\phi_j\}$ per variant.
3. Describe: Extract ELA features; cluster $\mathcal{F}$ into problem-classes $\mathcal{C}_k$.
4. Generalize & Refine: Update prompt to favor components with high $\phi_j$ for each class:

$$\mathcal{P}_{t+1} = \text{“Use component }c_j\text{ for }\mathcal{C}_k\text{ with weight }\phi_j\geq\tau.”$$

Over successive cycles, the LLM accrues “design knowledge” via in-context examples, explicit prompt templates, or retrieval-augmented priors, yielding progressively more specialized and interpretable algorithms for each problem class (Stein et al., 20 Nov 2025).

4. Algorithmic Example: Explainable, Class-Specific Heuristic Discovery

Consider a class $\mathcal{C}_1$ of highly multimodal continuous functions (high modality descriptor).

• Attribution analysis reveals that recombination operator $r_1$ and hyperparameter $\alpha$ are highly impactful ($\phi_{r_1}, \phi_\alpha$ large).
• The next prompt specifies: "Generate a CMA-ES variant with rank-$\mu$ recombination emphasizing $\alpha=0.4$ success-rule adaptation."
• The LLM-generated variant achieves a $15\%$ improvement in average gap to optimum on $\mathcal{C}_1$.

This example illustrates a data-driven, interpretable mechanism by which LLM-driven discovery moves beyond black-box search, constructing algorithmic variants that are rationalized, attributed, and matched to problem-class features (Stein et al., 20 Nov 2025).

5. Interplay with Benchmarking and Generalization

LLM-driven algorithm discovery depends critically on systematic, class-specific benchmarking to:

• Disentangle true innovation from memorization.
• Attribute algorithmic performance to explicit design elements.
• Identify generalizable patterns that transcend individual benchmarks.

Benchmarking platforms (e.g., COCO, IOHprofiler) provide not only performance data but also the basis for ELA-based problem-class clustering, supporting both reproducibility and interpretable advancement of the field.

Empirical finding: The closed-loop process accelerates algorithmic progress while simultaneously producing reusable scientific insight into when and why particular strategies succeed, as opposed to blind code search or purely performance-driven exploration (Stein et al., 20 Nov 2025).

6. Implications and Future Directions

LLM-driven algorithm discovery as formalized in this explainable, knowledge-loop framework marks a shift from opaque, purely automatic design to an interpretable, data-driven science of metaheuristics:

• Transition to Interpretable Scientific Discovery: Embedding explainability at every stage produces actionable knowledge regarding what works, for which problems, and why.
• Reusable Insights: The process yields not just isolated algorithms but reusable rules of thumb linking landscape descriptors to algorithm components.
• Acceleration of Progress: Systematic coupling of LLM creativity, benchmarking, and problem quantification outpaces legacy approaches that lack feedback-driven refinement.

The field is converging toward integrative, explanation-centric workflows in which LLMs serve both as creative engines and as learners of transferable, class- and component-level design insights (Stein et al., 20 Nov 2025).