HSEvo: Hessian Estimation & LLM Heuristics

• HSEvo is a dual-strand framework that integrates Hessian estimation for continuous optimization with LLM-driven heuristic design.
• It employs direct covariance adaptation and adaptive learning rules to achieve linear and superlinear convergence on convex and ill-conditioned problems.
• The LLM-based strand leverages explicit diversity metrics and harmony search to balance exploration and exploitation in programmatic heuristic search.

HSEvo refers to two distinct but influential strands in the literature: (1) the Hessian Estimation Evolution Strategy and its variants for black-box, derivative-free optimization, and (2) an adaptive LLM-based evolutionary framework for automatic heuristic design. Both lines are united by the principle of coupling evolution-inspired search with direct structural learning—of curvature in continuous landscapes or of diversity in programmatic heuristics—but are instantiated in different research domains.

1. Hessian Estimation Evolution Strategy: Concept and Motivation

Hessian Estimation Evolution Strategies (HE-ES), alternatively denoted HSEvo in some works, represent a class of black-box optimization algorithms that update the shape (covariance) of their search distributions by direct, local estimation of the curvature of the objective function. The methodology is positioned as a principled alternative to classic covariance adaptation schemes such as CMA-ES, targeting improved performance on ill-conditioned or highly anisotropic convex landscapes (Glasmachers et al., 2020, Glasmachers et al., 2020).

In its canonical form, HE-ES restricts itself to functions $f:\mathbb{R}^d\to\mathbb{R}$ that are at least twice continuously differentiable. The principal innovation is the estimation of the Hessian’s spectral structure via finite-difference curvature along orthogonal, randomly sampled directions, followed by a multiplicative update of the sampling covariance toward the inverse Hessian, the metric optimal for quadratic objectives.

2. Algorithmic Framework and Core Update Rules

HE-ES algorithms, including the (1+4)-HE-ES, proceed in iterations with the following elements (Glasmachers et al., 2020, Glasmachers et al., 2020):

• Sampling: In each generation $t$, draw a specified number of orthogonal Gaussian directions $\{b_i\}$ (via Gram–Schmidt orthogonalization with length-rescaling); for $\lambda=4$, two are used. Form mirrored offspring: $x^+_i = m^{(t)}+\sigma^{(t)}A^{(t)}b_i$, $x^-_i = m^{(t)}-\sigma^{(t)}A^{(t)}b_i$.
• Covariance (Shape) Update: Evaluate the parent and offspring, compute curvature estimates $$h_i = \frac{f(x^+_i)+f(x^-_i)-2f(m)}{\sigma^2\|b_i\|^2}$$, and construct the update matrix

$$G = I + (\gamma_1-1)u_1u_1^T + (\gamma_2-1)u_2u_2^T, \qquad \gamma_1=(h_2/h_1)^{1/4},\ \gamma_2=(h_1/h_2)^{1/4}$$

where $u_i$ are normalized $b_i$. The new shape is $t$0 and $t$1.

• Mean and Step-Size Adaptation: If the first offspring outperforms the parent, set $t$2, $t$3; otherwise, $t$4 is unchanged and $t$5.

This approach generalizes naturally to larger populations and higher numbers of directions, with updates symmetrically structured to maintain $t$6 (and thus $t$7 invariant), and using learning rates and trust-region style truncation of small/unstable curvatures (Glasmachers et al., 2020, Glasmachers, 16 May 2025).

3. Theoretical Guarantees: Covariance Stability and Convergence Rates

The most salient theoretical property of HE-ES/HSEvo is its provable shape convergence and rate guarantees on convex quadratics (Glasmachers et al., 2020):

• Covariance Stability: For $t$8, the iterative sequence of covariances $t$9 obeys

$\{b_i\}$0

with probability $\{b_i\}$1, for any initialization with $\{b_i\}$2. Both stability (bounded distance to $\{b_i\}$3 under any pre-metric) and actual convergence (distance tending to zero) hold.

• Linear Convergence (Quadratic Case): The proposed process achieves $\{b_i\}$4-linear convergence in terms of “volume-normalized” error, where the expected hitting time for $\{b_i\}$5 error satisfies

$\{b_i\}$6

with constants $\{b_i\}$7 independent of the condition number $\{b_i\}$8.

For general $\{b_i\}$9 objectives, the method exhibits robust (linear) global convergence as long as local curvature is faithfully estimated; ill-conditioning is neutralized rapidly once shape learning stabilizes (Glasmachers et al., 2020, Glasmachers et al., 2020).

4. Extensions: Hybrid Quasi-Newton/HE-ES and Superlinear Regimes

A recent extension of HSEvo incorporates quasi-Newton progression alongside evolution-inspired search, yielding a hybrid algorithm (“QN-ES” or HSEvo) with superlinear local convergence and improved final-phase performance on smooth convex problems (Glasmachers, 16 May 2025). The core features are:

• Gradient Estimation: Central difference over mirrored, block-orthogonal samples yields $\lambda=4$0 in the local metric.
• Two-Step Candidate Mean Update: Both traditional recombination and a quasi-Newton step $\lambda=4$1 are constructed; a dynamically weighted trust-region switch selects the candidate with better objective.
• Hessian Shape Tracking: The covariance update, enforced to maintain $\lambda=4$2, iteratively tracks the local inverse Hessian up to scale, relying on matrix-exponential updates tied to log-curvature deviations.
• Superlinear Convergence: Under standard smoothness and positive-definiteness assumptions,

$\lambda=4$3

as $\lambda=4$4, matching the asymptotics of classical BFGS (Glasmachers, 16 May 2025).

This hybrid framework exhibits two distinct phases: an initial “learning” period with robust, anisotropy-neutralizing progress, followed by rapid superlinear descent once the local metric is sufficiently adapted. Empirical studies confirm the approach outperforms standard ES and closes the gap to model-based trust-region methods such as NEWUOA on challenging, high-conditioning quadratics.

5. Alternative Line: HSEvo for LLM-Based Automatic Heuristic Design

A separate strand, “HSEvo: Elevating Automatic Heuristic Design with Diversity-Driven Harmony Search and Genetic Algorithm Using LLMs” (Dat et al., 2024), introduces HSEvo as an adaptive framework for LLM-based Evolutionary Program Search (LLM-EPS) in combinatorial optimization. This construction addresses the challenge of balancing diversity (exploration) and convergence (exploitation) in the space of executable heuristics.

• Diversity Quantification: Employs two entropy-inspired metrics—Shannon–Wiener Diversity Index (SWDI) based on clustering of code embeddings, and Cumulative Diversity Index (CDI) derived from minimum spanning tree metrics—for explicit monitoring of population spread in program space.
• Harmony Search Tuning: Periodically injects a Harmony Search (HS) tuner dedicated to exploitation of the current elite heuristic, optimizing extracted real-valued parameter sets.
• GA and LLM Integration: LLMs generate child heuristics via genetic crossover (with reflection) and mutation, maintaining both evolutionary exploration and local search focus. Adaptive transition between GA-driven exploration and HS-driven exploitation is dictated by the diversity indices and objective improvement signals.

Empirical results demonstrate HSEvo achieves or exceeds competitors’ diversity levels and consistently yields top objective scores (e.g., on BPO, TSP, and OP benchmarks), substantiating the value of explicit diversity management in LLM-based heuristic search. Ablative studies confirm the synergy of diversity metrics, HS tuning, and reflection/role-play initialization for effective, adaptive control (Dat et al., 2024).

6. Empirical Evaluation and Practical Implications

The HE-ES/HSEvo family has been benchmarked on the BBOB/COCO testbed (real-valued, noiseless functions), with key findings (Glasmachers et al., 2020, Glasmachers, 16 May 2025):

• Performance on Quadratic and Ill-Conditioned Problems: On smooth, convex, and ill-conditioned objectives, curvature adaptation yields rapid metric learning and convergence competitive with or superior to CMA-ES and NEWUOA.
• Global Robustness: The techniques are robust to moderate level violations of the underlying $\lambda=4$5 assumptions—such as under monotonic but non-affine value transformations or moderate ruggedness—due to the invariance properties of the direct curvature update.
• Limitations: On highly multimodal or pathological landscapes, the reliance on local curvature can misdirect the search, warranting restarts or diversity preservation mechanisms.
• Parameterization: Default values ($\lambda=4$6, $\lambda=4$7, $\lambda=4$8) suffice in most settings, but can be tuned upward (for noise) or downward (for non-convexity) (Glasmachers et al., 2020).

For programmatic LLM-EPS, HSEvo (Dat et al., 2024) provides an explicit template for scalable, diversity-controlled heuristic search, with quantifiable diversity-benefit trade-offs and straightforward integration of LLM-specific generation and mutation operators.

Family/Context	Domain	Defining Features
HE-ES/HSEvo	Continuous black-box optimization	Direct Hessian spectrum estimation, shape convergence, metric learning, superlinear hybridization
HSEvo (LLM-EPS)	Programmatic AHD with LLMs	Shannon-inspired diversity quantification, harmony search exploitation, LLM-guided GA

7. Research Directions and Open Questions

Current research is advancing both theoretical and practical aspects:

• Hybridization and Information Fusion: Combining Hessian estimation with natural-gradient methods or other forms of curvature information (e.g., block-diagonal or low-rank fits) remains an active area (Glasmachers et al., 2020).
• Noise Analysis and Global Optima: A complete complexity analysis of HE-ES under non-negligible noise remains open, especially for multi-modal and discontinuous domains.
• Diversity Mechanisms in Program Search: For LLM-based heuristic design, the efficacy of various diversity quantification schemes, combinations with alternative local search (e.g., MCTS, reinforcement learning), and scaling to larger parameter spaces merit further study (Dat et al., 2024).
• Robustness beyond Convexity: Stabilization of curvature learning in the presence of highly non-convex or adversarial objectives, as well as integration with restart or restart-scheduling protocols, are practical but open questions.

Overall, HSEvo represents a convergence of evolution-inspired optimization with explicit structural learning—curvature in continuous spaces, or diversity in programmatic heuristics—enabling principled, theoretically grounded, and empirically robust search methodologies across both continuous and combinatorial domains (Glasmachers et al., 2020, Glasmachers et al., 2020, Glasmachers, 16 May 2025, Dat et al., 2024).