Diagonal Hessian Approximation Based on Conjugacy Condition for Noisy Derivative-Free Optimization Problems in High Dimensions

Abstract: We consider large-scale noisy derivative-free optimization (DFO) problems in which only function values are available and gradient or subgradient information cannot be reliably estimated. Matrix-adaptation evolution strategies (MAES) and their limited-memory variants are among the most robust DFO methods under noise; however, their performance may deteriorate when the noise level is large. In such regimes, sorting and selection may misidentify informative sampled points, making the recombination step less reliable and weakening the scaling information used by affine or matrix-adaptation mechanisms. This can substantially reduce the efficiency of MAES-type methods, especially in high-dimensional settings. To address this limitation, we propose a DFO method that replaces the full affine-scaling matrix with a diagonal approximation constructed from conjugacy-type conditions. The proposed mechanism does not attempt to estimate gradients, subgradients, or interpolation models, nor does it learn dense covariance information from noisy rankings. Instead, it uses consecutive normalized recombination displacements in a conservative diagonal update, thereby limiting the influence of unreliable selection information while preserving the derivative-free structure of the underlying evolutionary framework. As a result, the method is computationally cheaper than full matrix-adaptation schemes and limited-memory affine-scaling variants, while providing a stable scaling mechanism in noisy environments. Numerical experiments on noisy benchmark problems show that the proposed method is competitive with, and often more efficient than, MAES-type baselines, particularly when the noise level is large and ranking-based selection becomes unreliable.

• The paper proposes a diagonal Hessian approximation method that replaces dense affine-scaling with noise-robust conjugacy-based updates in derivative-free optimization.
• It develops explicit quadratic and cubic penalty models enabling O(n) computational complexity while safeguarding stability in high-dimensional settings.
• Empirical evaluations demonstrate over 20% additional problems solved and near-parity with full matrix-adaptation methods, confirming its effectiveness in noisy environments.

Diagonal Hessian Approximation via Conjugacy Condition for Noisy Derivative-Free Optimization in High Dimensions

Problem Motivation and Context

High-dimensional derivative-free optimization (DFO), especially in the presence of noise where reliable gradient or subgradient information is unavailable, presents significant challenges for both robustness and computational efficiency. Matrix adaptation evolution strategies (MAES) and their limited-memory variants remain among the most successful DFO methods under moderate noise, leveraging affine-scaling or covariance matrix learning to enhance search efficiency. However, as the noise level increases, ranking and selection phases may misidentify informative sample points, leading to unreliable recombination and degraded scaling information, particularly in high-dimensional settings. This undermines the core mechanisms of matrix adaptation, as dense covariance updates become both inaccurate and computationally costly.

The paper introduces a DFO algorithm that replaces dense affine-scaling with a derivative-free diagonal approximation based on conjugacy relations among generated search directions. Rather than attempting to estimate gradients, subgradients, or noisy interpolation models, the algorithm leverages normalized recombination displacements within a conservative diagonal update, thus limiting the impact of unreliable selection information. This change yields a computationally lightweight mechanism that preserves search efficacy in noisy environments, outperforming matrix-adaptation baselines in high-noise regimes and scaling efficiently to very high dimensions (2606.20304).

Diagonal Hessian Approximation via Conjugacy-Based Penalty Models

The classical approach to second-order information in optimization relies on dense Hessian or quasi-Newton approximations. However, recent work on diagonal Hessian updates has leveraged least-change models constrained by secant equations or finite-difference approximations, typically requiring explicit first-order information. These methods cannot be directly employed in DFO frameworks, where only zero-order information (function values and generated steps) is available.

Inspired by the fundamental role of conjugacy in the analysis of conjugate gradient algorithms and quasi-Newton methods, the paper develops diagonal curvature approximations from zero-order conjugacy conditions among consecutive generated directions. For directions $d_{k-1}$ and $d_k$, the diagonal Hessian approximation $$\mathcal{P}_{k+1} = \mathrm{diag}(p_{1_{k+1}},\dots,p_{n_{k+1}})$$ is constructed to enforce

$d_{k-1}^\top \mathcal{P}_{k+1} d_k = 0,$

with additional penalty terms included to promote least-change and well-conditioning. The general diagonal update minimizes

$$\frac{1}{2}\|\mathcal{P}_{k+1} - \mathcal{P}_k\|_F^2 + \frac{\rho}{2}\|\mathcal{P}_{k+1}-\mathcal{P}_k^{-1}\|_F^2 + \frac{\mu}{2p+z} |\mathcal{C}_{(k-1,k)}(\mathcal{P}_{k+1})|^{2p+z},$$

where $\rho$ and $\mu$ are penalty parameters, and $p$, $z$ higher-order penalty controls.

Explicit analytical updates are derived for quadratic ($p=1,z=0$) and cubic ($d_k$0) penalties, yielding efficient $d_k$1 computation via rank-one updates and scalar solves. For $d_k$2, convexity guarantees unique solutions, and a fixed number of bisection steps suffice for the scalar solve. Careful safeguarding ensures positive definiteness and bounded eigenvalues for all iterates, critical for stability in high-dimensional stochastic optimization.

Integration into MAES-Type Mutation Schemes

The diagonal scaling matrix $d_k$3 directly replaces the affine-scaling matrix in the mutation phase of MAES, generating mutation points as

$d_k$4

The scaling adapts the mutation distribution based on diagonal curvature surrogates. The update for $d_k$5 is based on normalized recombination displacements:

$d_k$6

with safeguarding on the diagonal entries to maintain stability. Step-size adaptation proceeds via cumulative evolution paths and exponential update formulas familiar from CMAES-type methods.

Unlike dense covariance adaptation, the diagonal update avoids expensive storage and computation ($d_k$7 cost per iteration) and is robust to high noise levels where ranking phases may be unreliable. Variation across quadratic, cubic, and higher-order penalty models is allowed, providing flexibility in controlling sensitivity to conjugacy violations depending on problem noise characteristics.

Empirical Evaluation

Extensive experiments are conducted on the prince collection of benchmark problems, extending to noisy settings (absolute/relative uniform and Gaussian noise) with dimensions up to $d_k$8, totaling over 20,000 test instances. Key findings include:

• On medium-scale problems ($d_k$9), the best diagonal variant (DMADFO12, $$\mathcal{P}_{k+1} = \mathrm{diag}(p_{1_{k+1}},\dots,p_{n_{k+1}})$$0) preserves robustness and efficiency of full-matrix MADFO with less than 3% reduction in solved problems.
• On large-scale ($$\mathcal{P}_{k+1} = \mathrm{diag}(p_{1_{k+1}},\dots,p_{n_{k+1}})$$1), DMADFO12 solves more than 20% additional problems compared to LMMAES, with 66.4% overall robustness versus 32.3% for the limited-memory baseline.
• Data and performance profiles demonstrate that diagonal scaling achieves near-parity with full-matrix adaptation in terms of function evaluations, and provides significant efficiency gains relative to limited-memory covariance adaptation.
• The superior performance of DMADFO12 is entirely due to the refined diagonal scaling mechanism, confirming that most relevant curvature information can be captured by considering only coordinate-wise adaptation, without loss of robustness or accuracy.

Implications and Future Directions

The development of diagonal conjugacy-based scaling mechanisms offers a robust and scalable alternative to matrix-adaptation in high-dimensional noisy DFO. The results challenge the necessity of dense covariance adaptation in noisy settings, indicating that refined diagonal strategies can capture critical curvature information while avoiding computational overhead. The framework is strictly derivative-free, immediately deployable in black-box optimization scenarios where only function evaluations are available.

Practically, the approach enables efficient optimization of large-scale models in machine learning, engineering, and scientific computing where noise, dimensionality, and computational constraints preclude use of classical quasi-Newton or dense adaptation strategies. Theoretically, the conjugacy penalty models suggest new directions for limited-memory approximation and sparsity-promoting curvature adaptation, with potential for integration into advanced evolutionary and randomized algorithms.

Future work could augment the robustness of the method by incorporating statistically reliable displacement directions (e.g., trimmed recombination or repeated evaluations) to mitigate the influence of ranking errors, and explore hybrid models combining diagonal and sparse block-adaptation. Extensions to constrained optimization and stochastic minimax settings are feasible given the strict derivative-free architecture.

Conclusion

Diagonal conjugacy-based Hessian approximations yield robust, efficient, and scalable mutation-scaling mechanisms for high-dimensional noisy derivative-free optimization. This method preserves most of the efficacy of full matrix-adaptation evolution strategies, significantly improves over limited-memory baselines in large-scale settings, and enables practical deployment in computation- and storage-limited optimization environments. The approach opens promising avenues in the design of noise-aware, sparsity-promoting curvature adaptation for future evolution strategies and derivative-free solvers (2606.20304).