Arnoldi-based Sampling for High-dimensional Optimization using Imperfect Data

Abstract: We present a sampling strategy suitable for optimization problems characterized by high-dimensional design spaces and noisy outputs. Such outputs can arise, for example, in time-averaged objectives that depend on chaotic states. The proposed sampling method is based on a generalization of Arnoldi's method used in Krylov iterative methods. We show that Arnoldi-based sampling can effectively estimate the dominant eigenvalues of the underlying Hessian, even in the presence of inaccurate gradients. This spectral information can be used to build a low-rank approximation of the Hessian in a quadratic model of the objective. We also investigate two variants of the linear term in the quadratic model: one based on step averaging and one based on directional derivatives. The resulting quadratic models are used in a trust-region optimization framework called the Stochastic Arnoldi's Method (SAM). Numerical experiments highlight the potential of SAM relative to conventional derivative-based and derivative-free methods when the design space is high-dimensional and noisy.