Subspace-Based Approximate Hessian (ZO-SAH)

• Subspace-Based Approximate Hessian (ZO-SAH) is a zeroth-order method that estimates curvature via local quadratic fitting in randomly chosen two-dimensional subspaces.
• It leverages periodic subspace switching and function-value reuse to reduce the query cost from O(d²) to an amortized O(m/T) per iteration.
• The method achieves linear convergence under strong convexity by enforcing positive-definite Hessian surrogates and aggregating local Newton directions.

Searching arXiv for the core ZO-SAH paper and closely related subspace/Hessian-aware zeroth-order methods. arXiv search query: "Subspace-based Approximate Hessian Method for Zeroth-Order Optimization" Subspace-Based Approximate Hessian (ZO-SAH) denotes a class of zeroth-order optimization methods that incorporate curvature information without explicit gradients or full Hessians by constructing Hessian surrogates on low-dimensional subspaces from function-value queries. In its most direct formulation, ZO-SAH is a Newton-type zeroth-order method that works in randomly chosen two-dimensional subspaces, fits local quadratic models there, and combines the resulting restricted Hessians with periodic subspace switching and function-evaluation reuse to reduce query cost (Kim et al., 8 Jul 2025). Closely related arXiv work places the same idea within a broader family of subspace-restricted Hessian-aware zeroth-order methods, including low-rank Hessian-aware attacks, randomized subspace Newton schemes, and subspace quasi-Newton updates built from finite differences (Ye et al., 2018).

1. Problem setting and conceptual scope

The primary setting is black-box minimization of a smooth objective

$\min_{x \in \mathbb{R}^d} f(x),$

where only function values are available. The 2025 ZO-SAH formulation assumes that $f$ is twice continuously differentiable and develops its main guarantees under $\mu$-strong convexity, Lipschitz continuity of $\nabla f$ with constant $C_1$, Lipschitz continuity of the Hessian $Hf$ with constant $C_2$, and Armijo backtracking line search with a minimum step size (Kim et al., 8 Jul 2025).

The defining motivation is the query cost of second-order zeroth-order methods. Naive full Hessian estimation by coordinate-wise finite differences scales as $O(d^2)$ queries per iteration, which is usually prohibitive in high dimension. ZO-SAH addresses this by estimating curvature only inside low-dimensional subspaces, then using those restricted Hessians to compute Newton-like directions. In the dedicated ZO-SAH method, the restricted subspaces are two-dimensional; in related literature, the same subspace principle appears with $k$-dimensional sketches, Krylov subspaces, or deterministic subspaces built from past iterates (Kim et al., 8 Jul 2025).

A common misconception is that ZO-SAH estimates the ambient Hessian itself. The dedicated method does not do so. It selects an intermediate $m$-dimensional subspace $f$0, partitions it into $f$1 disjoint two-dimensional subspaces $f$2, estimates a $f$3 Hessian inside each $f$4, and aggregates the resulting local Newton directions back into the ambient space. This is a subspace-curvature method rather than a full-Hessian reconstruction (Kim et al., 8 Jul 2025).

2. Local quadratic fitting in random two-dimensional subspaces

At iteration $f$5, ZO-SAH selects a two-dimensional subspace with orthonormal basis $f$6. In implementation, the method first constructs an intermediate subspace $f$7 of dimension $f$8 via random coordinate selection and then partitions it into $f$9 disjoint two-dimensional subspaces $\mu$0. For each $\mu$1, an orthogonal projection matrix $\mu$2 represents the basis; equivalently, $\mu$3 can be viewed as $\mu$4 for that subspace (Kim et al., 8 Jul 2025).

Inside a given two-dimensional subspace, the method uses a local quadratic model

$\mu$5

with $\mu$6. A standard overdetermined least-squares construction recovers the coefficients from sampled function values, with the design rows

$\mu$7

The dedicated implementation uses a reduced variant: it sets the constant term to the function value at the center and sets the linear term by a two-point finite-difference gradient estimate, then solves only for the three second-order coefficients of the symmetric $\mu$8 Hessian (Kim et al., 8 Jul 2025).

Writing the local coordinates as $\mu$9, the model is

$\nabla f$0

with $\nabla f$1 and $\nabla f$2. The Hessian coefficients $\nabla f$3 are obtained from

$\nabla f$4

where

$\nabla f$5

and

$\nabla f$6

This yields a symmetric $\nabla f$7 subspace Hessian (Kim et al., 8 Jul 2025).

The subspace gradient itself is estimated by coordinate-wise two-point finite differences:

$\nabla f$8

with $\nabla f$9 in the reported experiments. Given the estimated gradient and Hessian, the local Newton step is

$C_1$0

Because the estimated $C_1$1 may be indefinite, the implementation enforces positive definiteness by replacing each eigenvalue $C_1$2 by $C_1$3, with $C_1$4 in experiments. Across all $C_1$5 subspaces, the ambient direction is aggregated as

$C_1$6

followed by Armijo line search (Kim et al., 8 Jul 2025).

3. Periodic subspace switching and query reuse

The distinctive practical device in ZO-SAH is periodic subspace switching with reuse of previously queried function values. At steps satisfying $C_1$7, the method chooses a new intermediate $C_1$8-dimensional subspace $C_1$9 by random coordinate selection and partitions it into $Hf$0 disjoint two-dimensional subspaces $Hf$1. Those subspaces are then kept fixed for a period of length $Hf$2, which allows cached values from earlier steps to be reused in both gradient and Hessian fitting (Kim et al., 8 Jul 2025).

The reuse mechanism is explicit. At the beginning of a period, the cache is empty, so for each two-dimensional subspace the method queries two gradient points and three additional random nearby points for least-squares fitting. At the next step it queries only the two gradient points needed to populate the reuse set. For subsequent steps inside the same period, it reuses the function values from the two preceding steps:

$Hf$3

where

$Hf$4

Thus the Hessian least-squares system can be refit without issuing new function queries on intermediate steps (Kim et al., 8 Jul 2025).

The amortized query accounting is one of the central claims of the method. Over a period of $Hf$5 steps, each two-dimensional subspace uses 5 function queries at the first step, 2 at the second, and 0 for steps $Hf$6. This gives an amortized cost per iteration per subspace of approximately $Hf$7, and across all $Hf$8 subspaces the paper states an amortized cost of approximately $Hf$9 queries per iteration, plus approximately $C_2$0 for the shared center value $C_2$1. The resulting per-iteration query cost is therefore $C_2$2, in contrast to $C_2$3 for full Hessian finite differences (Kim et al., 8 Jul 2025).

The arithmetic overhead is correspondingly small. For each subspace, the least-squares fit solves a $C_2$4 linear system for the second-order coefficients, and the Newton step requires only inversion or eigendecomposition of a $C_2$5 matrix. The overall computational cost per iteration is $C_2$6, with memory that scales linearly in $C_2$7 because the method stores the recent perturbation values needed for reuse (Kim et al., 8 Jul 2025).

4. Convergence theory and estimation error

Under the assumptions of $C_2$8-strong convexity, lower boundedness, Lipschitz gradient, Lipschitz Hessian, Armijo backtracking, and bounded perturbations, the paper proves a linear convergence rate in expectation over the random subspace selection:

$C_2$9

where $O(d^2)$0 and

$O(d^2)$1

with

$O(d^2)$2

Here, $O(d^2)$3 bounds the sampling radius in the subspace, $O(d^2)$4 is the number of least-squares samples, $O(d^2)$5 is a lower bound on the smallest eigenvalue of $O(d^2)$6, and $O(d^2)$7 is the eigenvalue-clipping floor (Kim et al., 8 Jul 2025).

The proof isolates the errors introduced by the zeroth-order estimators. The subspace gradient and Hessian obey the bounds

$O(d^2)$8

These inequalities formalize the trade-off between local model fidelity and numerical stability. Smaller finite-difference step sizes and smaller subspace sampling radii reduce truncation error, while larger $O(d^2)$9 improves conditioning of the least-squares design matrix (Kim et al., 8 Jul 2025).

Positive-definite enforcement is not only an implementation heuristic but also part of the descent mechanism. By replacing eigenvalues with their absolute values and clipping them from below, the method guarantees that $k$0 is a descent direction, making Armijo backtracking applicable. The proof then invokes the Polyak–Łojasiewicz inequality to derive the stated linear rate (Kim et al., 8 Jul 2025).

The dedicated ZO-SAH theory is specific to the strongly convex regime. Related subspace-based zero-order work broadens the theoretical picture. In particular, the subspace quasi-Newton method with gradient approximation establishes expected $k$1 stationarity for nonconvex problems, expected $k$2 suboptimality for convex problems, and linear convergence in expectation under a Polyak–Łojasiewicz condition, while operating with only $k$3 matrices and finite-difference subspace gradients (Miyaishi et al., 2024). This suggests that the broader subspace-Hessian idea is not limited to the strongly convex setting, although the specific 2D ZO-SAH proof is framed there.

5. Position within subspace and Hessian-aware zeroth-order optimization

The dedicated 2D ZO-SAH method belongs to a longer line of work in which curvature is estimated or exploited only on a restricted subspace. The relationships are easiest to see at the level of the object being approximated: not the full $k$4, but either a low-rank surrogate, a sketched Hessian $k$5, or a projected Hessian on a chosen span.

Paper	Core mechanism	Relation to ZO-SAH
"Hessian-Aware Zeroth-Order Optimization for Black-Box Adversarial Attack" (Ye et al., 2018)	Low-rank Hessian-aware preconditioning	Subspace-based $k$6 via power-method sketch
"Zeroth-Order Randomized Subspace Newton Methods" (Berglund et al., 2022)	Random sketching of $k$7 and $k$8	Newton step in randomized subspace
"Subspace Quasi-Newton Method with Gradient Approximation" (Miyaishi et al., 2024)	$k$9 quasi-Newton update with finite-difference subspace gradients	Deterministic-subspace ZO-SAH in spirit
"A matrix algebra approach to approximate Hessians" (Hare et al., 2023)	Function-values-only projected Hessians via pseudoinverses	Subspace Hessian from sampled directions

In ZO-HessAware, the update

$m$0

uses an approximate Hessian to shape both the sampling distribution and the update direction. Its subspace-based variant, ZOHA-PW, constructs a low-rank Hessian approximation

$m$1

where $m$2 spans a top-$m$3 eigenspace extracted by a noisy power method using only function queries. The paper identifies improved iteration complexity

$m$4

and query complexity

$m$5

when $m$6 (Ye et al., 2018).

ZO-RSN takes a different route. It sketches a randomized subspace matrix $m$7, estimates the projected gradient and Hessian via forward finite differences,

$m$8

solves the $m$9 Newton system in that subspace, and updates in $f$00. For strongly convex problems it proves global linear convergence and states a per-iteration query requirement of $f$01 function evaluations (Berglund et al., 2022).

The 2024 subspace quasi-Newton method is not named ZO-SAH in the paper, but the data explicitly characterizes it as ZO-SAH in spirit. It maintains a deterministic subspace basis $f$02, solves a $f$03 quasi-Newton system, updates a subspace inverse Hessian by BFGS, and replaces full gradients with directional finite-difference surrogates along both a random sketch $f$04 and the deterministic subspace $f$05. Its finite-difference implementation has overall per-iteration complexity $f$06 (Miyaishi et al., 2024).

The generalized simplex Hessian and generalized centered simplex Hessian provide a still more algebraic view of the same principle. Using only function evaluations and Moore–Penrose pseudoinverses, they construct either full Hessians or subspace/partial Hessians depending on the geometry of the sampled directions. The central statements are order-$f$07 accuracy for GSH and order-$f$08 accuracy for GCSH relative to the projected true Hessian, with explicit error bounds in underdetermined regimes (Hare et al., 2023). The follow-up paper makes the subspace interpretation explicit: for a basis $f$09, one can estimate

$f$10

by

$f$11

again using function values only (Jarry-Bolduc et al., 2023).

6. Empirical behavior, applicability, and limitations

The dedicated ZO-SAH paper evaluates the method on eight benchmark datasets covering logistic regression and deep neural network training tasks. The most compact synthetic result is on Rosenbrock, where ZO-SAH reaches the optimum with 201 calls, compared with 7,921 for RSPG. On logistic regression tasks over the datasets a3a, a5a, w1a, GISETTE, Olivetti, and phishing, the method is reported to achieve robust convergence and often the best or second-best final losses with far fewer queries. The paper also notes that gains depend on anisotropy: on GISETTE, whose average condition number is approximately 3.93, gains are smaller, whereas on Olivetti, whose average condition number is approximately 15.23, the advantage is larger (Kim et al., 8 Jul 2025).

On deep neural networks, the reported tasks are ResNet8 and ResNet20 training on CIFAR10 and CIFAR100. ZO-SAH is stated to converge faster than DeepZero and to reach the same loss with far fewer queries, approximately 50% fewer on ResNet8/CIFAR10 and approximately 29% fewer on ResNet20/CIFAR100. The reported test accuracies are 76.3% versus 73.4% on ResNet8/CIFAR10 and 48.8% versus 42.4% on ResNet20/CIFAR100 (Kim et al., 8 Jul 2025).

The ablation results clarify what part of the design matters. Reuse of past function values substantially reduces total query counts relative to fresh finite differences at every step. A diagonal-only Hessian approximation performs much worse than the 2D-subspace approximation, indicating that off-diagonal curvature inside each two-dimensional block is important. The period parameter $f$12 mediates a direct trade-off between reuse and adaptation, and the paper reports that a moderate value such as $f$13 balances both in deep networks (Kim et al., 8 Jul 2025).

The limitations are equally specific. Each update models curvature only in two dimensions at a time, so the method does not capture full high-dimensional interactions in a single step. It is sensitive to non-quadratic local behavior and to the sampling radius $f$14: if $f$15 is too large, the quadratic approximation deteriorates; if $f$16 is too small, the least-squares system can become ill-conditioned through a small $f$17. The paper accordingly proposes several extensions, including $f$18-dimensional subspaces with $f$19, adaptive subspace selection, trust-region mechanisms that adapt $f$20, accumulation of subspace Hessians into a global low-rank preconditioner, and parallel querying across multiple subspaces (Kim et al., 8 Jul 2025).

Taken together, these results position ZO-SAH as a second-order zeroth-order method whose defining contribution is not merely subspace restriction, but the combination of restricted Hessian fitting, positive-definite enforcement, periodic subspace switching, and aggressive function-evaluation reuse. In the broader arXiv landscape, that combination links it to low-rank Hessian-aware attacks, randomized subspace Newton methods, and subspace quasi-Newton schemes, while retaining a distinctive emphasis on constant-size two-dimensional curvature models and amortized query efficiency (Kim et al., 8 Jul 2025).