Hessian Chain Bracketing

• Hessian chain bracketing is a derivative-free optimization technique that approximates the Hessian matrix using structured function evaluations and nested-set constructions.
• It employs the generalized simplex gradient and gradient-difference matrix to achieve provable first-order accuracy and exactness on quadratic functions.
• Its flexible design supports both minimal poised sets and overdetermined schemes, enhancing numerical stability and practical algorithm performance.

Hessian chain bracketing, as instantiated in the nested-set Hessian approximation, is a second-order approximation method developed for derivative-free optimization (DFO). It generalizes the concept of the simplex gradient to yield a Hessian estimate using only function evaluations at structured directions from a base point, enabling rigorous second-order information to be incorporated into DFO routines without requiring derivatives. The approach stands out for provable first-order accuracy, exactness on quadratics, and flexibility regarding the geometry of evaluation sets. It also serves as a foundation for higher-level composite rules ("simplex calculus" and "quadratic calculus" Hessians), broadening its applicability in practical algorithm design (Hare et al., 2020).

1. Foundation and Construction

Let $f:\mathbb{R}^n \rightarrow \mathbb{R}$ be thrice continuously differentiable near a base point $x^0$. Two sets of direction vectors are considered: $S = [s^1 \dots s^m] \in \mathbb{R}^{n \times m}$ and $T = [t^1 \dots t^k] \in \mathbb{R}^{n \times k}$, such that all relevant offset points are in the domain of $f$. The generalized simplex gradient approximates the gradient by

$$\nabla_S f(x^0; T) := (T^\top)^\dagger \, [f(x^0 + t^j) - f(x^0)]_{j = 1 \ldots k}$$

where $(\cdot)^\dagger$ is the Moore–Penrose pseudoinverse.

The nested-set Hessian applies this principle twice: first computing simplex gradients in each $T$-slice, then differencing these in the $S$-directions to estimate directional second derivatives. Specifically, construct a matrix whose $i$th row is the difference

$x^0$0

and combine this via solution of a linear system in $x^0$1.

2. Mathematical Formulation

Define the gradient-difference matrix

$x^0$2

with $x^0$3th row given by the difference between the generalized simplex gradients at $x^0$4 and $x^0$5.

The nested-set Hessian is then

$x^0$6

This is equivalent to solving the best-fit linear system

$x^0$7

The construction makes no symmetry assumption, although in practice symmetrization may be applied post hoc as needed.

3. Evaluation Geometry and Error Bounds

Let

$x^0$8

Assume $x^0$9 and $S = [s^1 \dots s^m] \in \mathbb{R}^{n \times m}$0 have full row rank and $S = [s^1 \dots s^m] \in \mathbb{R}^{n \times m}$1 is Lipschitz with constant $S = [s^1 \dots s^m] \in \mathbb{R}^{n \times m}$2 in a neighborhood of $S = [s^1 \dots s^m] \in \mathbb{R}^{n \times m}$3. Then,

$S = [s^1 \dots s^m] \in \mathbb{R}^{n \times m}$4

where $S = [s^1 \dots s^m] \in \mathbb{R}^{n \times m}$5 and $S = [s^1 \dots s^m] \in \mathbb{R}^{n \times m}$6.

When $S = [s^1 \dots s^m] \in \mathbb{R}^{n \times m}$7, this simplifies to

$S = [s^1 \dots s^m] \in \mathbb{R}^{n \times m}$8

exhibiting $S = [s^1 \dots s^m] \in \mathbb{R}^{n \times m}$9 accuracy [(Hare et al., 2020), Proposition 4.1]. This bound incorporates both the geometry (radii and condition number) of the evaluation sets and the smoothness of $T = [t^1 \dots t^k] \in \mathbb{R}^{n \times k}$0.

4. Minimal and Overdetermined Evaluation Schemes

When $T = [t^1 \dots t^k] \in \mathbb{R}^{n \times k}$1 and $T = [t^1 \dots t^k] \in \mathbb{R}^{n \times k}$2 is invertible, a minimal "poised" set can be constructed. For each $T = [t^1 \dots t^k] \in \mathbb{R}^{n \times k}$3, define $T = [t^1 \dots t^k] \in \mathbb{R}^{n \times k}$4 whose columns are $T = [t^1 \dots t^k] \in \mathbb{R}^{n \times k}$5. Collecting points $T = [t^1 \dots t^k] \in \mathbb{R}^{n \times k}$6 yields exactly $T = [t^1 \dots t^k] \in \mathbb{R}^{n \times k}$7 distinct locations. This matches the count required for quadratic interpolation and thus matches the information-theoretic minimum for obtaining a full Hessian approximation [(Hare et al., 2020), Proposition 3.1; Corollary 3.2].

Nothing precludes the use of $T = [t^1 \dots t^k] \in \mathbb{R}^{n \times k}$8 and/or $T = [t^1 \dots t^k] \in \mathbb{R}^{n \times k}$9, and overdetermined schemes can be constructed. Overdetermined choices can reduce the operator norms $f$0, potentially improving numerical stability and tightening constants in the error bound, in exchange for additional function evaluations. The error analysis applies equally to such settings.

5. Calculus-Based Composite Hessian Approximations

The simplex and quadratic calculus Hessians are derived for composite functions, enabling reuse of existing simplex or quadratic models for derivative and Hessian calculations when functions are built from sums, products, quotients, or powers.

• Simplex calculus Hessian: Apply the usual componentwise product/quotient/power Hessian rules, replacing $f$1 with $f$2 and $f$3 with $f$4. For $f$5,

$f$6

The error in this approximation is $f$7 under the same regularity assumptions.

• Quadratic calculus Hessian: After forming the quadratic interpolant $f$8 over the poised set (without additional function evaluations), replace $f$9 by the gradient of $$\nabla_S f(x^0; T) := (T^\top)^\dagger \, [f(x^0 + t^j) - f(x^0)]_{j = 1 \ldots k}$$0 (which is exact on quadratics), and similarly for $$\nabla_S f(x^0; T) := (T^\top)^\dagger \, [f(x^0 + t^j) - f(x^0)]_{j = 1 \ldots k}$$1. This composite rule yields an exact Hessian when $$\nabla_S f(x^0; T) := (T^\top)^\dagger \, [f(x^0 + t^j) - f(x^0)]_{j = 1 \ldots k}$$2 and $$\nabla_S f(x^0; T) := (T^\top)^\dagger \, [f(x^0 + t^j) - f(x^0)]_{j = 1 \ldots k}$$3 are quadratic and maintains $$\nabla_S f(x^0; T) := (T^\top)^\dagger \, [f(x^0 + t^j) - f(x^0)]_{j = 1 \ldots k}$$4 error otherwise, generally with smaller constants [(Hare et al., 2020), Section 5].

6. Examples and Practical Implementation

In $$\nabla_S f(x^0; T) := (T^\top)^\dagger \, [f(x^0 + t^j) - f(x^0)]_{j = 1 \ldots k}$$5, the canonical minimal poised set with $$\nabla_S f(x^0; T) := (T^\top)^\dagger \, [f(x^0 + t^j) - f(x^0)]_{j = 1 \ldots k}$$6 leads to a collection of six points (including cross-pairwise offsets). Applying the nested-set Hessian in this scenario yields exact recovery of the Hessian for quadratic $$\nabla_S f(x^0; T) := (T^\top)^\dagger \, [f(x^0 + t^j) - f(x^0)]_{j = 1 \ldots k}$$7, and for general thrice differentiable $$\nabla_S f(x^0; T) := (T^\top)^\dagger \, [f(x^0 + t^j) - f(x^0)]_{j = 1 \ldots k}$$8, ensures $$\nabla_S f(x^0; T) := (T^\top)^\dagger \, [f(x^0 + t^j) - f(x^0)]_{j = 1 \ldots k}$$9 error. Numerical benchmarks are not extensively reported, but the theoretical error rate and recoverability on quadratics are established.

A plausible implication is that these approximations are well-suited for embedding in model-based trust-region or line-search DFO algorithms, where derivative information is unavailable or expensive. The approaches' first-order accuracy combined with flexibility regarding the number and placement of points confers substantial utility for high-dimensional and ill-conditioned problems in practice.

---

The nested-set Hessian, along with its calculus-based variants, provides a flexible, derivative-free second-order approximation framework with provable accuracy guarantees and exactness on quadratics, directly facilitating advanced model-based DFO algorithms (Hare et al., 2020).