---
title: Nested-Set Hessian
url: https://www.emergentmind.com/topics/nested-set-hessian
type: topic
---

# Nested-Set Hessian

The nested-set Hessian is a derivative-free, first-order accurate method for approximating the Hessian matrix $\nabla^2 f(x^0)$ of a twice-differentiable scalar function $f: \mathbb{R}^n \rightarrow \mathbb{R}$, specifically designed for use in settings where explicit derivatives are unavailable. Rooted in the theory of generalized simplex gradients, the approach constructs the Hessian by combining two nested difference schemes that efficiently reutilize function evaluations. For carefully chosen sample structures, the nested-set Hessian achieves a minimal evaluation cost of $(n+1)(n+2)/2$ while providing explicit error bounds of order $O(\Delta)$ in the sampling radii. Its framework further permits calculus-based composition rules and extension to overdetermined sets, supporting flexible application within model-based derivative-free optimization routines [2011.02584].

## 1. Mathematical Foundations and Formal Construction

Given a base point $x^0 \in \mathbb{R}^n$, the method relies on two sets of directions:
- Inner set $T = [t^1\, \ldots\, t^k] \in \mathbb{R}^{n \times k}$ (full row rank)
- Outer set $S = [s^1\, \ldots\, s^m] \in \mathbb{R}^{n \times m}$ (full row rank)

The generalized simplex gradient at $x^0$ with respect to $T$ is defined as 
\[
\nabla_s f(x^0; T) = (T^\top)^\dagger \delta_f(x^0; T),
\]
where 
\[
\delta_f(x^0; T) = [f(x^0 + t^j) - f(x^0)]_{j=1}^k,
\]
and $(\cdot)^\dagger$ denotes the Moore–Penrose pseudoinverse. For $k = n$ and nonsingular $T$, this recovers the classical simplex gradient.

The nested-set Hessian introduces a further set $S$ for a second layer of differencing. For each $s^i$, compute both $\nabla_s f(x^0 + s^i; T)$ and $\nabla_s f(x^0; T)$. Construct the matrix
\[
\Delta_\nabla(x^0; S, T) = 
\begin{bmatrix}
(\nabla_s f(x^0 + s^1; T) - \nabla_s f(x^0; T))^\top \\
\vdots \\
(\nabla_s f(x^0 + s^m; T) - \nabla_s f(x^0; T))^\top
\end{bmatrix}
\]
and set
\[
\nabla^2_s f(x^0; S, T) = (S^\top)^\dagger \Delta_\nabla(x^0; S, T).
\]
This estimator approximates directional second derivatives via differences of approximate gradients. The error bound is given by
\[
\|\nabla^2_s f(x^0;S,T) - \nabla^2 f(x^0)\| \leq \mathcal{C} \Delta_u,
\]
where $\Delta_u = \max\{\max_i \|s^i\|,\, \max_j \|t^j\|\}$, and $\mathcal{C}$ is explicit in the pseudoinverse norms and the Lipschitz constant of the true Hessian [2011.02584].

## 2. Evaluation Complexity and Minimal Point Sets

The generic requirement for function evaluations is at most $1 + k + m + mk$, comprising $x^0$, $x^0 + t^j$, $x^0 + s^i$, and $x^0 + s^i + t^j$ (for all $i,\,j$). However, if $m = k = n$ and both $S$ and $T$ have favourable structure, numerous points coincide and only
\[
(n+1)(n+2)/2
\]
distinct function evaluations are necessary. A minimal poised set for the nested-set Hessian consists of $(x^0; S, T)$ with $S, T \in \mathbb{R}^{n\times n}$, such that the collection 
\[
\{x^0\} \cup \{x^0 + s^i\}_{i=1}^n \cup \{x^0 + t^j\}_{j=1}^n \cup \{x^0 + s^i + t^j\}_{i,j=1}^n
\]
has cardinality exactly $(n+1)(n+2)/2$. One canonical construction uses $S = I_n$ and $T = E_k$ (specific difference matrices), and all full-row-rank $S, T$ can be generated via transformations from this canonical case [2011.02584].

The method readily generalizes to $m > n$ or $k > n$ (overdetermined geometry), enhancing stability via reduced pseudoinverse norms, though at the cost of additional function values.

| Configuration            | Function Evaluations | Accuracy    |
|--------------------------|---------------------|-------------|
| General ($m, k$)         | $1 + m + k + mk$    | $O(\Delta_u)$ |
| Minimal $m = k = n$      | $(n+1)(n+2)/2$      | $O(\Delta)$ |
| Quadratic Interpolation  | $(n+1)(n+2)/2$      | $O(\Delta^2)$ gradient / exact Hessian for quadratics |

## 3. Error Bounds and Stability Considerations

Assuming $f \in C^3$ near $x^0$ with Lipschitz constant $L_{\nabla^2 f}$, the error in the nested-set Hessian is bounded by
\[
\|\nabla^2_s f(x^0;S,T) - \nabla^2 f(x^0)\| 
\leq (m\sqrt{k}/3) L_{\nabla^2 f} \left(2 \frac{\Delta_u}{\Delta_\ell} + 3\right) \|(S/\Delta_S)^\dagger\| \|(T/\Delta_T)^\dagger\| \Delta_u,
\]
where $\Delta_\ell = \min\{\Delta_S, \Delta_T\}$. For $\Delta_S = \Delta_T = \Delta$,
\[
\|\ldots\| \leq \frac{5 m \sqrt{k}}{3} L_{\nabla^2 f} \|(S/\Delta)^\dagger\| \|(T/\Delta)^\dagger\| \Delta = O(\Delta).
\]
Stability is governed by the conditioning of $S^\top$ and $T^\top$ (through the pseudoinverse norms). For highly nonlinear $f$, $\Delta$ must be small to guarantee control of $O(\Delta)$ error [2011.02584].

## 4. Algorithmic Procedure

The computation proceeds through the following steps:

``` 
Input: x0 ∈ ℝⁿ, S=[s¹…sᵐ] ∈ ℝⁿ×ᵐ, T=[t¹…tᵏ] ∈ ℝⁿ×ᵏ
1. Evaluate f0 = f(x0)
2. For j = 1…k, evaluate fT[j] = f(x0 + t^j)
3. Compute δ0 = fT – f0 · 1ₖ, then ∇s0 = (Tᵀ)† δ0
4. For i = 1…m:
     Evaluate fS[i] = f(x0 + s^i)
     For j = 1…k, evaluate fST[i,j] = f(x0 + s^i + t^j)
     Compute δi[j] = fST[i,j] – fS[i]
     Compute ∇s_i = (Tᵀ)† δi
     Set row i of Δ_∇ = (∇s_i – ∇s0)ᵀ
5. Compute H = (Sᵀ)† Δ_∇
Output: Approximate Hessian H ≈ ∇²f(x⁰)
```

This nested structure enables efficient reuse of function values, particularly in minimal sets [2011.02584].

## 5. Calculus-Based Approximation Variants

Two calculus-based Hessian approximation schemes are developed to address composite expressions:
- **Simplex calculus Hessian** replaces gradient and Hessian terms in standard product, quotient, and power rules with the corresponding nested-set estimators, e.g.,
  \[
  \nabla^2_{sc}(f g) = (\nabla^2_s f) g(x^0) + \nabla_s f (\nabla_s g)^\top + \nabla_s g (\nabla_s f)^\top + (\nabla^2_s g) f(x^0)
  \]
  This variant exhibits $O(\Delta_u)$ error and exactness for affine components.

- **Quadratic calculus Hessian** constructs quadratic interpolation models $Q_f, Q_g$ for $f, g$ and uses these for gradient terms, while retaining nested-set Hessians in second-derivative terms. For composite $f g$:
  \[
  \nabla^2_{qc}(f g) = (\nabla^2_s f) g(x^0) + \nabla Q_f (\nabla Q_g)^\top + \nabla Q_g (\nabla Q_f)^\top + (\nabla^2_s g) f(x^0)
  \]
  This formula yields $O(\Delta_u)$ accuracy and is exact whenever all factors are quadratic [2011.02584].

## 6. Comparative Assessment and Applicability

Compared to alternative approaches, the nested-set Hessian matches the minimal evaluation cost of quadratic interpolation while simplifying implementation by relying on nested simplex-gradient calculations rather than solving a global system. In contrast, finite-difference Hessians using central difference of gradients require $O(n^2)$ gradient evaluations—translating to $O(n^3)$ function values due to the cost of gradient estimation.

The nested-set Hessian offers:
- Explicit $O(\Delta_u)$ error for the Hessian estimator,
- Minimal $(n+1)(n+2)/2$ cost for well-structured sets,
- Flexible extension to overdetermined (stabilized) configurations,
- Compatibility with calculus-based rules supporting combinations of estimated Hessians.

Its limitations include only first-order ($O(\Delta)$) accuracy, sensitivity to ill-conditioning in $S^\top$ and $T^\top$, and the need for sufficiently small $\Delta_u$ to control error for highly nonlinear functions. Quadratic interpolation, by contrast, achieves higher gradient accuracy ($O(\Delta^2)$) and exactness for quadratics but is more complex to implement [2011.02584].

Overall, the nested-set Hessian supplies an efficient, robust, and implementable framework for derivative-free second-order information that is readily integrated into model-based optimization routines and advanced calculus operations on estimated models.

Source: https://www.emergentmind.com/topics/nested-set-hessian