Zero-Order Gradient Consistency Residue

• Zero-Order Gradient Consistency Residue is a diagnostic measure that quantifies the mismatch between surrogate gradients and target derivatives across optimization and simulation frameworks.
• It is analyzed using methods such as complex-step, Gaussian smoothing, and residual feedback, with biases typically diminishing at an order of O(δ²) and influencing convergence rates and oracle complexity.
• The concept is applied in diverse areas including zeroth-order optimization, nonsmooth stochastic programming, neural signed distance functions, and conservative SPH, highlighting its role in error correction and numerical stability.

Searching arXiv for the cited papers and closely related work to ground the article in the literature. I’m checking the arXiv record for the core paper and adjacent works on zeroth-order consistency residue. Zero-order gradient consistency residue denotes a family of discrepancy measures that compare a gradient surrogate, a smoothed gradient, or a conservative discrete gradient with its target object. In the cited literature, the target may be the true gradient $\nabla f(x)$, the stationarity condition $0\in \nabla f_\eta(x)+N_X(x)$, the vanishing gradient of a constant field, or the parallelism of level-set normals. Taken together, these works indicate that the term is not a single standardized scalar but a unifying diagnostic for bias, residual stationarity, and discrete consistency across zeroth-order optimization, nonsmooth stochastic programming, neural signed distance fields, and conservative SPH (Jongeneel et al., 2021, Marrinan et al., 2023, Wang et al., 24 Jul 2025, Ma et al., 2023).

1. Definitions and terminological scope

The literature instantiates the residue through several non-equivalent but structurally related definitions (Jongeneel et al., 2021, Shibaev et al., 2020, Marrinan et al., 2023, Yuan et al., 18 May 2026, Wang et al., 24 Jul 2025, Ma et al., 2023).

Domain	Residue definition	Vanishing condition
Complex-step zeroth-order optimization	$$R(x;\delta)=\mathbb E[\hat G(x;\delta)]-\nabla f(x)=\nabla f_\delta(x)-\nabla f(x)$$	Smoothed gradient matches $\nabla f(x)$ as $\delta\to0$
Gaussian smoothing with noisy oracle	$$R_\mu(x,\delta)\equiv\|\nabla \tilde f_\mu(x,\delta)-\nabla f(x)\|_*$$	Smoothed noisy gradient matches true gradient
Nonsmooth constrained optimization	$R_\eta(x)=x-\Pi_X[x-\nabla f_\eta(x)]$	$0\in \nabla f_\eta(x)+N_X(x)$
ZO hard-thresholding	$R(x):=\|\hat g_{q,\mu}(x)-\nabla f(x)\|_2$	Finite-difference estimator matches $\nabla f(x)$
Conservative SPH	$0\in \nabla f_\eta(x)+N_X(x)$0 or $0\in \nabla f_\eta(x)+N_X(x)$1	Constant fields have zero discrete gradient
Neural SDFs	$0\in \nabla f_\eta(x)+N_X(x)$2	Level-set gradients are parallel

The common structure is a mismatch between an implementable surrogate and the derivative-based object one would use in a first-order or consistency-exact formulation. What changes across domains is the operational meaning of consistency: unbiasedness after smoothing, projected stationarity, exact annihilation of constants, or alignment of normals. This suggests that “residue” functions as a domain-specific error observable rather than a universally fixed mathematical quantity.

2. Bias, variance, and oracle complexity in zeroth-order optimization

In static zeroth-order optimization, the residue is usually the bias between the expectation of a random gradient estimator and the true gradient. For the one-point complex-step method, if $0\in \nabla f_\eta(x)+N_X(x)$3 is real-analytic and admits a holomorphic extension to $0\in \nabla f_\eta(x)+N_X(x)$4, with $0\in \nabla f_\eta(x)+N_X(x)$5 uniform on $0\in \nabla f_\eta(x)+N_X(x)$6, the estimator is

$0\in \nabla f_\eta(x)+N_X(x)$7

and the smoothed surrogate is

$0\in \nabla f_\eta(x)+N_X(x)$8

The multivariate Cauchy–Riemann identities and the divergence theorem give

$0\in \nabla f_\eta(x)+N_X(x)$9

Accordingly, the consistency residue is

$$R(x;\delta)=\mathbb E[\hat G(x;\delta)]-\nabla f(x)=\nabla f_\delta(x)-\nabla f(x)$$0

Under $$R(x;\delta)=\mathbb E[\hat G(x;\delta)]-\nabla f(x)=\nabla f_\delta(x)-\nabla f(x)$$1, the Taylor expansion of the holomorphic extension yields

$$R(x;\delta)=\mathbb E[\hat G(x;\delta)]-\nabla f(x)=\nabla f_\delta(x)-\nabla f(x)$$2

so the bias is $$R(x;\delta)=\mathbb E[\hat G(x;\delta)]-\nabla f(x)=\nabla f_\delta(x)-\nabla f(x)$$3. Corollary 3.5 gives

$$R(x;\delta)=\mathbb E[\hat G(x;\delta)]-\nabla f(x)=\nabla f_\delta(x)-\nabla f(x)$$4

so as $$R(x;\delta)=\mathbb E[\hat G(x;\delta)]-\nabla f(x)=\nabla f_\delta(x)-\nabla f(x)$$5 the dominant term is $$R(x;\delta)=\mathbb E[\hat G(x;\delta)]-\nabla f(x)=\nabla f_\delta(x)-\nabla f(x)$$6 and the variance remains $$R(x;\delta)=\mathbb E[\hat G(x;\delta)]-\nabla f(x)=\nabla f_\delta(x)-\nabla f(x)$$7. By contrast, a classical one-point finite-difference estimator based on $$R(x;\delta)=\mathbb E[\hat G(x;\delta)]-\nabla f(x)=\nabla f_\delta(x)-\nabla f(x)$$8 has $$R(x;\delta)=\mathbb E[\hat G(x;\delta)]-\nabla f(x)=\nabla f_\delta(x)-\nabla f(x)$$9 as $\nabla f(x)$0. Projected-gradient-descent-style schemes using $\nabla f(x)$1 attain the same oracle-complexity bounds as state-of-the-art two-point methods: for convex $\nabla f(x)$2-smooth $\nabla f(x)$3, $\nabla f(x)$4; for $\nabla f(x)$5-strongly convex $\nabla f(x)$6, $\nabla f(x)$7; and for nonconvex $\nabla f(x)$8-smooth $\nabla f(x)$9, $\delta\to0$0 (Jongeneel et al., 2021).

For Gaussian smoothing with an inexact oracle $\delta\to0$1 satisfying $\delta\to0$2, the estimator

$\delta\to0$3

satisfies $\delta\to0$4. Under a Hölder-continuous gradient,

$\delta\to0$5

The analysis explicitly decomposes

$\delta\to0$6

where the middle term is the consistency residue and the last term is a zero-mean variance term. In the smooth case $\delta\to0$7, the dependence on dimension improves to $\delta\to0$8 for achieving $\delta\to0$9, and the tolerated noise can be chosen of order $$R_\mu(x,\delta)\equiv\|\nabla \tilde f_\mu(x,\delta)-\nabla f(x)\|_*$$0 (Shibaev et al., 2020).

This performance must be read against the information-theoretic lower bound for zero-order stochastic gradient estimation. In the model where an estimator has access only to noisy function values, the minimax error is $$R_\mu(x,\delta)\equiv\|\nabla \tilde f_\mu(x,\delta)-\nabla f(x)\|_*$$1. The classical coordinate-wise finite-difference method is not minimax optimal: its $$R_\mu(x,\delta)\equiv\|\nabla \tilde f_\mu(x,\delta)-\nabla f(x)\|_*$$2-risk is $$R_\mu(x,\delta)\equiv\|\nabla \tilde f_\mu(x,\delta)-\nabla f(x)\|_*$$3 when third derivatives are bounded but nonzero, and $$R_\mu(x,\delta)\equiv\|\nabla \tilde f_\mu(x,\delta)-\nabla f(x)\|_*$$4 when the third-order term vanishes. The lower bound therefore isolates an intrinsic gap between generic zero-order access and the performance of standard finite differences (Alabdulkareem et al., 2020).

3. Residual feedback and variance-reduced one-point methods

In online zeroth-order optimization with time-varying objectives $$R_\mu(x,\delta)\equiv\|\nabla \tilde f_\mu(x,\delta)-\nabla f(x)\|_*$$5, one-point residual feedback estimates the gradient using the residual between two feedback points at consecutive time instants. With $$R_\mu(x,\delta)\equiv\|\nabla \tilde f_\mu(x,\delta)-\nabla f(x)\|_*$$6 or sampled uniformly from the unit sphere, the estimator is

$$R_\mu(x,\delta)\equiv\|\nabla \tilde f_\mu(x,\delta)-\nabla f(x)\|_*$$7

Because

$$R_\mu(x,\delta)\equiv\|\nabla \tilde f_\mu(x,\delta)-\nabla f(x)\|_*$$8

the “old” terms from $$R_\mu(x,\delta)\equiv\|\nabla \tilde f_\mu(x,\delta)-\nabla f(x)\|_*$$9 cancel out in expectation, yielding

$R_\eta(x)=x-\Pi_X[x-\nabla f_\eta(x)]$0

The bias relative to $R_\eta(x)=x-\Pi_X[x-\nabla f_\eta(x)]$1 is therefore the smoothing error

$R_\eta(x)=x-\Pi_X[x-\nabla f_\eta(x)]$2

Under $R_\eta(x)=x-\Pi_X[x-\nabla f_\eta(x)]$3-Lipschitz continuity and bounded temporal variation $R_\eta(x)=x-\Pi_X[x-\nabla f_\eta(x)]$4, the second moment obeys a contraction inequality and telescopes to a uniform bound $R_\eta(x)=x-\Pi_X[x-\nabla f_\eta(x)]$5. By contrast, the classical one-point estimator $R_\eta(x)=x-\Pi_X[x-\nabla f_\eta(x)]$6 requires $R_\eta(x)=x-\Pi_X[x-\nabla f_\eta(x)]$7 and has $R_\eta(x)=x-\Pi_X[x-\nabla f_\eta(x)]$8. The residual-feedback assumption is therefore bounded change rather than bounded function magnitude. The corresponding regret bounds are $R_\eta(x)=x-\Pi_X[x-\nabla f_\eta(x)]$9 for convex–Lipschitz objectives, $0\in \nabla f_\eta(x)+N_X(x)$0 for convex–smooth objectives, and vanishing average gradient-regret under the stated $0\in \nabla f_\eta(x)+N_X(x)$1 and $0\in \nabla f_\eta(x)+N_X(x)$2 conditions in the nonconvex settings. Numerical experiments on LQR control with drifting dynamics and multi-agent resource allocation with varying penalty show that residual feedback matches the performance of the two-point oracle and greatly outperforms the classical one-point method (Zhang et al., 2020).

A different residue mechanism appears in zero-order hard-thresholding for $0\in \nabla f_\eta(x)+N_X(x)$3-constrained optimization. There, the estimator

$0\in \nabla f_\eta(x)+N_X(x)$4

induces the residue

$0\in \nabla f_\eta(x)+N_X(x)$5

Under Restricted Strong Smoothness, $0\in \nabla f_\eta(x)+N_X(x)$6 and

$0\in \nabla f_\eta(x)+N_X(x)$7

while the mean-squared residue satisfies

$0\in \nabla f_\eta(x)+N_X(x)$8

with all three $0\in \nabla f_\eta(x)+N_X(x)$9-terms decaying as $R(x):=\|\hat g_{q,\mu}(x)-\nabla f(x)\|_2$0. The difficulty is that hard-thresholding is expansive: $R(x):=\|\hat g_{q,\mu}(x)-\nabla f(x)\|_2$1 Thus any gradient error is amplified by $R(x):=\|\hat g_{q,\mu}(x)-\nabla f(x)\|_2$2. The variance-reduced memory-based estimator $R(x):=\|\hat g_{q,\mu}(x)-\nabla f(x)\|_2$3 mitigates this contradiction and removes SZOHT’s restriction on the number of random directions, yielding linear convergence of the form

$R(x):=\|\hat g_{q,\mu}(x)-\nabla f(x)\|_2$4

and zeroth-order query complexity

$R(x):=\|\hat g_{q,\mu}(x)-\nabla f(x)\|_2$5

This suggests that in sparse constrained settings, controlling the residue is not only a bias-variance issue but also an operator-stability issue (Yuan et al., 18 May 2026).

4. Residual mappings for nonsmooth nonconvex stochastic optimization

For nonsmooth expectation-valued objectives $R(x):=\|\hat g_{q,\mu}(x)-\nabla f(x)\|_2$6 over a closed convex set $R(x):=\|\hat g_{q,\mu}(x)-\nabla f(x)\|_2$7, smoothing is used to define a differentiable surrogate

$R(x):=\|\hat g_{q,\mu}(x)-\nabla f(x)\|_2$8

where $R(x):=\|\hat g_{q,\mu}(x)-\nabla f(x)\|_2$9 is uniform on the unit ball or sphere scaled by $\nabla f(x)$0. The smoothed function satisfies $\nabla f(x)$1, and $\nabla f(x)$2 exists. Approximate stationarity is measured by the projected-gradient residual

$\nabla f(x)$3

or, when $\nabla f(x)$4,

$\nabla f(x)$5

This residual vanishes if and only if $\nabla f(x)$6. Because spherical smoothing gives $\nabla f(x)$7, the implication

$\nabla f(x)$8

shows that a zero residual for the smoothed problem is an $\nabla f(x)$9-Clarke-stationary point of the original problem. More quantitatively, if $0\in \nabla f_\eta(x)+N_X(x)$00, one obtains an $0\in \nabla f_\eta(x)+N_X(x)$01-Clarke-stationary solution (Marrinan et al., 2023).

The smoothing-enabled variance-reduced zeroth-order gradient framework (VRG-ZO) and the zeroth-order stochastic quasi-Newton scheme (VRSQN-ZO) use this residual as the central convergence observable. For VRG-ZO, if the step sizes are constant but sufficiently small or diminishing, and the mini-batch sizes grow so that $0\in \nabla f_\eta(x)+N_X(x)$02, then

$0\in \nabla f_\eta(x)+N_X(x)$03

To ensure $0\in \nabla f_\eta(x)+N_X(x)$04, the required number of iterations is $0\in \nabla f_\eta(x)+N_X(x)$05 and the total number of function evaluations is $0\in \nabla f_\eta(x)+N_X(x)$06. For VRSQN-ZO, the corresponding iteration and sample complexities are $0\in \nabla f_\eta(x)+N_X(x)$07 and $0\in \nabla f_\eta(x)+N_X(x)$08, respectively. In this setting, “residual” is not the bias of a single gradient estimator; it is the projected distance to the KKT-like stationarity condition of the smoothed problem.

5. Gradient-consistency residue in neural signed distance functions

In neural signed distance functions, gradient consistency is formulated geometrically rather than as a stochastic-estimation bias. Let $0\in \nabla f_\eta(x)+N_X(x)$09 be the network SDF and $0\in \nabla f_\eta(x)+N_X(x)$10 its zero-level set. For a query $0\in \nabla f_\eta(x)+N_X(x)$11, the cosine-distance residue between gradients at $0\in \nabla f_\eta(x)+N_X(x)$12 and at a point $0\in \nabla f_\eta(x)+N_X(x)$13 on any level set $0\in \nabla f_\eta(x)+N_X(x)$14 is

$0\in \nabla f_\eta(x)+N_X(x)$15

To avoid evaluating all level sets, the method projects $0\in \nabla f_\eta(x)+N_X(x)$16 onto the zero-level set along the local SDF gradient,

$0\in \nabla f_\eta(x)+N_X(x)$17

and uses

$0\in \nabla f_\eta(x)+N_X(x)$18

The batch loss is

$0\in \nabla f_\eta(x)+N_X(x)$19

which emphasizes points near the surface (Ma et al., 2023).

The stated rationale is that gradient consistency in the field, indicated by the parallelism of level sets, is the key factor affecting inference accuracy when signed distance supervision is unavailable. By projecting every query onto $0\in \nabla f_\eta(x)+N_X(x)$20 and enforcing parallel gradients at $0\in \nabla f_\eta(x)+N_X(x)$21 and $0\in \nabla f_\eta(x)+N_X(x)$22, the method “anchors” all level sets to the zero set in a pairwise fashion and propagates the zero level set to everywhere in the field. Empirically, the alignment term improves both numerical and visual accuracy across point-cloud and multi-view settings. Reported results include CD on the SIREN “Thai” object improving from $0\in \nabla f_\eta(x)+N_X(x)$23 to $0\in \nabla f_\eta(x)+N_X(x)$24, Stanford Scan with NeuralPull+Ours improving from CD $0\in \nabla f_\eta(x)+N_X(x)$25 to $0\in \nabla f_\eta(x)+N_X(x)$26 and NC $0\in \nabla f_\eta(x)+N_X(x)$27 to $0\in \nabla f_\eta(x)+N_X(x)$28, the 3D Scene “Lounge” improving from CD $0\in \nabla f_\eta(x)+N_X(x)$29 to $0\in \nabla f_\eta(x)+N_X(x)$30 and NC $0\in \nabla f_\eta(x)+N_X(x)$31 to $0\in \nabla f_\eta(x)+N_X(x)$32, DTU mean CD for NeuS improving from $0\in \nabla f_\eta(x)+N_X(x)$33 to $0\in \nabla f_\eta(x)+N_X(x)$34, and ScanNet mean CD for MonoSDF improving from $0\in \nabla f_\eta(x)+N_X(x)$35 to $0\in \nabla f_\eta(x)+N_X(x)$36 with F-score increasing from $0\in \nabla f_\eta(x)+N_X(x)$37 to $0\in \nabla f_\eta(x)+N_X(x)$38. Ablations further report best performance around $0\in \nabla f_\eta(x)+N_X(x)$39, better performance for $0\in \nabla f_\eta(x)+N_X(x)$40 with $0\in \nabla f_\eta(x)+N_X(x)$41 than uniform weighting, and stronger alignment from cosine residues than from $0\in \nabla f_\eta(x)+N_X(x)$42 MSE on normalized gradients. A common misconception is that this residue is equivalent to signed-distance supervision; the paper instead treats it as a general regularization term that can be used upon different methods, including NeuralPull and NeuS.

6. Conservative SPH, zero-order consistency failure, and background pressure

In conservative SPH, zero-order consistency requires the discrete gradient of a constant field to vanish exactly. For the standard anti-symmetric discretization,

$0\in \nabla f_\eta(x)+N_X(x)$43

zero-order consistency demands

$0\in \nabla f_\eta(x)+N_X(x)$44

The quantity

$0\in \nabla f_\eta(x)+N_X(x)$45

is the zero-order gradient consistency residue. Applying the operator to $0\in \nabla f_\eta(x)+N_X(x)$46 gives

$0\in \nabla f_\eta(x)+N_X(x)$47

so failed zero-order consistency is exactly the nonvanishing gradient of a constant field. The standard Kernel-Gradient Correction matrix

$0\in \nabla f_\eta(x)+N_X(x)$48

yields the non-conservative corrected gradient

$0\in \nabla f_\eta(x)+N_X(x)$49

A straightforward conservative analogue,

$0\in \nabla f_\eta(x)+N_X(x)$50

still leaves the residue

$0\in \nabla f_\eta(x)+N_X(x)$51

which in general does not vanish. Reverse-KGC instead swaps the correction matrices,

$0\in \nabla f_\eta(x)+N_X(x)$52

and, under the KGC-matrix relaxation condition

$0\in \nabla f_\eta(x)+N_X(x)$53

one obtains

$0\in \nabla f_\eta(x)+N_X(x)$54

In the circle-of-radius-1 convergence study with $0\in \nabla f_\eta(x)+N_X(x)$55, RKGC with B-relaxation retains a clean $0\in \nabla f_\eta(x)+N_X(x)$56 slope down to $0\in \nabla f_\eta(x)+N_X(x)$57, with $0\in \nabla f_\eta(x)+N_X(x)$58 errors dropping from $0\in \nabla f_\eta(x)+N_X(x)$59 at coarse resolution to $0\in \nabla f_\eta(x)+N_X(x)$60 at fine resolution, whereas SKGC degenerates to first order and NKGC stagnates or deteriorates (Zhang et al., 2024).

A later analysis links the same residue to non-physical numerical damping in conservative SPH fluid dynamics. For the conservative pressure-force discretization,

$0\in \nabla f_\eta(x)+N_X(x)$61

one can write

$0\in \nabla f_\eta(x)+N_X(x)$62

so the second term is precisely the zero-order gradient-consistency residue for $0\in \nabla f_\eta(x)+N_X(x)$63. If a constant background pressure $0\in \nabla f_\eta(x)+N_X(x)$64 is added, the residue becomes

$0\in \nabla f_\eta(x)+N_X(x)$65

Hence background pressure amplifies the residual force in the direction of the consistency error (Wang et al., 24 Jul 2025).

Scenario	Setup	Reported effect
Pressure-driven channel flow	$0\in \nabla f_\eta(x)+N_X(x)$66, $0\in \nabla f_\eta(x)+N_X(x)$67, no correction	Velocity loss $0\in \nabla f_\eta(x)+N_X(x)$68, $0\in \nabla f_\eta(x)+N_X(x)$69, $0\in \nabla f_\eta(x)+N_X(x)$70, $0\in \nabla f_\eta(x)+N_X(x)$71
Inviscid standing wave	$0\in \nabla f_\eta(x)+N_X(x)$72, $0\in \nabla f_\eta(x)+N_X(x)$73	Fitted decay rates $0\in \nabla f_\eta(x)+N_X(x)$74
FDA nozzle	2D normal/reversed, 3D normal	Losses $0\in \nabla f_\eta(x)+N_X(x)$75, $0\in \nabla f_\eta(x)+N_X(x)$76, $0\in \nabla f_\eta(x)+N_X(x)$77 without RKGC; a few percent with RKGC

These results address a recurring misconception in SPH practice: particle regularization or transport velocity formulation may mitigate disorder, but the zero-order consistency issue still significantly damps the flow in a long channel for both laminar and turbulent simulations. The RKGC correction is proved effective in reducing the residue effect, yet its correction capability is fundamentally limited, especially at finite resolution, near boundaries, or under unavoidable high background pressure. In this literature, the residue is therefore not merely a local truncation artifact; it is treated as the common root cause of excessive numerical dissipation in pressure-driven channels, gravity-driven free-surface flows, and complex engineering geometries such as the FDA nozzle.