Zeroth-Order Optimization on High-Res Images (ZO-high)

• The paper introduces ZO-high as derivative-free optimization for high-res images, tackling black-box objectives and extreme dimensionality challenges.
• It outlines specialized gradient estimators—including two-point, one-point multi-direction, and block-coordinate methods—to manage bias, variance, and high query costs.
• The design space spans hierarchical training, compressed-sensing techniques, and adaptive variance-aware updates, offering efficient strategies for deep CNNs and adversarial image attacks.

Searching arXiv for the cited papers and topic-specific context. Zeroth-order optimization on high-resolution images, often abbreviated ZO-high, denotes derivative-free optimization in image spaces whose dimensionality is determined by large spatial resolutions, typically with variables $x \in [0,1]^{H\times W\times C}$ and ambient dimension $d=HWC$. The setting is motivated by black-box objectives, non-differentiable pipelines, and regimes in which backpropagation is inaccessible or unaffordable. Its central difficulty is not merely the absence of gradients, but the interaction between image resolution, network depth, estimator variance, and query budget: coordinate-wise finite differences scale with spatial extent, random-direction estimators incur variance that grows with $d$, and conventional layer-by-layer zeroth-order training of deep networks exhibits quadratic dependence on depth (Cao et al., 11 Feb 2026, Shu et al., 3 Feb 2025, Jin et al., 22 Oct 2025).

1. Problem class and scaling bottlenecks

The canonical zeroth-order problem is

$\min_{x\in \mathcal{X}\subset \mathbb{R}^d} f(x),$

where only function values are observable (Cai et al., 2021). In the image setting, $d=HWC$ can be very large even before any transform-domain lifting, and the effective dimension may become larger still in wavelet or continuous-wavelet representations; one reported audio-wavelet configuration reaches $d=1{,}776{,}000$, while an ImageNet discrete-wavelet configuration is reported with $d\approx 676{,}353$ (Cai et al., 2021).

Two scaling bottlenecks dominate ZO-high. First, high-resolution CNNs produce large feature maps $A\in \mathbb{R}^{C\times H\times W}$, so estimating gradients for all spatial positions in standard zeroth-order schemes would require $O(HW)$ forward queries per layer (Cao et al., 11 Feb 2026). Second, in random-direction estimators the variance grows linearly in the ambient dimension. For the one-point multi-direction estimator used in R-AdaZO, the coordinate-wise variance bound is

$$\Sigma^2 \triangleq \frac{8(\sigma^2 + C^2)d}{K\mu^2},$$

which decreases with the number of directions $d=HWC$0 and increases as $d=HWC$1 becomes small (Shu et al., 3 Feb 2025). This creates the standard bias-variance-query trade-off: larger $d=HWC$2 reduces estimator variance but increases query cost, while smaller $d=HWC$3 reduces smoothing bias but amplifies numerical sensitivity and stochastic noise.

Depth introduces an additional, distinct obstacle. In non-hierarchical activation-space ZO for deep networks of depth $d=HWC$4 and average width $d=HWC$5, repeated long forward paths induce $d=HWC$6 query complexity (Cao et al., 11 Feb 2026). In high-resolution models, the depth and resolution effects compound: the optimization variable is large, the intermediate activations are spatially extensive, and the cost of each function evaluation itself scales with the input size.

2. Gradient estimators and objective formulations

A standard reference point in ZO-high is the two-point estimator

$d=HWC$7

which estimates the gradient of a smoothed objective and is widely used in black-box optimization (Cao et al., 11 Feb 2026). R-AdaZO instead develops its theory around a one-point, multi-direction estimator with directions sampled uniformly from the sphere:

$d=HWC$8

for which

$d=HWC$9

The latter inequality makes explicit that smoothing bias scales with both $d$0 and $d$1 (Shu et al., 3 Feb 2025).

Coordinate and block estimators form the main alternative to random-direction methods. In the constrained block framework of ZOB-GDA and ZOB-SGDA, for a block $d$2 of size $d$3 and coordinate unit vector $d$4, one-sided finite differences are defined by

$d$5

with the block estimator

$d$6

Under this model, a single gradient step requires $d$7 shifted queries plus, at most, a single base evaluation, so the per-step query cost is $d$8 (Jin et al., 22 Oct 2025).

ZO-high objectives are often written either directly in image space or in a transform domain. A standard adversarial formulation is

$d$9

with projection onto an $\min_{x\in \mathcal{X}\subset \mathbb{R}^d} f(x),$0 ball and clipping to valid pixel bounds (Shu et al., 3 Feb 2025). In the wavelet-domain attack studied with ZO-BCD, the objective is written as

$\min_{x\in \mathcal{X}\subset \mathbb{R}^d} f(x),$1

optionally replacing the $\min_{x\in \mathcal{X}\subset \mathbb{R}^d} f(x),$2 term by an $\min_{x\in \mathcal{X}\subset \mathbb{R}^d} f(x),$3 or $\min_{x\in \mathcal{X}\subset \mathbb{R}^d} f(x),$4 constraint (Cai et al., 2021). These formulations are structurally different but share the same ZO constraint: the optimizer receives values, not gradients.

3. Hierarchical zeroth-order training for deep image models

Hierarchical Zeroth-Order optimization (HZO) addresses the depth bottleneck by decomposing the composite network along depth rather than perturbing the full parameter vector or propagating finite differences layer by layer. Writing

$\min_{x\in \mathcal{X}\subset \mathbb{R}^d} f(x),$5

and, for a subnetwork,

$\min_{x\in \mathcal{X}\subset \mathbb{R}^d} f(x),$6

HZO bisects the interval $\min_{x\in \mathcal{X}\subset \mathbb{R}^d} f(x),$7 at $\min_{x\in \mathcal{X}\subset \mathbb{R}^d} f(x),$8, computes the intermediate activation $\min_{x\in \mathcal{X}\subset \mathbb{R}^d} f(x),$9, estimates the Jacobian of the right subnetwork by symmetric coordinate perturbations,

$d=HWC$0

and propagates the target signal via

$d=HWC$1

At the leaves, it applies the local delta rule

$d=HWC$2

The recursion reconstructs the chain rule over depth using Jacobian-target products rather than backpropagated gradients (Cao et al., 11 Feb 2026).

The main complexity result is a reduction from

$d=HWC$3

where $d=HWC$4 is depth and $d=HWC$5 is average width. The argument is based on a balanced binary recursion over depth: at recursion level $d=HWC$6, the right-side path length shrinks by $d=HWC$7 while the number of subproblems grows as $d=HWC$8, leaving per-level work constant at $d=HWC$9 and total work $d=1{,}776{,}000$0 (Cao et al., 11 Feb 2026). This is a structural reduction in query complexity, not a mere constant-factor engineering improvement.

The stability analysis is tied to the layer-wise Lipschitz factor $d=1{,}776{,}000$1. Under $d=1{,}776{,}000$2-Lipschitz continuity and $d=1{,}776{,}000$3-smoothness, the recursive HZO error satisfies

$d=1{,}776{,}000$4

Near the unitary limit, $d=1{,}776{,}000$5,

$d=1{,}776{,}000$6

so cumulative error scales linearly rather than exponentially in depth (Cao et al., 11 Feb 2026). The paper associates this regime with residual connections or orthogonal or normalized initialization, and reports that HZO maintains high gradient cosine similarity, approximately $d=1{,}776{,}000$7 to $d=1{,}776{,}000$8, in deep ResNets.

For CNNs on large images, HZO adds Spatial Parallel Perturbation (SPP). If $d=1{,}776{,}000$9 is the effective receptive field in the next layer and $d\approx 676{,}353$0 is chosen so that

$d\approx 676{,}353$1

then perturbations at those positions are independent with respect to the next layer. The reported consequence is a reduction of per-layer spatial query complexity from $d\approx 676{,}353$2 to $d\approx 676{,}353$3, making ZO-high effectively resolution-independent in the number of queries, although the cost of each forward pass still scales with $d\approx 676{,}353$4 and $d\approx 676{,}353$5 (Cao et al., 11 Feb 2026).

The empirical profile reported for HZO is training on CIFAR-10 to $d\approx 676{,}353$6 test accuracy in $d\approx 676{,}353$7 hours on a single NVIDIA A100 GPU, compared with a DeepZero baseline at $d\approx 676{,}353$8 in $d\approx 676{,}353$9 hours on $A\in \mathbb{R}^{C\times H\times W}$0V100, together with $A\in \mathbb{R}^{C\times H\times W}$1 top-1 accuracy on ImageNet-10 and gradient cosine similarity greater than $A\in \mathbb{R}^{C\times H\times W}$2 throughout training (Cao et al., 11 Feb 2026). The implementation notes further state that float32 is the minimum recommended precision for deep networks, float64 improves stability at increased cost, float16 leads to instability due to subtractive cancellation in finite differences, and GELU yields higher cosine similarity than ReLU.

4. Huge-scale block-coordinate and compressed-sensing methods

ZO-BCD targets the “huge-scale” regime, explicitly described as $A\in \mathbb{R}^{C\times H\times W}$3, where even basic vector operations on the decision variable are infeasible (Cai et al., 2021). Its starting point is that full-dimensional zeroth-order estimators are impractical: coordinatewise finite differences require $A\in \mathbb{R}^{C\times H\times W}$4 queries per gradient and $A\in \mathbb{R}^{C\times H\times W}$5 memory per iteration, while compressed-sensing full-gradient schemes still incur at least $A\in \mathbb{R}^{C\times H\times W}$6 memory and FLOPs in the sparse regime.

The algorithm partitions $A\in \mathbb{R}^{C\times H\times W}$7 into $A\in \mathbb{R}^{C\times H\times W}$8 blocks and updates one randomly selected block per iteration. For a chosen block $A\in \mathbb{R}^{C\times H\times W}$9, it uses one-sided finite differences with $O(HW)$0 sensing directions $O(HW)$1,

$O(HW)$2

stacks them as $O(HW)$3, and recovers a sparse block gradient through

$O(HW)$4

implemented via CoSaMP (Cai et al., 2021). The paper distinguishes ZO-BCD-R, which uses i.i.d. Rademacher directions, from ZO-BCD-RC, which uses rows of a circulant matrix generated by a single Rademacher seed vector. Under the stated assumptions, the total query complexity to attain an $O(HW)$5-optimal solution is $O(HW)$6 for both variants.

The block strategy is paired with an “almost equisparse” effect: if blocks are assigned by a random permutation and are equal-sized, then with high probability each block has sparsity approximately

$O(HW)$7

which reduces per-iteration query count by approximately $O(HW)$8 (Cai et al., 2021). The per-iteration computational profile is then reported as $O(HW)$9 FLOPs and memory for ZO-BCD-R, versus $$\Sigma^2 \triangleq \frac{8(\sigma^2 + C^2)d}{K\mu^2},$$0 FLOPs and $$\Sigma^2 \triangleq \frac{8(\sigma^2 + C^2)d}{K\mu^2},$$1 memory for ZO-BCD-RC.

The circulant construction is central for ZO-high because it compresses both storage and matrix-vector multiply costs. Rather than storing a dense sensing matrix of size $$\Sigma^2 \triangleq \frac{8(\sigma^2 + C^2)d}{K\mu^2},$$2, ZO-BCD-RC stores only the seed vector and the row index set, reducing storage from $$\Sigma^2 \triangleq \frac{8(\sigma^2 + C^2)d}{K\mu^2},$$3 to $$\Sigma^2 \triangleq \frac{8(\sigma^2 + C^2)d}{K\mu^2},$$4, and it exploits FFT-based circular convolutions to accelerate the CoSaMP inner loop (Cai et al., 2021).

The image-specific results are reported in transform space. On ImageNet with Inception-v3 in a three-level Daubechies “db45” DWT domain, ZO-BCD-R achieves attack success rate $$\Sigma^2 \triangleq \frac{8(\sigma^2 + C^2)d}{K\mu^2},$$5, mean $$\Sigma^2 \triangleq \frac{8(\sigma^2 + C^2)d}{K\mu^2},$$6 distortion $$\Sigma^2 \triangleq \frac{8(\sigma^2 + C^2)d}{K\mu^2},$$7, and mean queries to first success $$\Sigma^2 \triangleq \frac{8(\sigma^2 + C^2)d}{K\mu^2},$$8; ZO-BCD-RC achieves $$\Sigma^2 \triangleq \frac{8(\sigma^2 + C^2)d}{K\mu^2},$$9, $d=HWC$00, and $d=HWC$01; and a large-coefficients-only variant with approximately $d=HWC$02 dimension reduction achieves $d=HWC$03, $d=HWC$04, and $d=HWC$05 (Cai et al., 2021). The same paper reports untargeted baselines ZO-SCD at $d=HWC$06 ASR with $d=HWC$07 distortion $d=HWC$08 and $d=HWC$09 queries, ZO-SGD at $d=HWC$10, $d=HWC$11, and $d=HWC$12, ZO-AdaMM at $d=HWC$13, $d=HWC$14, and $d=HWC$15, and ZORO at $d=HWC$16, $d=HWC$17, and $d=HWC$18. In targeted audio attacks using a Morse CWT representation with $d=HWC$19 frequencies, ZO-BCD is reported to achieve attack success rate $d=HWC$20, mean relative loudness $d=HWC$21 dB, and mean queries approximately $d=HWC$22 (Cai et al., 2021).

5. Adaptive variance-aware optimization in image-scale dimensions

R-AdaZO addresses a different failure mode of ZO-high: the underutilization of moment information in adaptive zeroth-order methods (Shu et al., 3 Feb 2025). Its first update is the momentum recursion

$d=HWC$23

with no bias correction, specifically to preserve the variance-reduction effect of the first moment. The paper’s variance bound is

$d=HWC$24

so the first moment reduces variance by the factor $d=HWC$25 relative to the raw ZO estimator (Shu et al., 3 Feb 2025).

The second update departs from ZO-AdaMM by replacing $d=HWC$26 with $d=HWC$27:

$d=HWC$28

The update is then

$d=HWC$29

The paper’s interpretation is that the refined second moment better approximates the variance-free ideal than the conventional choice based on $d=HWC$30, because the variance contribution to $d=HWC$31 is reduced from $d=HWC$32 to $d=HWC$33 (Shu et al., 3 Feb 2025). This is the basis for its variance-aware convergence framework.

The convergence statement is given for non-convex objectives under bounded value, coordinate-wise Lipschitz smoothness, and bounded evaluation variance. The final bound takes the form

$d=HWC$34

with explicit dependence on $d=HWC$35, the smoothed-gradient bias, and the moment statistics (Shu et al., 3 Feb 2025). The paper states that this is the first variance-aware convergence framework for adaptive ZO methods and that R-AdaZO converges faster than ZO-AdaMM, with a speedup proportional to

$d=HWC$36

in the variance-dominant regime.

For high-resolution images, the method retains the query profile of standard direction-based ZO: $d=HWC$37 function evaluations per iteration for the one-point estimator, or $d=HWC$38 for a two-point symmetric estimator (Shu et al., 3 Feb 2025). The reported practical guidance is to use $d=HWC$39, $d=HWC$40, small $d=HWC$41, and task-dependent $d=HWC$42; for high-dimensional images, the paper recommends $d=HWC$43 in $d=HWC$44, $d=HWC$45 in $d=HWC$46, small $d=HWC$47 or $d=HWC$48 such as $d=HWC$49 to $d=HWC$50, and structured directions such as patch-wise perturbations or block-coordinate sampling. On MNIST adversarial attacks, the reported mean iterations to success are $d=HWC$51k $d=HWC$52k for ZO-AdaMM, $d=HWC$53k $d=HWC$54k for AdaZO, and $d=HWC$55k $d=HWC$56k for R-AdaZO, corresponding to an approximately $d=HWC$57 speedup (Shu et al., 3 Feb 2025). Although MNIST is not high-resolution, the paper explicitly frames the method as suitable for large-$d=HWC$58 image settings because the variance term scales with $d=HWC$59.

6. Constrained block ZO and the structure of query efficiency

Query-efficient constrained ZO on high-resolution images is represented by ZOB-GDA and ZOB-SGDA, which solve

$d=HWC$60

with compact dual set

$d=HWC$61

(Jin et al., 22 Oct 2025). The stationarity measure is built from the primal gradient and projected dual ascent:

$d=HWC$62

and an $d=HWC$63-stationary point satisfies

$d=HWC$64

The two algorithms differ in their smoothing mechanism. ZOB-GDA applies block coordinate descent-ascent directly to $d=HWC$65, while ZOB-SGDA introduces

$d=HWC$66

updates $d=HWC$67 using the block estimator of $d=HWC$68, updates $d=HWC$69 by projected ascent on $d=HWC$70, and evolves the auxiliary variable by

$d=HWC$71

The reported rates are

$d=HWC$72

for ZOB-GDA and

$d=HWC$73

for ZOB-SGDA (Jin et al., 22 Oct 2025). These imply overall query complexities $d=HWC$74 and $d=HWC$75 respectively.

A frequent misunderstanding in ZO-high is to treat small block size as eliminating dimension dependence. The explicit statement of the block theory is narrower: if $d=HWC$76, then a single step uses $d=HWC$77 queries, but the overall query complexity of ZOB-SGDA remains $d=HWC$78 because $d=HWC$79 still appears in the convergence rate (Jin et al., 22 Oct 2025). A related misunderstanding concerns “resolution independence.” In HZO with Spatial Parallel Perturbation, the number of Jacobian queries can become effectively independent of $d=HWC$80, but the cost of each forward pass still scales with $d=HWC$81 and $d=HWC$82 (Cao et al., 11 Feb 2026).

For image applications, the reported practical instantiations use contiguous patches or channel groups as blocks, for example $d=HWC$83 or $d=HWC$84, one-sided finite differences with $d=HWC$85 for $d=HWC$86, and box or $d=HWC$87 projections by clipping (Jin et al., 22 Oct 2025). The same source notes that if constraints such as $d=HWC$88 or total variation are analytic, then their block finite differences can be computed without additional black-box queries. The stated limitations are equally structural: the theory assumes smoothness and a compact dual set, random block selection is analyzed under uniform sampling rather than semantically structured patches, and variance increases when the constraints themselves are black-box and noisy (Jin et al., 22 Oct 2025).

Taken together, the current ZO-high literature separates into several complementary lines. HZO reduces depth-induced recomputation in trainable deep networks; ZO-BCD exploits sparsity and transform-domain structure in huge-scale black-box optimization; R-AdaZO improves variance handling in adaptive random-direction methods; and ZOB-SGDA formalizes the trade-off between blockwise single-step efficiency and optimal overall constrained query complexity (Cao et al., 11 Feb 2026, Cai et al., 2021, Shu et al., 3 Feb 2025, Jin et al., 22 Oct 2025). This suggests that ZO-high is best understood not as a single algorithmic family, but as a design space organized around three axes: estimator structure, geometric decomposition, and the source of scalability bottlenecks.