---
title: Projection-Free Zeroth-Order Frank–Wolfe
url: https://www.emergentmind.com/topics/projection-free-zeroth-order-frank-wolfe
type: topic
---

# Projection-Free Zeroth-Order Frank–Wolfe

Projection-free zeroth-order Frank–Wolfe algorithms constitute a class of optimization procedures for constrained problems where only function evaluations (zeroth-order information) are available and projection onto the constraint set is computationally prohibitive. These methods blend zeroth-order (gradient-free) optimization with the conditional gradient (Frank–Wolfe) algorithm, leveraging linear minimization oracles (LMOs) over the feasible set in place of full projections. Recent advances unite variance reduction, acceleration, and momentum to approach the complexity bounds and empirical performance of first-order projection-free algorithms, fundamentally extending Frank–Wolfe methodology to black-box, highly structured, or combinatorial regimes.

## 1. Problem Formulation and Oracle Setting

The core setting involves minimizing a convex (or occasionally nonconvex) objective
$$
\min_{x \in \mathcal{C}} f(x) = \frac{1}{n}\sum_{i=1}^n f_i(x)
$$
where $\mathcal{C} \subset \mathbb{R}^d$ is a compact, convex set with diameter $D$. The paradigm assumes the following oracle access:
- **Function Query Oracle (FQO)**: Given $(i, x)$, outputs $f_i(x)$ (for finite-sum) or $f(x)$ (for general stochastic objectives).
- **Linear Oracle (LO)**: Given $g \in \mathbb{R}^d$, outputs $v^* = \arg\min_{v \in \mathcal{C}} \langle g, v \rangle$.
- **No direct gradient access** is assumed. Instead, finite-difference estimators are constructed using coordinate or randomly sampled directions.

The prototypical coordinate-wise two-point estimator with smoothing parameter $\mu > 0$ is given by
$$
\widehat{\nabla}_{\mathrm{coord}} f_i(x) = \sum_{j=1}^d \frac{f_i(x+\mu e_j) - f_i(x-\mu e_j)}{2\mu} e_j,
$$
with bias/variance controlled by $\mu$ and the function smoothness (e.g., $E\|\widehat{\nabla}_{\mathrm{coord}} f_i(x)-\nabla f_i(x)\|^2 \leq \mu^2 L^2 d$ in [2109.08858]).

## 2. Algorithmic Structure: Projection-Free Zeroth-Order Frank–Wolfe

Projection-free zeroth-order Frank–Wolfe algorithms inherit the conditional gradient update structure, substituting gradient information with zeroth-order surrogates. The general step at iteration $t$ is:

1. **Estimate gradient** $\widehat{g}_t$ using finite-differences.
2. **Linear oracle call**: $v_t = \arg\min_{v \in \mathcal{C}} \langle \widehat{g}_t, v\rangle$.
3. **Update**: $x_{t+1} = (1-\gamma_t)x_t + \gamma_t v_t$, with an appropriate stepsize $\gamma_t$.

Contemporary variants—such as ZO-ARCS [2109.08858], momentum-corrected trackers [2107.06534], and accelerated stochastic zeroth-order FW (Acc-SZOFW) [2007.12625]—employ outer–inner loop architectures, variance-reduced zeroth-order gradient approximations, and generalized momentum. A canonical example is the ZO-ARCS procedure, which maintains a pivot point $\tilde x$, computes a full zeroth-order gradient at $\tilde x$, and performs inner variance-reduced stochastic FW iterations with extrapolation and averaging sequences.

### Pseudocode Fragment: ZO-ARCS Outer–Inner Loop
```python
for epoch s = 1, ..., S:
    tilde_g = zeroth_order_gradient(full_dataset, tilde_x)
    x_0, bar_x_0, tilde_x = previous_epoch_output
    for t = 1,...,T:
        i_t = random_sample(n)
        underline_x_t = extrapolation(bar_x_{t−1}, x_{t−1}, tilde_x)
        G_t = zeroth_order_gradient(f_{i_t}, underline_x_t) - zeroth_order_gradient(f_{i_t}, tilde_x) + tilde_g
        x_t = CondG(G_t, x_{t−1}, underline_x_t)
        bar_x_t = momentum_update(bar_x_{t−1}, x_t, tilde_x)
    output = weighted_average(bar_x_{1:T})
```
The CondG subroutine approximately solves the quadratic-regularized FW subproblem using only LO calls [2109.08858].

## 3. Variance Reduction and Acceleration Mechanisms

Variance reduction in zeroth-order Frank–Wolfe is principally achieved by referencing full or large-batch gradient surrogates at anchor points, then combining them with fresh function evaluation differences:

$$
G_t = \widehat{\nabla} f_{i_t}(\underline{x}_t) - \widehat{\nabla} f_{i_t}(\tilde{x}) + \tilde{g}
$$

This mechanism ensures $\mathbb{E}[G_t] = \nabla f(\underline{x}_t)$ while the variance $\mathrm{Var}(G_t)$ diminishes as the number of inner iterations increases. Acceleration typically employs multiple sequences (e.g., $(\underline{x}_t, x_t, \bar{x}_t)$ or $(x_t, y_t, z_t)$ in [2109.08858], [2007.12625]), mirroring Nesterov's or Lan's accelerated frameworks, but adapted for the conditional gradient context and fully decoupled from direct gradient computation.

Momentum-based updates are constructed so that averaging and extrapolation facilitate faster reduction of the optimality gap. In the ARCS framework:
$$
\underline{x}_t = (1-\alpha_s-p_s)\bar{x}_{t-1} + \alpha_s x_{t-1} + p_s \tilde{x},
$$
with analogous update expressions for $\bar{x}_t$ [2109.08858].

## 4. Convergence Rates and Oracle Complexity

Projection-free zeroth-order Frank–Wolfe algorithms now achieve rates matching or improving upon classic first-order projection-free methods and surpassing early zeroth-order schemes in both sample and iteration complexity.

- **Convex case (finite-sum):** For ZO-ARCS, to reach $\mathbb{E}[f(\tilde x)-f(x^*)] \leq \epsilon$,
  $$
  N_\text{FQO} =
  \begin{cases}
    O(nd \ln(D_0/\epsilon)), & \text{if } \epsilon \geq O(D_0/n) \\
    O(nd \ln n + d\sqrt{n D_0 / \epsilon}), & \text{otherwise}
  \end{cases}
  $$
  with $N_{\rm LO}$ calls scaling as $O(\epsilon^{-2})$ or $O(n^2 + n/\epsilon)$ [2109.08858].
- **Composite constraints and affine feasibilities:** ZO-FW achieves $O(\epsilon^{-2} m)$ FQO calls and $O(\delta^{-2})$ LMO calls for $\epsilon$-optimality and $\delta$-feasibility [2107.06534].
- **Nonconvex setting:** Algorithms such as Acc-SZOFW attain $O(d\sqrt{n}\epsilon^{-2})$ finite-sum and $O(d\epsilon^{-3})$ stochastic FQO complexities for $\epsilon$-stationarity [2007.12625].
- **Early single-direction ZO-FW:** Achieved $O(d^{1/3} / T^{1/3})$ primal gap decay in convex settings and $O(d^{1/3} / T^{1/4})$ Frank–Wolfe gap decay in nonconvex, reflecting tight known bounds for single-direction estimation [1810.03233].

These results demonstrate that, up to dimension-dependent factors, modern projection-free zeroth-order methods reach the iteration–complexity and rate benchmarks of their first-order counterparts, and in many regimes achieve substantial query savings over projection-based black-box methods.

## 5. Projection-Free Property and Linear Oracles

At no point do these algorithms compute explicit projections onto $\mathcal{C}$. Instead, updates rely entirely on LMOs, which are typically much cheaper (in both theory and numerics) for structured constraint sets such as simplices, nuclear-norm balls, or polytopes. For example, on the simplex, an LO reduces to a coordinate selection, and for the spectrahedron, an LO becomes an extremal eigenvector computation.

Conditional gradient subproblems within inner loops are solved via iterative LO calls that terminate once an optimality gap falls below a schedule-dependent tolerance. This design ensures that the full iteration remains within the feasible set and that each step exploits the low computational complexity of the LMO for appropriately structured $\mathcal{C}$ [2109.08858], [2107.06534].

## 6. Empirical Validation and Applications

Experimental results across multiple domains validate the practical impact of projection-free zeroth-order Frank–Wolfe:

- **Low-rank matrix completion:** Using nuclear-norm ball constraints, ZO-ARCS outperforms prior zeroth-order and first-order projection-free methods in sample efficiency and suboptimality decay on image completion and LIBSVM tasks [2109.08858].
- **SDP-type applications:** ZO-FW achieves the theoretically predicted rates and closely tracks first-order counterparts on sparse covariance estimation, k-means clustering relaxations, and sparsest-cut problems. The trimmed-FW variant empirically skips 20%–40% of LMO calls with negligible loss [2107.06534].
- **Robust black-box classification and adversarial attacks:** Accelerated methods achieve 10–100× query speedups compared to previous deterministic zeroth-order and stochastic conditional gradient methods [2007.12625].
- **Complex combinatorial equilibrium optimization:** ZO-Stackelberg incorporates projection-free Frank–Wolfe for fast inner equilibrium computation under zeroth-order leader updates, achieving orders-of-magnitude wall-clock and memory improvements over differentiation-based approaches [2602.23277].
- On high-dimensional problems, query complexity aligns with predicted $d$- and $n$-scaled rates, with empirical gap to fully gradient-based methods closing as variance-reduction and acceleration mechanisms are deployed.

## 7. Extensions, Limitations, and Future Directions

Projection-free zeroth-order Frank–Wolfe is now a mature high-performance framework for black-box and large-scale constrained optimization:

- **Variance-reduced and accelerated extensions** generalize to stochastic and finite-sum settings, nonconvex and composite domains, and admit higher-order correction (SPIDER, SARAH) and adaptive stepsizes [2109.08858], [2007.12625].
- **Generalization to combinatorial polytopes** and nonsmooth objectives, including stratified sampling to maintain oracle efficiency in massive discrete domains [2602.23277].
- **Open fronts include** random directional or Gaussian-smoothing estimators for settings where coordinate queries are expensive, incorporation of saddle-point and nonconvex regimes via negative curvature detection, and scalability to distributed or federated settings.
- **Dimension dependence** remains a critical bottleneck in very high-dimensional spaces unless domain-specific structure (e.g., sparsity, low rank) can be algorithmically exploited [1810.03233].
- **Practical limitations** involve computational overhead when the LO is expensive, the tuning of variance-reduction schedules, and possible degradation in non-smooth or poorly conditioned regimes.

The projection-free zeroth-order Frank–Wolfe framework, especially as crystallized in the ARCS algorithm, now constitutes a leading methodology for constraint-rich, black-box convex and nonconvex optimization, efficiently bridging the gap between first-order projection-free solvers and purely function-based oracles [2109.08858], [2107.06534], [2007.12625], [1810.03233], [2602.23277].

Source: https://www.emergentmind.com/topics/projection-free-zeroth-order-frank-wolfe