Projection-Free Frank–Wolfe Variants

• Projection-free Frank–Wolfe variants are a class of algorithms for constrained optimization that replace expensive projection steps with a linear minimization oracle, ensuring feasibility and inducing structured solutions.
• They offer rigorous convergence guarantees and complexity bounds, with applications spanning high-dimensional machine learning, semidefinite programming, and large-scale statistical modeling.
• Modern adaptations, including stochastic, zeroth-order, adaptive, and distributed methods, enhance performance in various regimes, addressing practical challenges like noisy gradients and communication constraints.

Projection-free Frank–Wolfe (FW) variants constitute a foundational algorithmic paradigm for constrained optimization where projection onto the feasible set is costly or structurally prohibitive, yet linear minimization (via a Linear Minimization Oracle, LMO) is tractable. The projection-free property enables application across domains such as high-dimensional machine learning, semidefinite programming, and large-scale statistical modeling. Modern research has produced a rich taxonomy of Frank–Wolfe variants, encompassing stochastic, zeroth-order, adaptive, distributed, and structure-exploiting methods, all with rigorous complexity and structural results.

1. Classical Projection-Free Frank–Wolfe and Theory

The classic Frank–Wolfe (FW) algorithm addresses

$\min_{x \in \mathcal{D}} f(x)$

where $f: \mathcal{D} \to \mathbb{R}$ is convex and $\mathcal{D}$ is a compact convex set. At each iteration, instead of a projection onto $\mathcal{D}$, one solves the linearized subproblem: $$s_k = \arg\min_{s \in \mathcal{D}} \langle \nabla f(x_k), s \rangle$$ and updates $x_{k+1} = x_k + \gamma_k (s_k - x_k)$ with suitable step-size $\gamma_k$ (e.g., $2/(k+2)$ or line search). This projection-free update preserves feasibility and often induces sparsity or low-rank structure, depending on $\mathcal{D}$ (Jaggi, 2011).

Theoretical guarantees include an $O(1/\epsilon)$ duality gap convergence and worst-case matching lower bounds on the required support size or matrix rank in $f: \mathcal{D} \to \mathbb{R}$0 and nuclear-norm constrained problems. The trade-off is between avoiding projections and potentially slow convergence, especially near polytope boundaries (Bomze et al., 2021).

2. Stochastic and Zeroth-Order Projection-Free Variants

Stochastic projection-free Frank–Wolfe methods extend the framework to empirical risk minimization and finite-sum or expectation settings: $f: \mathcal{D} \to \mathbb{R}$1 with $f: \mathcal{D} \to \mathbb{R}$2 typically difficult to project onto. Stochastic FW (SFW) and its variants use unbiased minibatch gradients, with the same LMO-based update as classical FW. For nonconvex settings, SFW converges in the Frank–Wolfe gap at rate $f: \mathcal{D} \to \mathbb{R}$3 with a suitable choice of batch size and diminishing step size (Pokutta et al., 2020). Momentum-augmented SFW (MSFW) accelerates practical performance by tracking a moving average of stochastic gradients.

Zeroth-order projection-free Frank–Wolfe methods estimate gradients via finite differences or random-directional derivatives, and move towards the minimizing atom of the LMO with respect to these estimates (Sahu et al., 2018). Under convexity and smoothness, convergence rates are $f: \mathcal{D} \to \mathbb{R}$4 for the primal gap, with matching dependence on the dimension—nullifying the need for projections or explicit gradients, and extending projection-free optimization to black-box and simulation-based settings (Akhtar et al., 2021).

3. Structural, Adaptive, and Distributed Variants

Projection-free Frank–Wolfe variants have been developed for specific structural and large-scale settings:

• Composite Constraints: For problems with a structured feasible region

$f: \mathcal{D} \to \mathbb{R}$5

with $f: \mathcal{D} \to \mathbb{R}$6 hard to project onto but $f: \mathcal{D} \to \mathbb{R}$7 easy, indicator penalization and Moreau envelopes enable the use of Frank–Wolfe updates while enforcing feasibility via smooth penalty approximations (Akhtar et al., 2021).

• Trimmed and Lazy LMOs: To reduce the LMO computational burden, algorithms skip LMO calls if the minimization direction changes below a threshold, while retaining theoretical convergence rates. Such “trimmed” variants provably maintain sublinear convergence with asymptotically fewer LMO calls (Akhtar et al., 2021, Bomze et al., 2021).
• Adaptive Gradients: AdaFW methods blend adaptive-metric information (AdaGrad-style diagonal preconditioning) into the FW subproblem, yielding more informed update directions and substantially improved convergence in practical, ill-conditioned settings (Combettes et al., 2020). Even in the nonconvex case, the gap decays as $f: \mathcal{D} \to \mathbb{R}$8.
• Distributed and Quantized: Quantized Frank–Wolfe (QFW) methods address network-constrained learning. Gradients are compressed using unbiased stochastic quantizers before aggregation, dramatically reducing per-round communication while maintaining $f: \mathcal{D} \to \mathbb{R}$9 (convex) or $\mathcal{D}$0 (nonconvex) rates. These methods integrate variance reduction and momentum for improved sample complexity and scaling (Zhang et al., 2019, Zhang, 2021).

4. Nonconvex and Structured Objective Extensions

Projection-free FW variants have been generalized to nonconvex and structured nonconvex objectives:

• Smooth Nonconvex Objectives: Stochastic FW ensures that the expected Frank–Wolfe gap converges as $\mathcal{D}$1 for smooth nonconvex functions (Pokutta et al., 2020).
• Difference-of-Convex (DC) Programming: The projection-free DC–Frank–Wolfe (Dc-Fw) method targets problems $\mathcal{D}$2 where $\mathcal{D}$3 is smooth convex and $\mathcal{D}$4 convex. By linearizing $\mathcal{D}$5 at the current iterate and optimizing the resulting convex surrogate with an inner FW loop, stationary points are attained in $\mathcal{D}$6 LMO calls, with further gradient-efficient conditional gradient sliding (CGS) reductions (Maskan et al., 11 Mar 2025).
• Low-Rank and Sparse Matrix Completion: In nuclear-norm constrained problems, classical FW may lead to high-rank iterates. The Rank-Drop Steps variant introduces certified rank-decreasing steps, bounding the rank of iterates without projections, and empirically achieving lower final ranks and runtimes than FW or away-step variants (Cheung et al., 2017).

5. Self-Concordant and Generalized Smoothness Variants

Classical FW convergence relies on global Lipschitz continuity, limiting its applicability to loss functions with unbounded curvature. Recent projection-free variants replace this assumption with local, self-concordant, or generalized self-concordant analytic models:

• Self-Concordant FW: Adaptive step-size rules leverage the self-concordant structure, using local norm and curvature to guarantee $\mathcal{D}$7 convergence, or linear rates under additional local strong convexity (via a local LMO) (Dvurechensky et al., 2020).
• Generalized Self-Concordant Analysis: For GSC losses, analytic step-size or backtracking allows projection-free convergence under much weaker smoothness than classical analyses, supporting non-Lipschitz or barrier-type losses ubiquitous in statistics and ML (Dvurechensky et al., 2020).

6. Advanced Online and Large-Scale Regimes

Projection-free Frank–Wolfe has also been adapted to online and large-scale regimes:

• Online Frank–Wolfe (OFW): Standard projection-free methods achieve $\mathcal{D}$8 regret. Near-optimal parameter tuning and potential-based proof techniques certify explicit constants and establish that additional LMO calls or different parameterizations do not improve the regret exponent for pure online FW schemes without further assumptions (Weibel et al., 6 Jun 2025).
• Follow-the-Perturbed-Leader Hybrid: Smoothing and FPL-based projection-free updates achieve the improved $\mathcal{D}$9 regret in smooth online convex optimization, matching the best known for pure LMO-based updates (Hazan et al., 2020).
• Gradient-Free DR-Submodular Maximization: Black-Box Continuous Greedy algorithms, projection-free and requiring only function evaluations, achieve tight information-theoretic approximation for monotone DR-submodular maximization without gradient information (Zhang, 2021).

7. Applications, Empirical Performance, and Domain-Specific Trade-Offs

Projection-free Frank–Wolfe variants have been successfully deployed in deep neural network training (with per-layer $\mathcal{D}$0-norm constraints), convex and nonconvex matrix optimization, large-scale SVMs, portfolio selection under polytope constraints, and CNN pruning. Momentum-augmented SFW and its domain adaptations outperform classical and even projection-based baselines in high-dimensional regimes, neural network test accuracy, and inference time (Pokutta et al., 2020, Shili et al., 30 Nov 2025).

A summary of structural and computational properties of key projection-free FW variants:

Variant/Class	Structure Preserved	Complexity	Per-Iterate Cost
Classical FW	Sparse/Low-Rank	$\mathcal{D}$1	1 LMO + 1 grad
SFW/MSFW	Sparse/Low-Rank	$\mathcal{D}$2	1 LMO + 1 minibatch grad
Zeroth-Order FW	Sparse/Low-Rank	$\mathcal{D}$3	1 LMO + O(d) function evals
AdaFW/AdaSVRF	Sparse/Low-Rank	$\mathcal{D}$4 primal gap	$\mathcal{D}$5 LMOs, adapt. metric
QFW/Distributed	Sparse/Low-Rank	$\mathcal{D}$6/$\mathcal{D}$7	1 LMO + compressed grad
Rank-Drop FW	Controlled matrix rank	$\mathcal{D}$8	1 LMO + 1 SVD update
Self-Concordant FW	Sparse/Low-Rank	$\mathcal{D}$9/linear (LLOO)	1 LMO + Hessian-vector
Online FW	N/A	$$s_k = \arg\min_{s \in \mathcal{D}} \langle \nabla f(x_k), s \rangle$$0 regret	1 LMO + 1 grad

Projection-free Frank–Wolfe methods offer a flexible, scalable, and structurally robust alternative to projection-based constrained optimization, supporting state-of-the-art statistical estimation, machine learning, and large-scale signal processing applications (Bomze et al., 2021, Pokutta et al., 2020, Cheung et al., 2017, Jha, 2021).