Simplex Frank-Wolfe Optimization

• Simplex Frank-Wolfe is a conditional gradient method specialized for convex optimization over the probability simplex, using a linear minimization oracle to select the vertex with the smallest gradient value.
• The algorithm achieves an O(1/t) convergence rate with sparse iterates by incorporating convex averaging and adding at most one new vertex to the support at each step.
• Enhanced variants, including away-steps, pairwise, and adaptive techniques, refine convergence behavior and can yield linear convergence under stronger geometric or smoothness conditions.

Simplex Frank-Wolfe is the specialization of the Frank-Wolfe, or conditional gradient, method to optimization over the probability simplex

$$\Delta^n=\{x\in\mathbb{R}^n:\;x_i\ge 0,\ \sum_{i=1}^n x_i=1\}.$$

In this setting, projection onto the feasible region is replaced by a linear minimization oracle over the simplex, so each iteration selects a vertex and moves by convex averaging. The specialization is unusually explicit: the oracle reduces to selecting the coordinate with smallest gradient value, the iterate remains a convex combination of simplex vertices, the Frank-Wolfe duality gap is available in closed form, and the support of the iterate grows by at most one atom per step. Classical analysis gives the familiar $O(1/t)$ primal decrease and $O(1/\varepsilon)$ complexity for an $\varepsilon$-small duality gap, while later work develops away-step, pairwise, in-face, boosted, stochastic, Bregman, self-concordant, and auto-conditioned variants tailored to the simplex and to polyhedral structure more broadly (Jaggi, 2011, Wirth et al., 2024, Giang-Tran et al., 15 May 2026).

1. Problem formulation on the simplex

The canonical problem is

$\min_{x\in\Delta^n} f(x),$

with $f$ convex. In the smooth case emphasized in the standard simplex exposition, one assumes access to a gradient oracle and a bounded curvature constant

$$C_f := \sup_{\substack{x,s\in \Delta^n,\ \alpha\in[0,1] \ y = x+\alpha(s-x)}} \frac{1}{\alpha^2}\bigl( f(y) - f(x) - \langle y-x, \nabla f(x)\rangle \bigr).$$

If $\nabla f$ is $L$-Lipschitz on $\Delta^n$, then

$O(1/t)$0

because $O(1/t)$1 in the Euclidean norm; for twice-differentiable $O(1/t)$2,

$O(1/t)$3

These facts make the simplex case a particularly transparent instance of the general curvature-based Frank-Wolfe theory (Jaggi, 2011).

The linear minimization oracle on $O(1/t)$4 has a closed form. Given $O(1/t)$5,

$O(1/t)$6

is always a vertex:

$O(1/t)$7

The update

$O(1/t)$8

therefore preserves feasibility without any projection step. In finite dimensions, this is also consistent with the Banach-space formulation of Frank-Wolfe, where the linear subproblem minimizes the linear functional induced by the Fréchet derivative over a compact convex set, and over a simplex the minimizer is again a vertex (Xu, 2017).

This vertex structure is the defining simplification of simplex Frank-Wolfe. The method operates over the convex hull of the standard basis, and every iterate lies in the convex hull of the initial point and the vertices chosen so far. That geometry underlies essentially all later simplex-specific results: explicit duality-gap formulas, sparse iterates, active-set reasoning, and polyhedral linear-rate refinements (Jaggi, 2011).

2. Core algorithm, certificates, and classical convergence

For the simplex, the Frank-Wolfe duality gap at $O(1/t)$9 is

$O(1/\varepsilon)$0

It is an efficiently computable certificate that upper-bounds the primal suboptimality $O(1/\varepsilon)$1. The associated “poor man’s dual” value is

$O(1/\varepsilon)$2

so that $O(1/\varepsilon)$3 and $O(1/\varepsilon)$4 is a valid lower bound on the optimum (Jaggi, 2011).

The central descent estimate is

$O(1/\varepsilon)$5

With the standard open-loop schedule

$O(1/\varepsilon)$6

one obtains the primal bound

$O(1/\varepsilon)$7

If the linear oracle is only approximate with additive accuracy $O(1/\varepsilon)$8, the bound becomes $O(1/\varepsilon)$9. Exact line search,

$\varepsilon$0

retains the same $\varepsilon$1 complexity and may improve numerical stability (Jaggi, 2011).

The duality-gap statement is slightly subtler than the primal estimate. Let

$\varepsilon$2

If one runs Frank-Wolfe for $\varepsilon$3 iterations with $\varepsilon$4 and then for another $\varepsilon$5 iterations with fixed $\varepsilon$6, there exists $\varepsilon$7 such that

$\varepsilon$8

With the approximate oracle, the same statement holds with $\varepsilon$9 (Jaggi, 2011).

A recurring misconception is that simplex Frank-Wolfe is automatically linearly convergent. The basic method is not: its classical guarantee is $\min_{x\in\Delta^n} f(x),$0 in primal error and $\min_{x\in\Delta^n} f(x),$1 in dual-gap complexity. Linear rates require stronger structure, different directions, or refined geometry, as later sections discuss (Jaggi, 2011, Wirth et al., 2024).

3. Sparsity, computational profile, and stopping behavior

Simplex Frank-Wolfe is also a sparse constructive method. If the algorithm starts from a vertex, then each step adds at most one vertex to the convex combination, so

$\min_{x\in\Delta^n} f(x),$2

Combining this with the $\min_{x\in\Delta^n} f(x),$3 iteration bound yields an $\min_{x\in\Delta^n} f(x),$4 upper bound on the support size of an $\min_{x\in\Delta^n} f(x),$5-approximate solution. The order is tight. For

$\min_{x\in\Delta^n} f(x),$6

on $\min_{x\in\Delta^n} f(x),$7, one has $\min_{x\in\Delta^n} f(x),$8, the best $\min_{x\in\Delta^n} f(x),$9-sparse objective value is $f$0, and the duality gap of any $f$1 with $f$2 satisfies

$f$3

Thus any $f$4-approximate solution must satisfy $f$5, and with scaling $f$6 one obtains the matching dependence $f$7. Away steps may improve constants in practice by trimming active weights, but the $f$8 order cannot be improved (Jaggi, 2011).

The computational profile is correspondingly simple. Each iteration requires a gradient evaluation, one pass over the gradient to compute

$f$9

and the sparse convex-combination update

$$C_f := \sup_{\substack{x,s\in \Delta^n,\ \alpha\in[0,1] \ y = x+\alpha(s-x)}} \frac{1}{\alpha^2}\bigl( f(y) - f(x) - \langle y-x, \nabla f(x)\rangle \bigr).$$0

The linear oracle is therefore $$C_f := \sup_{\substack{x,s\in \Delta^n,\ \alpha\in[0,1] \ y = x+\alpha(s-x)}} \frac{1}{\alpha^2}\bigl( f(y) - f(x) - \langle y-x, \nabla f(x)\rangle \bigr).$$1 on $$C_f := \sup_{\substack{x,s\in \Delta^n,\ \alpha\in[0,1] \ y = x+\alpha(s-x)}} \frac{1}{\alpha^2}\bigl( f(y) - f(x) - \langle y-x, \nabla f(x)\rangle \bigr).$$2, memory can be restricted to the active atoms and their coefficients, and sparse iterates admit compact representations. By contrast, Euclidean projection onto the simplex typically requires sorting or a water-filling step and costs $$C_f := \sup_{\substack{x,s\in \Delta^n,\ \alpha\in[0,1] \ y = x+\alpha(s-x)}} \frac{1}{\alpha^2}\bigl( f(y) - f(x) - \langle y-x, \nabla f(x)\rangle \bigr).$$3 per iteration, which explains the standard claim that the simplex case is one of the clearest regimes where projection-free optimization is algorithmically attractive (Jaggi, 2011).

The duality gap is also the natural stopping rule. On the simplex,

$$C_f := \sup_{\substack{x,s\in \Delta^n,\ \alpha\in[0,1] \ y = x+\alpha(s-x)}} \frac{1}{\alpha^2}\bigl( f(y) - f(x) - \langle y-x, \nabla f(x)\rangle \bigr).$$4

is inexpensive to compute and gives a certified upper bound on primal suboptimality. For very large $$C_f := \sup_{\substack{x,s\in \Delta^n,\ \alpha\in[0,1] \ y = x+\alpha(s-x)}} \frac{1}{\alpha^2}\bigl( f(y) - f(x) - \langle y-x, \nabla f(x)\rangle \bigr).$$5, randomized linear minimization can be used by sampling a subset of coordinates and picking the best sampled coordinate; the classical analysis states that if the linear oracle succeeds with probability $$C_f := \sup_{\substack{x,s\in \Delta^n,\ \alpha\in[0,1] \ y = x+\alpha(s-x)}} \frac{1}{\alpha^2}\bigl( f(y) - f(x) - \langle y-x, \nabla f(x)\rangle \bigr).$$6, then running roughly $$C_f := \sup_{\substack{x,s\in \Delta^n,\ \alpha\in[0,1] \ y = x+\alpha(s-x)}} \frac{1}{\alpha^2}\bigl( f(y) - f(x) - \langle y-x, \nabla f(x)\rangle \bigr).$$7 more iterations retains the convergence rate in expectation, in analogy with stochastic-gradient phenomena (Jaggi, 2011).

4. Accelerated and linearly convergent simplex variants

The most established route to linear convergence on simplices is to modify the direction choice. In concave maximization over $$C_f := \sup_{\substack{x,s\in \Delta^n,\ \alpha\in[0,1] \ y = x+\alpha(s-x)}} \frac{1}{\alpha^2}\bigl( f(y) - f(x) - \langle y-x, \nabla f(x)\rangle \bigr).$$8, which covers standard SVM dual training after sign reversal, the classical away-step method chooses an ascent vertex $$C_f := \sup_{\substack{x,s\in \Delta^n,\ \alpha\in[0,1] \ y = x+\alpha(s-x)}} \frac{1}{\alpha^2}\bigl( f(y) - f(x) - \langle y-x, \nabla f(x)\rangle \bigr).$$9 and an away vertex $\nabla f$0 from the active set. The “SWAP” variant uses the pairwise direction

$\nabla f$1

so mass is moved only between two coordinates. The method performs exact line search on both the standard Frank-Wolfe and SWAP directions and takes the step with larger objective improvement. Under the paper’s assumptions—twice continuous differentiability and Robinson’s strong sufficient condition—SWAP enjoys global convergence and an asymptotic linear rate, while the number of iterations to reach duality gap $\nabla f$2 is bounded by $\nabla f$3 and also admits a logarithmic bound for sufficiently small $\nabla f$4 (Allende et al., 2013).

A different linear-rate mechanism is strict complementarity. For smooth convex minimization over polytopes with quadratic growth, the away-step method with line search can converge linearly at a rate depending explicitly on the dimension of the optimal face rather than on the ambient dimension. On the simplex, if the optimal face has dimension $\nabla f$5, then the relevant geometric constants simplify to $\nabla f$6, $\nabla f$7, and $\nabla f$8, so the rate becomes of the form

$\nabla f$9

after a warm-up phase. The same analysis proves a robustness statement: if strict complementarity holds and the gradient perturbation is small enough, then the optimal face, hence the optimal support, is preserved (Garber, 2020).

Recent affine-invariant analyses make this dependence on simplex facial structure more explicit. For vanilla Frank-Wolfe, away-step Frank-Wolfe, blended pairwise Frank-Wolfe, and in-face Frank-Wolfe, the rates follow from an affine-invariant error bound together with an extended curvature constant. On $L$0, Euclidean $L$1-smoothness yields extended curvature $L$2, and when the relevant error bound has degree $L$3, the template gives linear convergence for all four variants. In standard form $L$4, the simplex has rank $L$5, so the in-face theorem specializes to a per-iteration linear contraction

$L$6

with $L$7 (Wirth et al., 2024).

Two more recent simplex-specific constructions push the same theme. “Boosting Frank-Wolfe by Chasing Gradients” replaces the single-atom direction by a normalized convex combination of atom differences chosen through a matching-pursuit-style inner loop. On the simplex, each inner round is still an $L$8 oracle call, feasibility is preserved because the returned $L$9 lies in $\Delta^n$0, and every accepted inner round improves the cosine alignment with $\Delta^n$1 by at least $\Delta^n$2. The method keeps the classical $\Delta^n$3 worst-case bound, but under PL structure and sufficiently many alignment-improving rounds it admits a practical

$\Delta^n$4

behavior (Combettes et al., 2020). The 2025 “Simplex Frank-Wolfe” line introduces a Simplex Linear Minimization Oracle, or SLMO, which is a local linear optimization oracle implemented with essentially the same complexity as the standard simplex LMO. Its SFW and refined rSFW variants achieve linear convergence for $\Delta^n$5-smooth and $\Delta^n$6-strongly convex objectives under exact line search, short step, and a constant step; for SFW,

$\Delta^n$7

and rSFW supports refinement steps that can incorporate away-step and pairwise strategies (Wang et al., 29 Sep 2025).

For convex quadratic objectives over the simplex, corrective schemes provide yet another route. The reduced problem on the active set is a small simplex-constrained quadratic program in the barycentric coefficients, and the paper on quadratic corrections uses this to define QC-MNP and QC-LP corrections, extending blended pairwise and fully corrective Frank-Wolfe. On the simplex these corrections interact naturally with pairwise directions $\Delta^n$8 and closed-form quadratic line search, while the enclosing Corrective Frank-Wolfe framework preserves the usual $\Delta^n$9 convex guarantee and linear convergence under sharpness (Halbey et al., 3 Jun 2025).

5. Beyond Euclidean smooth convexity

Simplex Frank-Wolfe is not limited to the standard Euclidean $O(1/t)$00-smooth setting. In Banach spaces, the Frank-Wolfe algorithm is formulated for convex differentiable objectives over closed bounded convex sets, with either line minimization or open-loop steps satisfying $O(1/t)$01 and $O(1/t)$02. The paper introduces a curvature constant of order $O(1/t)$03,

$O(1/t)$04

and proves $O(1/t)$05 convergence. If $O(1/t)$06 is $O(1/t)$07-Hölder continuous, this yields $O(1/t)$08; if $O(1/t)$09 is Lipschitz, it recovers $O(1/t)$10. The same framework extends to composite objectives through a generalized Frank-Wolfe subproblem (Xu, 2017).

Relative smoothness and Bregman geometry produce a different simplex specialization. With a convex reference function $O(1/t)$11 and Bregman divergence

$O(1/t)$12

the adaptive variant for relatively smooth objectives on $O(1/t)$13 chooses the step size through backtracking and the triangle scaling property. For the negative entropy

$O(1/t)$14

$O(1/t)$15 becomes the Kullback-Leibler divergence, and for a simplex vertex $O(1/t)$16 one has

$O(1/t)$17

when $O(1/t)$18 is strictly positive. The main sublinear bound is

$O(1/t)$19

which reduces to the classical $O(1/t)$20 Frank-Wolfe rate when $O(1/t)$21 (Vyguzov et al., 2024).

Generalized self-concordance yields another non-Lipschitz regime. For $O(1/t)$22 generalized self-concordant objectives, a monotonic Frank-Wolfe variant uses the open-loop step

$O(1/t)$23

together with an accept/reject test for domain feasibility and function decrease. After an explicit burn-in index $O(1/t)$24, the method achieves

$O(1/t)$25

and an analogous $O(1/t)$26 bound for the Frank-Wolfe gap. On the simplex, $O(1/t)$27, and the paper also gives linear rates for away-step and blended pairwise variants with backtracking (Carderera et al., 2021).

Stochastic and fully adaptive step-size rules have been developed specifically for projection-free regimes. The one-sample stochastic Frank-Wolfe method, 1-SFW, uses a single stochastic sample per iteration and a momentum estimator corrected by an unbiased gradient-difference term. On $O(1/t)$28 the oracle remains $O(1/t)$29 with $O(1/t)$30. In the convex case it achieves the optimal

$O(1/t)$31

sample complexity for $O(1/t)$32-suboptimality; in the nonconvex case it reaches an $O(1/t)$33-first-order stationary point, measured by the simplex Frank-Wolfe gap, in $O(1/t)$34 iterations (Zhang et al., 2019). The auto-conditioned framework replaces the global smoothness constant in closed-loop step sizes by the local estimator

$O(1/t)$35

updates

$O(1/t)$36

and then uses

$O(1/t)$37

On the simplex this applies uniformly to standard, away-step, and pairwise Frank-Wolfe; the paper proves convergence to stationary points in the nonconvex setting, recovers the standard convex sublinear guarantees without prior knowledge of a global $O(1/t)$38, and obtains accelerated rates for specific variants under additional structure (Giang-Tran et al., 15 May 2026).

6. Applications, analogies, and conceptual boundaries

The simplex setting appears directly in several application classes. The original projection-free analysis explicitly lists SVMs and polytope distance, boosting and mixture estimation, multiple kernel learning through convex combinations of base kernels, portfolio selection, and smallest enclosing ball problems as examples fitting $O(1/t)$39 (Jaggi, 2011). In large-scale SVM training, simplex Frank-Wolfe becomes especially concrete because the dual objective is quadratic, the oracle is a coordinate scan, and pairwise or away-style variants such as SWAP are closely related to Gilbert, MDM, and SMO-type procedures (Allende et al., 2013).

The simplex case also serves as the prototypical model for broader conditional-gradient phenomena. Over the $O(1/t)$40 ball, the oracle selects a signed coordinate and the same $O(1/t)$41 sparsity law holds. Over the nuclear norm ball, the analogue is rank: Hazan-style Frank-Wolfe returns $O(1/t)$42-solutions of rank $O(1/t)$43, and the lower bound is again $O(1/t)$44. The 2011 projection-free framework further extends to max-norm-constrained SDP and to diagonally dominant symmetric matrices, where sparsity or low rank plays the role that support size plays on the simplex (Jaggi, 2011).

A common conceptual confusion is to identify simplex Frank-Wolfe with the classical simplex algorithm for linear programming. They are different. The simplex method pivots among basic feasible solutions and maintains equality feasibility $O(1/t)$45 and nonnegativity at all times, whereas Frank-Wolfe on a simplex performs convex averaging of extreme points through linearization of a smooth objective. Related primal-dual Frank-Wolfe algorithms for linear programming therefore operate on compact saddle-point reformulations with simplex-like or capped nonnegative $O(1/t)$46 domains, rather than reproducing simplex pivots (Hough et al., 2024).

Taken together, these developments make simplex Frank-Wolfe more than a pedagogical special case. It is the cleanest instance in which the conditional-gradient philosophy becomes explicit: linear minimization over a polytope collapses to vertex selection, certificates are cheap, sparsity is intrinsic, and refinements of direction choice or geometry can be studied with unusual precision. The simplex remains the benchmark domain on which the distinctions between sublinear and linear convergence, between projection-free and projection-based first-order methods, and between Euclidean, Bregman, stochastic, and affine-invariant analyses are most transparent (Jaggi, 2011, Wirth et al., 2024).