Papers
Topics
Authors
Recent
Search
2000 character limit reached

Simplex Frank-Wolfe Optimization

Updated 14 July 2026
  • Simplex Frank-Wolfe is a conditional gradient method specialized for convex optimization over the probability simplex, using a linear minimization oracle to select the vertex with the smallest gradient value.
  • The algorithm achieves an O(1/t) convergence rate with sparse iterates by incorporating convex averaging and adding at most one new vertex to the support at each step.
  • Enhanced variants, including away-steps, pairwise, and adaptive techniques, refine convergence behavior and can yield linear convergence under stronger geometric or smoothness conditions.

Simplex Frank-Wolfe is the specialization of the Frank-Wolfe, or conditional gradient, method to optimization over the probability simplex

Δn={x∈Rn:  xi≥0, ∑i=1nxi=1}.\Delta^n=\{x\in\mathbb{R}^n:\;x_i\ge 0,\ \sum_{i=1}^n x_i=1\}.

In this setting, projection onto the feasible region is replaced by a linear minimization oracle over the simplex, so each iteration selects a vertex and moves by convex averaging. The specialization is unusually explicit: the oracle reduces to selecting the coordinate with smallest gradient value, the iterate remains a convex combination of simplex vertices, the Frank-Wolfe duality gap is available in closed form, and the support of the iterate grows by at most one atom per step. Classical analysis gives the familiar O(1/t)O(1/t) primal decrease and O(1/ε)O(1/\varepsilon) complexity for an ε\varepsilon-small duality gap, while later work develops away-step, pairwise, in-face, boosted, stochastic, Bregman, self-concordant, and auto-conditioned variants tailored to the simplex and to polyhedral structure more broadly (Jaggi, 2011, Wirth et al., 2024, Giang-Tran et al., 15 May 2026).

1. Problem formulation on the simplex

The canonical problem is

min⁡x∈Δnf(x),\min_{x\in\Delta^n} f(x),

with ff convex. In the smooth case emphasized in the standard simplex exposition, one assumes access to a gradient oracle and a bounded curvature constant

Cf:=sup⁡x,s∈Δn, α∈[0,1]y=x+α(s−x)1α2(f(y)−f(x)−⟨y−x,∇f(x)⟩).C_f := \sup_{\substack{x,s\in \Delta^n,\ \alpha\in[0,1] \\ y = x+\alpha(s-x)}} \frac{1}{\alpha^2}\bigl( f(y) - f(x) - \langle y-x, \nabla f(x)\rangle \bigr).

If ∇f\nabla f is LL-Lipschitz on Δn\Delta^n, then

Cf≤12 diam(Δn)2 L=L,C_f \le \tfrac{1}{2}\,\mathrm{diam}(\Delta^n)^2\,L = L,

because diam(Δn)=2\mathrm{diam}(\Delta^n)=\sqrt{2} in the Euclidean norm; for twice-differentiable ff,

Cf≤sup⁡z∈Δnλmax⁡(∇2f(z)).C_f \le \sup_{z\in \Delta^n} \lambda_{\max}\bigl(\nabla^2 f(z)\bigr).

These facts make the simplex case a particularly transparent instance of the general curvature-based Frank-Wolfe theory (Jaggi, 2011).

The linear minimization oracle on Δn\Delta^n has a closed form. Given xt∈Δnx_t\in\Delta^n,

st∈arg⁡min⁡s∈Δn⟨∇f(xt),s⟩s_t \in \arg\min_{s\in\Delta^n}\langle \nabla f(x_t),s\rangle

is always a vertex:

st=ejt,jt∈arg⁡min⁡i∈[n][∇f(xt)]i.s_t=e_{j_t},\qquad j_t\in\arg\min_{i\in[n]}[\nabla f(x_t)]_i.

The update

xt+1=(1−γt)xt+γtstx_{t+1}=(1-\gamma_t)x_t+\gamma_t s_t

therefore preserves feasibility without any projection step. In finite dimensions, this is also consistent with the Banach-space formulation of Frank-Wolfe, where the linear subproblem minimizes the linear functional induced by the Fréchet derivative over a compact convex set, and over a simplex the minimizer is again a vertex (Xu, 2017).

This vertex structure is the defining simplification of simplex Frank-Wolfe. The method operates over the convex hull of the standard basis, and every iterate lies in the convex hull of the initial point and the vertices chosen so far. That geometry underlies essentially all later simplex-specific results: explicit duality-gap formulas, sparse iterates, active-set reasoning, and polyhedral linear-rate refinements (Jaggi, 2011).

2. Core algorithm, certificates, and classical convergence

For the simplex, the Frank-Wolfe duality gap at xtx_t is

g(xt)=max⁡s∈Δn⟨∇f(xt),xt−s⟩=⟨∇f(xt),xt⟩−min⁡i[∇f(xt)]i.g(x_t)=\max_{s\in\Delta^n}\langle \nabla f(x_t),x_t-s\rangle =\langle \nabla f(x_t),x_t\rangle-\min_i[\nabla f(x_t)]_i.

It is an efficiently computable certificate that upper-bounds the primal suboptimality f(xt)−f(x∗)f(x_t)-f(x^*). The associated “poor man’s dual” value is

ω(xt)=f(xt)+min⁡s∈Δn⟨s−xt,∇f(xt)⟩,\omega(x_t)=f(x_t)+\min_{s\in\Delta^n}\langle s-x_t,\nabla f(x_t)\rangle,

so that g(xt)=f(xt)−ω(xt)g(x_t)=f(x_t)-\omega(x_t) and ω(xt)\omega(x_t) is a valid lower bound on the optimum (Jaggi, 2011).

The central descent estimate is

f(xt+1)≤f(xt)−γ g(xt)+γ2Cf,γ∈[0,1].f(x_{t+1})\le f(x_t)-\gamma\,g(x_t)+\gamma^2 C_f,\qquad \gamma\in[0,1].

With the standard open-loop schedule

γt=2t+2,\gamma_t=\frac{2}{t+2},

one obtains the primal bound

f(xt)−f(x∗)≤4Cft+2,t≥1.f(x_t)-f(x^*)\le \frac{4C_f}{t+2},\qquad t\ge 1.

If the linear oracle is only approximate with additive accuracy εt′=γtCf\varepsilon'_t=\gamma_t C_f, the bound becomes 8Cft+2\frac{8C_f}{t+2}. Exact line search,

γt∈arg⁡min⁡γ∈[0,1]f(xt+γ(st−xt)),\gamma_t\in\arg\min_{\gamma\in[0,1]} f\bigl(x_t+\gamma(s_t-x_t)\bigr),

retains the same O(1/ε)O(1/\varepsilon) complexity and may improve numerical stability (Jaggi, 2011).

The duality-gap statement is slightly subtler than the primal estimate. Let

K=⌈4Cfε⌉.K=\left\lceil \frac{4C_f}{\varepsilon}\right\rceil.

If one runs Frank-Wolfe for KK iterations with γt=2t+2\gamma_t=\frac{2}{t+2} and then for another K+1K+1 iterations with fixed γ=2K+2\gamma=\frac{2}{K+2}, there exists t^∈{K,…,2K+1}\hat t\in\{K,\dots,2K+1\} such that

g(xt^)≤ε.g(x_{\hat t})\le \varepsilon.

With the approximate oracle, the same statement holds with K=⌈8Cf/ε⌉K=\lceil 8C_f/\varepsilon\rceil (Jaggi, 2011).

A recurring misconception is that simplex Frank-Wolfe is automatically linearly convergent. The basic method is not: its classical guarantee is O(Cf/t)O(C_f/t) in primal error and O(Cf/ε)O(C_f/\varepsilon) in dual-gap complexity. Linear rates require stronger structure, different directions, or refined geometry, as later sections discuss (Jaggi, 2011, Wirth et al., 2024).

3. Sparsity, computational profile, and stopping behavior

Simplex Frank-Wolfe is also a sparse constructive method. If the algorithm starts from a vertex, then each step adds at most one vertex to the convex combination, so

supp(xt)≤t+1.\mathrm{supp}(x_t)\le t+1.

Combining this with the O(1/ε)O(1/\varepsilon) iteration bound yields an O(1/ε)O(1/\varepsilon) upper bound on the support size of an ε\varepsilon-approximate solution. The order is tight. For

f(x)=∥x∥22f(x)=\|x\|_2^2

on Δn\Delta^n, one has Cf=2C_f=2, the best kk-sparse objective value is $1/k$, and the duality gap of any x∈Δnx\in\Delta^n with card(x)≤k<n\mathrm{card}(x)\le k<n satisfies

g(x)=2∥x∥22≥2k.g(x)=2\|x\|_2^2\ge \frac{2}{k}.

Thus any ε\varepsilon-approximate solution must satisfy card(x)≥Ω(1/ε)\mathrm{card}(x)\ge \Omega(1/\varepsilon), and with scaling f(x)=γ∥x∥22f(x)=\gamma\|x\|_2^2 one obtains the matching dependence Ω(Cf/ε)\Omega(C_f/\varepsilon). Away steps may improve constants in practice by trimming active weights, but the Θ(1/ε)\Theta(1/\varepsilon) order cannot be improved (Jaggi, 2011).

The computational profile is correspondingly simple. Each iteration requires a gradient evaluation, one pass over the gradient to compute

jt=arg⁡min⁡i[∇f(xt)]i,j_t=\arg\min_i [\nabla f(x_t)]_i,

and the sparse convex-combination update

xt+1=(1−γt)xt+γtejt.x_{t+1}=(1-\gamma_t)x_t+\gamma_t e_{j_t}.

The linear oracle is therefore O(n)O(n) on Δn\Delta^n, memory can be restricted to the active atoms and their coefficients, and sparse iterates admit compact representations. By contrast, Euclidean projection onto the simplex typically requires sorting or a water-filling step and costs O(nlog⁡n)O(n\log n) per iteration, which explains the standard claim that the simplex case is one of the clearest regimes where projection-free optimization is algorithmically attractive (Jaggi, 2011).

The duality gap is also the natural stopping rule. On the simplex,

g(xt)=⟨∇f(xt),xt⟩−min⁡i[∇f(xt)]ig(x_t)=\langle \nabla f(x_t),x_t\rangle-\min_i[\nabla f(x_t)]_i

is inexpensive to compute and gives a certified upper bound on primal suboptimality. For very large nn, randomized linear minimization can be used by sampling a subset of coordinates and picking the best sampled coordinate; the classical analysis states that if the linear oracle succeeds with probability pp, then running roughly $1/p$ more iterations retains the convergence rate in expectation, in analogy with stochastic-gradient phenomena (Jaggi, 2011).

4. Accelerated and linearly convergent simplex variants

The most established route to linear convergence on simplices is to modify the direction choice. In concave maximization over Δn\Delta^n, which covers standard SVM dual training after sign reversal, the classical away-step method chooses an ascent vertex ei∗e_{i^*} and an away vertex ej∗e_{j^*} from the active set. The “SWAP” variant uses the pairwise direction

dtSWAP=ei∗−ej∗,d_t^{SWAP}=e_{i^*}-e_{j^*},

so mass is moved only between two coordinates. The method performs exact line search on both the standard Frank-Wolfe and SWAP directions and takes the step with larger objective improvement. Under the paper’s assumptions—twice continuous differentiability and Robinson’s strong sufficient condition—SWAP enjoys global convergence and an asymptotic linear rate, while the number of iterations to reach duality gap ε\varepsilon is bounded by K≤Q+M′/εK\le Q+M'/\varepsilon and also admits a logarithmic bound for sufficiently small ε\varepsilon (Allende et al., 2013).

A different linear-rate mechanism is strict complementarity. For smooth convex minimization over polytopes with quadratic growth, the away-step method with line search can converge linearly at a rate depending explicitly on the dimension of the optimal face rather than on the ambient dimension. On the simplex, if the optimal face has dimension k=∣S∗∣−1k=|S^*|-1, then the relevant geometric constants simplify to ψ∗=1\psi^*=1, ξ∗=1\xi^*=1, and D=2D=\sqrt2, so the rate becomes of the form

f(xt)−f∗≤Cexp⁡(−ρ(t−t0)),ρ=Θ ⁣(min⁡{1,α/(βk)}),f(x_t)-f^*\le C\exp(-\rho(t-t_0)), \qquad \rho=\Theta\!\left(\min\{1,\alpha/(\beta k)\}\right),

after a warm-up phase. The same analysis proves a robustness statement: if strict complementarity holds and the gradient perturbation is small enough, then the optimal face, hence the optimal support, is preserved (Garber, 2020).

Recent affine-invariant analyses make this dependence on simplex facial structure more explicit. For vanilla Frank-Wolfe, away-step Frank-Wolfe, blended pairwise Frank-Wolfe, and in-face Frank-Wolfe, the rates follow from an affine-invariant error bound together with an extended curvature constant. On Δn\Delta^n, Euclidean LsL_s-smoothness yields extended curvature L=2LsL=2L_s, and when the relevant error bound has degree θ=12\theta=\tfrac12, the template gives linear convergence for all four variants. In standard form Ax=b, x≥0Ax=b,\ x\ge0, the simplex has rank m=1m=1, so the in-face theorem specializes to a per-iteration linear contraction

f(xt)−f∗≤(f(x0)−f∗)⋅(1−ρ)tf(x_t)-f^*\le (f(x_0)-f^*)\cdot(1-\rho)^t

with ρ=min⁡{μf/(8L),1/2}\rho=\min\{\mu_f/(8L),1/2\} (Wirth et al., 2024).

Two more recent simplex-specific constructions push the same theme. “Boosting Frank-Wolfe by Chasing Gradients” replaces the single-atom direction by a normalized convex combination of atom differences chosen through a matching-pursuit-style inner loop. On the simplex, each inner round is still an O(n)O(n) oracle call, feasibility is preserved because the returned gtg_t lies in conv⁡{ei−xt}\operatorname{conv}\{e_i-x_t\}, and every accepted inner round improves the cosine alignment with −∇f(xt)-\nabla f(x_t) by at least δ\delta. The method keeps the classical O(1/t)O(1/t) worst-case bound, but under PL structure and sufficiently many alignment-improving rounds it admits a practical

O(e−ωt)O(e^{-\omega t})

behavior (Combettes et al., 2020). The 2025 “Simplex Frank-Wolfe” line introduces a Simplex Linear Minimization Oracle, or SLMO, which is a local linear optimization oracle implemented with essentially the same complexity as the standard simplex LMO. Its SFW and refined rSFW variants achieve linear convergence for LL-smooth and μ\mu-strongly convex objectives under exact line search, short step, and a constant step; for SFW,

f(xk)−f(x∗)≤μd022exp⁡ ⁣(−μ4Ln2k),f(x_k)-f(x^*)\le \frac{\mu d_0^2}{2}\exp\!\left(-\frac{\mu}{4Ln^2}k\right),

and rSFW supports refinement steps that can incorporate away-step and pairwise strategies (Wang et al., 29 Sep 2025).

For convex quadratic objectives over the simplex, corrective schemes provide yet another route. The reduced problem on the active set is a small simplex-constrained quadratic program in the barycentric coefficients, and the paper on quadratic corrections uses this to define QC-MNP and QC-LP corrections, extending blended pairwise and fully corrective Frank-Wolfe. On the simplex these corrections interact naturally with pairwise directions ei∗−ej∗e_{i^*}-e_{j^*} and closed-form quadratic line search, while the enclosing Corrective Frank-Wolfe framework preserves the usual O(1/T)O(1/T) convex guarantee and linear convergence under sharpness (Halbey et al., 3 Jun 2025).

5. Beyond Euclidean smooth convexity

Simplex Frank-Wolfe is not limited to the standard Euclidean LL-smooth setting. In Banach spaces, the Frank-Wolfe algorithm is formulated for convex differentiable objectives over closed bounded convex sets, with either line minimization or open-loop steps satisfying γk→0\gamma_k\to0 and ∑kγk=∞\sum_k\gamma_k=\infty. The paper introduces a curvature constant of order σ∈(1,2]\sigma\in(1,2],

C(σ)=sup⁡x,s∈C, γ∈(0,1]y=x+γ(s−x)f(y)−f(x)−(y−x,f′(x))γσ,C(\sigma)=\sup_{\substack{x,s\in C,\ \gamma\in(0,1] \\ y=x+\gamma(s-x)}} \frac{f(y)-f(x)-(y-x,f'(x))}{\gamma^\sigma},

and proves O(1/kσ−1)O(1/k^{\sigma-1}) convergence. If f′f' is ν\nu-Hölder continuous, this yields O(1/kν)O(1/k^\nu); if f′f' is Lipschitz, it recovers O(1/k)O(1/k). The same framework extends to composite objectives through a generalized Frank-Wolfe subproblem (Xu, 2017).

Relative smoothness and Bregman geometry produce a different simplex specialization. With a convex reference function hh and Bregman divergence

Dh(x,y)=h(x)−h(y)−⟨∇h(y),x−y⟩,D_h(x,y)=h(x)-h(y)-\langle \nabla h(y),x-y\rangle,

the adaptive variant for relatively smooth objectives on Δn\Delta_n chooses the step size through backtracking and the triangle scaling property. For the negative entropy

h(x)=∑ixilog⁡xi,h(x)=\sum_i x_i\log x_i,

DhD_h becomes the Kullback-Leibler divergence, and for a simplex vertex sk=eiks_k=e_{i_k} one has

Dh(sk,xk)=−log⁡xk,ikD_h(s_k,x_k)=-\log x_{k,i_k}

when xkx_k is strictly positive. The main sublinear bound is

f(xk)−f∗≤(2k+2)γ−1max⁡0≤j≤k−1LjR2,f(x_k)-f^*\le \left(\frac{2}{k+2}\right)^{\gamma-1}\max_{0\le j\le k-1} L_jR^2,

which reduces to the classical O(1/k)O(1/k) Frank-Wolfe rate when γ=2\gamma=2 (Vyguzov et al., 2024).

Generalized self-concordance yields another non-Lipschitz regime. For (M,ν)(M,\nu) generalized self-concordant objectives, a monotonic Frank-Wolfe variant uses the open-loop step

γt=2t+2\gamma_t=\frac{2}{t+2}

together with an accept/reject test for domain feasibility and function decrease. After an explicit burn-in index TνT_\nu, the method achieves

h(xt)≤4(Tν+1)t+1max⁡{h(x0),LfL0D2ων(1/2)}h(x_t)\le \frac{4(T_\nu+1)}{t+1}\max\{h(x_0),L_f^{L_0}D^2\omega_\nu(1/2)\}

and an analogous O(1/t)O(1/t) bound for the Frank-Wolfe gap. On the simplex, D=2D=\sqrt2, and the paper also gives linear rates for away-step and blended pairwise variants with backtracking (Carderera et al., 2021).

Stochastic and fully adaptive step-size rules have been developed specifically for projection-free regimes. The one-sample stochastic Frank-Wolfe method, 1-SFW, uses a single stochastic sample per iteration and a momentum estimator corrected by an unbiased gradient-difference term. On Δd\Delta^d the oracle remains vt=eitv_t=e_{i_t} with it=arg⁡min⁡i[dt]ii_t=\arg\min_i[d_t]_i. In the convex case it achieves the optimal

O(1/ε2)\mathcal{O}(1/\varepsilon^2)

sample complexity for ε\varepsilon-suboptimality; in the nonconvex case it reaches an ε\varepsilon-first-order stationary point, measured by the simplex Frank-Wolfe gap, in O(1/ε3)\mathcal{O}(1/\varepsilon^3) iterations (Zhang et al., 2019). The auto-conditioned framework replaces the global smoothness constant in closed-loop step sizes by the local estimator

l(x,y)=2∣f(y)−f(x)−∇f(x)⊤(y−x)∣∥y−x∥2,l(x,y)=\frac{2|f(y)-f(x)-\nabla f(x)^\top(y-x)|}{\|y-x\|^2},

updates

Lt+1=max⁡{l(xt,xˉt+1),rtLt},L_{t+1}=\max\{l(x_t,\bar x_{t+1}),r_tL_t\},

and then uses

γt=min⁡{⟨∇f(xt),dt⟩Lt∥dt∥2,γmax⁡}.\gamma_t=\min\left\{\frac{\langle \nabla f(x_t),d_t\rangle}{L_t\|d_t\|^2},\gamma_{\max}\right\}.

On the simplex this applies uniformly to standard, away-step, and pairwise Frank-Wolfe; the paper proves convergence to stationary points in the nonconvex setting, recovers the standard convex sublinear guarantees without prior knowledge of a global LL, and obtains accelerated rates for specific variants under additional structure (Giang-Tran et al., 15 May 2026).

6. Applications, analogies, and conceptual boundaries

The simplex setting appears directly in several application classes. The original projection-free analysis explicitly lists SVMs and polytope distance, boosting and mixture estimation, multiple kernel learning through convex combinations of base kernels, portfolio selection, and smallest enclosing ball problems as examples fitting Δn\Delta^n (Jaggi, 2011). In large-scale SVM training, simplex Frank-Wolfe becomes especially concrete because the dual objective is quadratic, the oracle is a coordinate scan, and pairwise or away-style variants such as SWAP are closely related to Gilbert, MDM, and SMO-type procedures (Allende et al., 2013).

The simplex case also serves as the prototypical model for broader conditional-gradient phenomena. Over the ℓ1\ell_1 ball, the oracle selects a signed coordinate and the same Θ(1/ε)\Theta(1/\varepsilon) sparsity law holds. Over the nuclear norm ball, the analogue is rank: Hazan-style Frank-Wolfe returns ε\varepsilon-solutions of rank O(1/ε)O(1/\varepsilon), and the lower bound is again Ω(1/ε)\Omega(1/\varepsilon). The 2011 projection-free framework further extends to max-norm-constrained SDP and to diagonally dominant symmetric matrices, where sparsity or low rank plays the role that support size plays on the simplex (Jaggi, 2011).

A common conceptual confusion is to identify simplex Frank-Wolfe with the classical simplex algorithm for linear programming. They are different. The simplex method pivots among basic feasible solutions and maintains equality feasibility Ax=bAx=b and nonnegativity at all times, whereas Frank-Wolfe on a simplex performs convex averaging of extreme points through linearization of a smooth objective. Related primal-dual Frank-Wolfe algorithms for linear programming therefore operate on compact saddle-point reformulations with simplex-like or capped nonnegative ℓ1\ell_1 domains, rather than reproducing simplex pivots (Hough et al., 2024).

Taken together, these developments make simplex Frank-Wolfe more than a pedagogical special case. It is the cleanest instance in which the conditional-gradient philosophy becomes explicit: linear minimization over a polytope collapses to vertex selection, certificates are cheap, sparsity is intrinsic, and refinements of direction choice or geometry can be studied with unusual precision. The simplex remains the benchmark domain on which the distinctions between sublinear and linear convergence, between projection-free and projection-based first-order methods, and between Euclidean, Bregman, stochastic, and affine-invariant analyses are most transparent (Jaggi, 2011, Wirth et al., 2024).

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to Simplex Frank-Wolfe.