Near-Optimal Acceleration for Smooth $\ell_p$ / $\ell_q$ Nondual Convex First-Order Oracle Optimization

Abstract: We study the optimization of convex objectives with $(L,κ-1)$-Hölder-continuous gradients in $\ell_q$ over $R B_p<sup>d$, $1&lt;κ\le 2$. (MG26) provides selectors with a movement bound for the problem of chasing high-dimensional convex nested sets for every $p&lt;q$ and generally reduces Lipschitz convex optimization to bounds on the movement of selectors. We couple that movement with Hölder descent yielding a polynomial-runtime first-order method whose feasible output, in the high-dimensional regime $T\le d$ and for $p&lt;\min{q,2}$, has error $$\widetilde O_{κ,p,q}!\left( \frac{LR^{κ}{T^{{κ(1+1/p-(1/q-1/2)_+)-1}}}} \right),$$ after $T$ queries to a first-order oracle, solving the COLT 2015 open problem of (Guz15), up to logarithmic factors. At $(p,q)=(1,2)$, the rate is $\widetilde{O}(LR<sup>κ/T<sup>{2κ-1})$, including $\widetilde{O}(LR<sup>2/T<sup>{3})$ cubic decay in the smooth case.