Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
131 tokens/sec
GPT-4o
10 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Alternating Stochastic Variance-Reduced Algorithms with Optimal Complexity for Bilevel Optimization (2404.11377v2)

Published 17 Apr 2024 in math.OC

Abstract: This paper studies the unconstrained nonconvex-strongly-convex bilevel optimization problem. A common approach to solving this problem is to alternately update the upper-level and lower-level variables using (biased) stochastic gradients or their variants, with the lower-level variable updated either one step or multiple steps. In this context, we propose two alternating stochastic variance-reduced algorithms, namely ALS-SPIDER and ALS-STORM, which introduce an auxiliary variable to estimate the hypergradient for updating the upper-level variable. ALS-SPIDER employs the SPIDER estimator for updating variables, while ALS-STORM is a modification of ALS-SPIDER designed to avoid using large batch sizes in every iteration. Theoretically, both algorithms can find an $\epsilon$-stationary point of the bilevel problem with a sample complexity of $O(\epsilon{-1.5})$ for arbitrary constant number of lower-level variable updates. To the best of our knowledge, they are the first algorithms to achieve the optimal complexity of $O(\epsilon{-1.5})$ when performing multiple updates on the lower-level variable. Numerical experiments are conducted to illustrate the efficiency of our algorithms.

Summary

We haven't generated a summary for this paper yet.