On iteratively regularized first-order methods for simple bilevel optimization

Abstract: We consider simple bilevel optimization problems where the goal is to compute among the optimal solutions of a composite convex optimization problem, one that minimizes a secondary objective function. Our main contribution is threefold. (i) When the upper-level objective is a composite strongly convex function, we propose an iteratively regularized proximal gradient method in that the regularization parameter is updated at each iteration under a prescribed rule. We establish the asymptotic convergence of the generated iterate to the unique optimal solution. Further, we derive simultaneous sublinear convergence rates for suitably defined infeasibility and suboptimality error metrics. When the optimal solution set of the lower-level problem admits a weak sharp minimality condition, utilizing a constant regularization parameter, we show that this method achieves simultaneous linear convergence rates. (ii) For addressing the setting in (i), we also propose a regularized accelerated proximal gradient method. We derive quadratically decaying sublinear convergence rates for both infeasibility and suboptimality error metrics. When weak sharp minimality holds, a linear convergence rate with an improved dependence on the condition number is achieved. (iii) When the upper-level objective is a smooth nonconvex function, we propose an inexactly projected iteratively regularized gradient method. Under suitable assumptions, we derive new convergence rate statements for computing a stationary point of the simple bilevel problem. We present preliminary numerical experiments for resolving three instances of ill-posed linear inverse problems.