Low-Rank Bilevel Optimization

• Low-rank bilevel programs are a class of optimization frameworks that approximate second-order derivatives to scale stochastic bilevel problems.
• They introduce methods like BSG-N-FD and BSG-1, using finite differences and rank-1 approximations to bypass explicit Hessian evaluations.
• These approaches offer improved computational efficiency and competitive convergence, especially in large-scale and constrained settings.

Low-rank bilevel programs constitute a class of large-scale stochastic bilevel optimization algorithms designed to mitigate the computational and memory burdens typically incurred by second-order derivatives in bilevel problems—especially when the lower-level is unconstrained or features complex constraints. The central innovation is to employ low-rank or approximate representations for the Hessian and Jacobian blocks that arise in hypergradient computation, yielding methods that scale efficiently with problem dimension, operate without explicit Hessian formation, and maintain competitive convergence properties in both noiseless and stochastic contexts (Giovannelli et al., 2021).

1. Structure of Stochastic Bilevel Optimization and Computational Challenges

Bilevel optimization, in its stochastic and large-scale setting, is prevalent in contemporary machine learning tasks including continual learning, neural architecture search, hyperparameter tuning, and adversarial training. The generic bilevel problem can be summarized as: $$\min_{x \in X} f_u(x, y) \quad \text{subject to} \quad y \in \arg\min_{y \in Y(x)} f_\ell(x, y)$$ where $f_u$ and $f_\ell$ are typically expectations over data. The solution $y(x)$ leads to an upper-level objective $f(x) = f_u(x, y(x))$, whose gradient (the hypergradient) involves evaluation of both first- and second-order derivatives: $$\nabla f(x) = \nabla_x f_u(x, y(x)) - \nabla^2_{xy} f_\ell(x, y(x)) [\nabla^2_{yy} f_\ell(x, y(x))]^{-1} \nabla_y f_u(x, y(x))$$ A standard bilevel stochastic gradient (BSG) method estimates the necessary gradients and Hessian blocks through stochastic sampling, minibatching, and noisy measurements. Explicit calculation or inversion of the Hessian $\nabla^2_{yy} f_\ell \in \mathbb{R}^{m \times m}$ is computationally infeasible for large $m$, and evaluating or differentiating KKT systems in the presence of lower-level constraints further compounds the challenge.

2. Low-rank Bilevel Algorithms: BSG-N-FD and BSG-1

Two principal variants are introduced to alleviate the computational complexity associated with the adjoint system in hypergradient computation:

2.1 BSG-N-FD ("Newton + Finite Differences")

This approach uses an iterative linear solve (such as Conjugate Gradient, CG) for the adjoint system

$$\nabla^2_{yy} f_\ell(x_k, \tilde{y}_k) \lambda = \nabla_y f_u(x_k, \tilde{y}_k)$$

but replaces explicit Hessian-vector products by two-point finite difference approximations: $$H v \approx \frac{\nabla_y f_\ell(x, y + \epsilon v) - \nabla_y f_\ell(x, y - \epsilon v)}{2\epsilon}$$ Similarly, cross-Hessian-vector products are approximated by finite differences. The resulting descent direction is formed as

$$d_k = -\left(g^u_x(x_k, y_k) - \text{Cross‐Hessian‐FD} \cdot \lambda\right)$$

This procedure fully avoids assembling or storing Hessian matrices, and all necessary computations are realized through first-order gradient computations and finite differencing.

2.2 BSG-1 ("Rank-1 Approximation")

This variant uses rank-1 outer product approximations to avoid Hessians entirely: $$\nabla^2_{xy} f_\ell \approx \nabla_x f_\ell\,\nabla_y f_\ell^T, \quad \nabla^2_{yy} f_\ell \approx \nabla_y f_\ell\,\nabla_y f_\ell^T$$ The adjoint system can then be solved in closed-form: $$\lambda = \frac{\nabla_y f_\ell^T \nabla_y f_u}{\|\nabla_y f_\ell\|^2}$$ yielding the rank-1 hypergradient update: $$\nabla f \approx \nabla_x f_u - \left(\frac{\nabla_y f_\ell^T \nabla_y f_u}{\|\nabla_y f_\ell\|^2}\right)\nabla_x f_\ell$$ In the constrained lower-level scenario, similar rank-1 approximations are used for the KKT-block matrices.

3. Computational Complexity and Memory Analysis

The complexity of low-rank methods stands in contrast to full-rank approaches:

Method	Per Iteration Cost	Memory Usage
BSG-N-FD	$O((p+1)(n+m))$ (p: CG steps)	Vectors of length $\max(n,m)$
BSG-1	$O(n+m)$	$O(n+m)$ numbers
Full-rank (BSG-H)	$O(m^2)$–$O(m^3)$	$O(m^2)$

BSG-N-FD requires a fixed small number of CG iterations ($p$) for the adjoint solve, each costing two gradients in the lower-level variable. BSG-1 uses only first-order gradients with simple vector arithmetic, and thus achieves minimal storage and computation. Full Hessian-based BSG-H is substantially more expensive due to dense matrix operations.

4. Convergence Properties under Inexactness

Theoretical analysis demonstrates that as long as the hypergradient is computed with residual error $O(\alpha_k)$ (where $\alpha_k$ is the stepsize), and LL solution error and stochastic noise are similarly controlled, the BSG iterates achieve standard rates:

• Nonconvex cases: $O(1/\sqrt{K})$ decrease in expected gradient norm.
• Strongly convex cases: $O(1/K)$ expected suboptimality gap.
• Convex cases: $O(1/\sqrt{K})$ for the minimum function value over iterates.

The low-rank methods introduce an additional "adjoint residual." If this is bounded by $C_e \alpha_k$, convergence rates mirror those of exact BSG. While BSG-1 is a heuristic not directly subsumed by the general theory, empirically, its residuals decrease with stepsize, so it often achieves comparable rates (Giovannelli et al., 2021).

5. Empirical Performance and Practical Considerations

Experiments with synthetic quadratic bilevel problems ($n = m = 300$), both unconstrained and with lower-level (LL) constraints, and with neural continual learning on CIFAR-10 (using DNNs with ∼200K parameters), reveal the following:

• On unconstrained problems:
  • BSG-N-FD and BSG-H (full Hessian) exhibit fastest convergence in both iteration count and wall-clock time.
  • BSG-1 performs slower in iterations but surpasses DARTS and remains efficient in runtime.
• With lower-level constraints and moderate/heavy gradient/Hessian noise:
  • BSG-N-FD remains robust and outperforms SIGD (for linear constraints) and BSG-H (which degrades under high noise).
  • For quadratic constraints, only BSG-N-FD and BSG-H apply; BSG-N-FD is more stable.
• In continual learning with constraints to prevent catastrophic forgetting:
  • BSG-N-FD and BSG-1 enforce constraints with minimal additional computational cost and outperform StocBiO and DARTS on aggregate.

Parameter selection for BSG-N-FD involves a finite difference offset $\epsilon \approx 0.1$ and typically 3–5 CG/GMRES steps per update; excessive values degrade accuracy or increase cost. For BSG-1, performance is optimal for lower-level objectives with Gauss–Newton structure (e.g. cross-entropy) but degrades elsewhere.

6. Truncation, Rank Selection, and Method Scope

Low-rank methods can trade off cost and accuracy via truncation of the Neumann series (in more general BSG schemes), where the truncation depth $q$ dictates approximation error $O((1-\|B\|)^{q+1})$ and cost $O(q)$. BSG-1 always employs a rank-1 approximation, performing best for least-squares-type LL losses. In highly non-linear or non-least-squares LLs, BSG-N-FD is preferable due to its flexibility and robustness.

7. Connections to Related Work

Low-rank bilevel algorithms relate to broader efforts in efficient bilevel optimization, including implicit differentiation with stochastic gradients, two-timescale frameworks, and constrained optimization via smoothed or penalty-based approaches. Notable references include "Differentiable Architecture Search" (DARTS), StocBiO, and linearly constrained bilevel solvers. The comparative empirical and theoretical landscape is detailed in works by Giovannelli, Kent, and Vicente (Giovannelli et al., 2021), Liu et al. (DARTS), and others cited within (Giovannelli et al., 2021).

These low-rank approaches substantially reduce computational barriers, broadening the feasibility of stochastic bilevel optimization with large parameter dimensions and constrained formulations in modern machine learning.