Bi-Level Optimization Scheme

• Bi-Level Optimization Scheme is a hierarchical framework where the upper-level problem’s solution is constrained by a nested lower-level convex composite optimization.
• Dynamic Tikhonov regularization transforms the constrained problem into a series of unconstrained minimization tasks, enabling efficient convergence via techniques like FBi-PG acceleration.
• The framework provides rigorous convergence guarantees and trade-offs between inner and outer problems, validated through numerical experiments in regression and classification tasks.

A bi-level optimization scheme is a hierarchical optimization framework involving two nested optimization problems in which the solution of the upper-level (leader) problem is implicitly constrained by the optimizer(s) of a lower-level (follower) problem. Bi-level schemes have become foundational in modeling complex learning, design, and decision systems where objectives are entangled through nested or hierarchical structure, as encountered in hyperparameter tuning, meta-learning, multi-task learning, recommendation, adversarial design, and regularized statistical estimation.

1. Mathematical Formulation of Bi-level Optimization

A canonical convex composite bi-level optimization scheme takes the form:

• Lower-level (inner) problem (P):

$$\min_{x \in \mathbb{R}^n}\;\; \varphi(x) := f(x) + g(x)$$

where $f$ is convex, continuously differentiable (with Lipschitz $\nabla f$), and $g$ is proper, lower-semicontinuous, convex (may be extended-valued).

• Upper-level (outer) problem (OP):

$$\min_{x \in X^*} \;\; \omega(x) := \sigma(x) + \psi(x)$$

$X^* := \arg\min \{ \varphi(x) \}$, $\sigma$ convex and differentiable (Lipschitz $\nabla \sigma$), $\psi$ proper, lower-semicontinuous, convex. The notation $\beta := \beta_f + \beta_\sigma$ collects the Lipschitz constants.

The upper-level problem optimizes over the (possibly set-valued) solution set of the lower-level problem, creating a challenging compositional constraint (Merchav et al., 2024).

2. Dynamic Tikhonov Regularization and Objective Coupling

To address both tractability and convergence, bi-level composite convex problems are reformulated via a dynamically regularized (Tikhonov) composite objective:

$f$0

where at iteration $f$1,

• $f$2, $f$3, $f$4,
• As $f$5, $f$6, so penalized minimizers approach the original constrained bi-level solution set.

This strategy transmutes the constrained bi-level problem into a sequence of unconstrained but parameter-coupled minimization tasks, amenable to proximal algorithms. The variable rate $f$7 enables a tunable tradeoff between inner and outer-level convergence (Merchav et al., 2024).

3. Fast Bi-level Proximal Gradient (FBi-PG) Algorithm

The Fast Bi-level Proximal Gradient (FBi-PG) method, a FISTA-type algorithm, applies acceleration and carefully scheduled regularization for optimal convergence:

Algorithmic Steps:

1. Initialization:
   • $f$8, $f$9, $\nabla f$0
2. For $\nabla f$1
   • Compute regularized objective: $\nabla f$2
   • Define $\nabla f$3, $\nabla f$4 as above
   • Extrapolation: $\nabla f$5
   • Proximal step: $\nabla f$6
   • Acceleration: $\nabla f$7

The method leverages dynamic regularization, FISTA-style momentum, and convexity structure. It enforces optimality in the lower level through iterative shrinkage and alignment in the outer, leading to "simultaneous" progress (Merchav et al., 2024).

4. Theoretical Convergence Regimes and Trade-offs

FBi-PG provides rigorous rates:

• Inner-level (last-iterate) rate: For $\nabla f$8, $\nabla f$9 (optimal for composite convex), recovers $g$0 or $g$1 for $g$2, $g$3 for $g$4.
• Outer-level (best-iterate) rate: For $g$5, $g$6.
• Simultaneous rates ($g$7): $g$8, $g$9.
• Under Hölderian error bounds: If $$\min_{x \in X^*} \;\; \omega(x) := \sigma(x) + \psi(x)$$0 for some $$\min_{x \in X^*} \;\; \omega(x) := \sigma(x) + \psi(x)$$1 (an error-bound on the lower-level), for $$\min_{x \in X^*} \;\; \omega(x) := \sigma(x) + \psi(x)$$2 one has $$\min_{x \in X^*} \;\; \omega(x) := \sigma(x) + \psi(x)$$3 and $$\min_{x \in X^*} \;\; \omega(x) := \sigma(x) + \psi(x)$$4.

The parameter $$\min_{x \in X^*} \;\; \omega(x) := \sigma(x) + \psi(x)$$5 controls the inner/outer rate trade-off: $$\min_{x \in X^*} \;\; \omega(x) := \sigma(x) + \psi(x)$$6 accelerates inner-level at the expense of the outer, $$\min_{x \in X^*} \;\; \omega(x) := \sigma(x) + \psi(x)$$7 does the opposite. Strong convexity at either level is not assumed—only convexity and Lipschitz gradients are required. Error-bound assumptions are only necessary for improved simultaneous/pointwise rates (Merchav et al., 2024).

5. Practical Implementation and Numerical Results

FBi-PG was benchmarked on:

• Songs Release-Year Prediction: Inner is least-squares regression ($$\min_{x \in X^*} \;\; \omega(x) := \sigma(x) + \psi(x)$$8), outer $$\min_{x \in X^*} \;\; \omega(x) := \sigma(x) + \psi(x)$$9-sparsity penalty. FBi-PG with $X^* := \arg\min \{ \varphi(x) \}$0 was compared to Bi-SG and FISTA with fixed $X^* := \arg\min \{ \varphi(x) \}$1. FBi-PG achieved superior inner-level convergence and traced the full Pareto frontier in outer-vs-inner trade-off plots, unlike fixed-regularization schemes which stalled.
• Fake-News Classification: Inner is logistic loss on 500 samples, outer is $X^* := \arg\min \{ \varphi(x) \}$2 penalty. FBi-PG achieved high-accuracy at both levels; fixed $X^* := \arg\min \{ \varphi(x) \}$3 methods stalled in inner and never reached sparsity.

Numerical experiments confirmed that a dynamically decaying regularization schedule outperformed both fixed-regularization and classical nested optimization approaches, with rates consistent with the theoretical analysis (Merchav et al., 2024).

6. Comparative Analysis and Broader Impact

The FBi-PG scheme marks a significant advance in first-order bi-level optimization. In contrast to fixed Tikhonov or double-loop schemes, it achieves the fastest known convergence rates under minimal convexity requirements. In the convex composite bi-level context, these rates are optimal unless additional restrictive assumptions (e.g., strong convexity or error-bound conditions) hold.

By unifying Tikhonov regularization and FISTA acceleration, FBi-PG closes several open gaps: it provides a seamless trade-off between levels, supports composite extended-valued constraints, and delivers robust performance on realistic tasks where both levels are structured and regularized. The dynamic penalization principle underlying FBi-PG is broadly applicable in regularized ML, signal processing, and control applications requiring hierarchical or nested optimization (Merchav et al., 2024).

7. Assumptions, Limitations, and Applicability

• No strong convexity is assumed for either objective; only convexity and Lipschitz gradient are needed.
• The approach is tailored to convex, composite structures with proper, l.s.c. regularizers.
• Error-bound assumptions (Hölderian) are only required for simultaneous/pointwise convergence, not for basic rate guarantees.
• The method is generic to any problem of the form min $X^* := \arg\min \{ \varphi(x) \}$4 s.t. $X^* := \arg\min \{ \varphi(x) \}$5 with proximable structures.

FBi-PG is therefore applicable to a wide range of ML and signal processing problems where both the main task and the regularization/selection criteria are composed of differentiable and non-differentiable convex functions.

---

References

• Dynamic FISTA for Convex Composite Bi-Level Optimization (Merchav et al., 2024)