Bi-level and Dual-Embedding Optimization

• Bi-level and dual-embedding optimization is a framework that nests an upper-level problem over a lower-level one, using dual variables to enforce constraints and regularize solutions.
• The methodology reformulates complex nested problems into single-level systems via first-order methods, enhancing convergence and handling nonconvexity in challenges like meta-learning and robust adaptation.
• Applications include prompt-free fine-tuning of vision models, hyperparameter optimization, and dynamic system identification, demonstrating reduced overfitting and improved scalability.

Bi-level and dual-embedding optimization refers to a family of optimization paradigms where one optimization task ("upper-level") is performed with a subordinate ("lower-level") optimization problem nested inside it, and where embedding—either via explicit parameterizations or via dual/primal-dual variables—is systematically exploited to encode constraints, structure, or regularization. This framework is foundational in areas such as meta-learning, hyperparameter optimization, neural architecture search, robust model adaptation, and dynamical system identification. Both bi-level optimization (BLO) and dual-embedding methodologies have advanced to address issues of computational tractability, convergence, overfitting, and constraint management in a range of complex machine learning and control settings.

1. Formal Bi-Level Optimization Structure

A general bi-level optimization problem is expressed as

$$\min_{x\in\mathcal X,\;y\in\mathcal Y} \; F(x, y) \quad \text{s.t.} \quad y \in \arg\min_{u \in \mathcal Y} G(x, u)$$

where $x$ (outer variable) is selected to minimize the "upper-level" objective $F(x, y)$, while $y$ (inner variable) must itself be a minimizer of a "lower-level" objective $G(x, \cdot)$, potentially subject to additional constraints. This nested structure introduces significant computational and analytical challenges, especially when $G(x, \cdot)$ is nonconvex or has multiple minimizers, or when the lower-level problem includes constraints that couple $x$ and $y$.

Single-level reformulations are achieved either by substituting first-order optimality (KKT) conditions or via Lagrangian dualization, but naive elimination often leads to ill-posed problems or intractable Jacobian/Hessian computations (Liu et al., 2022, Sow et al., 2022, Jiang et al., 2024).

2. Dual-Embedding and Primal-Dual Reformulation

Dual-embedding refers to embedding the lower-level optimality or constraints into the single-level problem not only through the original variables but also via explicit dual multipliers or auxiliary embeddings. For unconstrained lower-level problems, the method introduces dual variables $v$ enforcing the LL optimality conditions within a unified KKT system: $$\begin{aligned} \nabla_x F(x,y) - \nabla^2_{xy}f(x,y)v &= 0, \ \nabla_y F(x,y) - \nabla^2_{yy}f(x,y)v &= 0, \ \nabla_y f(x,y) &= 0. \ \end{aligned}$$ This enables single-loop first-order updates, eliminates the necessity for repeated inner high-accuracy optimization, and yields efficient, convergent algorithms such as BAGDC (Liu et al., 2022).

In constrained BLO, Lagrangian or penalty terms with dual variables $x$0 or $x$1 explicitly encode the feasibility of the inner solution (e.g., $x$2). These updates are realized in primal-dual or saddle-point algorithms such as PDBO (Sow et al., 2022) and BLOCC (Jiang et al., 2024). Dual embedding thus generalizes both KKT-based and constraint-penalized approaches and is particularly effective in handling multiple LL optima, nonconvexity, or complicated constraint sets.

3. Representative Algorithms and Algorithmic Structures

BLO-SAM: Overfitting-Preventing Bi-Level Fine-Tuning

The BLO-SAM method is a bi-level framework for prompt-free, data-efficient fine-tuning of SAM for semantic segmentation. The optimization alternates between training model weights $x$3 on one subset $x$4 (lower level) and learning a prompt embedding $x$5 on a disjoint subset $x$6 (upper level), with respective updates: $x$7 where the total loss $x$8 trades off cross-entropy and Dice segmentation losses. The dual-embedding aspect here is the learnable prompt embedding $x$9, replacing explicit user prompts, and the bi-level structure is enforced through separate dataset splits and alternating optimization. Overfitting is reduced as $F(x, y)$0 (hyperparameter) never interacts with the same data used to train $F(x, y)$1 (Zhang et al., 2024).

Primal-Dual Bilevel Optimizer (PDBO) and BLOCC

PDBO (Sow et al., 2022) and BLOCC (Jiang et al., 2024) convert bi-level problems with (possibly multiple) LL optima or coupled LL constraints into single-level saddle-point/penalty problems. For example, PDBO uses a smoothed value-function constraint,

$F(x, y)$2

and realizes updates by alternating projected gradient steps in $F(x, y)$3 (primal) and $F(x, y)$4 (dual).

BLOCC addresses BLO with coupled constraints via a max-min reformulation: $F(x, y)$5 where $F(x, y)$6 are coupled LL constraints and $F(x, y)$7 is the LL value function. BLOCC alternates inner saddle-point solves in $F(x, y)$8 and projected gradient steps in $F(x, y)$9, with established convergence guarantees and complexity rates.

BAGDC: Alternating Gradient with Dual Correction

BAGDC (Liu et al., 2022) accelerates traditional GBLO/IGBLO schemes by making dual correction steps explicit. LL variables $y$0 are updated by a single-step gradient descent, dual variables $y$1 by an explicit correction based on the KKT block, and $y$2 by a corrected hypergradient step. This design removes the requirement for repeated inner-loop solves and demonstrates $y$3 convergence rates, applicable to settings with either strongly convex or merely convex lower-level objectives.

4. Applications in Machine Learning and Modeling

Bi-level and dual-embedding optimization underlie key advances across various scientific and engineering domains:

• Prompt-free Vision Model Fine-tuning: BLO-SAM enables fully automatic semantic segmentation in new domains (e.g., medical imaging), outperforms SOTA, and crucially provides resistance to overfitting in few-shot regimes by splitting dataset exposure between parameter and embedding training (Zhang et al., 2024).
• Hyperparameter Optimization and Meta-Learning: These techniques provide a rigorous, scalable foundation for choosing architecture, learning rates, or constraints, naturally navigating settings with multiple inner minima or coupled constraints (Sow et al., 2022, Jiang et al., 2024).
• Learning Dynamical System Embeddings: Koopman operator identification for nonlinear-control settings can be robustly formulated as a bi-level problem with dual embeddings, simultaneously learning both the encoder (state lifting) and the linear dynamics in the lifted space. This guarantees long-horizon consistency and places dynamic constraints on the embedding itself (Huang et al., 2023).
• Infrastructure Optimization and Network Design: Genuinely large-scale, constrained bi-level problems (as in transportation or infrastructure planning) are tractable using primal-dual penalty approaches (e.g., BLOCC), which can handle thousands of coupled constraints and variables efficiently (Jiang et al., 2024).

5. Key Implementation Strategies and Hyperparameters

Each methodology demands specific design choices:

Method	Embedding/Variable Structure	Main Hyperparameters
BLO-SAM	Prompt embedding $y$4	$y$5, LR=5e-3, LoRA rank $y$6
PDBO	Dual variable $y$7	$y$8 (regularization), step sizes
BLOCC	Duals $y$9 (LL constraints)	$G(x, \cdot)$0, inner/outer loop counts, step $G(x, \cdot)$1
BAGDC	Dual $G(x, \cdot)$2 (KKT multiplier)	$G(x, \cdot)$3, $G(x, \cdot)$4 schedule
Koopman bi-level	Encoder $G(x, \cdot)$5, decoder $G(x, \cdot)$6, $G(x, \cdot)$7	$G(x, \cdot)$8 (embedding dim), $G(x, \cdot)$9 (ridge), $G(x, \cdot)$0 (outer loss weight)

BLO-SAM, for example, uses AdamW optimizers with cosine decay, first-order approximation for backpropagation through the lower level, and LoRA modules injected only into the mask decoder, with the embedding learned as a trainable vector broadcast to all prompt positions (Zhang et al., 2024).

Dual-embedding-based methods all favor first-order (gradient-only) solutions, employ projector operations to handle constraints (typically Euclidean or simplex projections), and avoid nested second-order (Hessian/Jacobian) solves—a major efficiency advantage (Sow et al., 2022, Liu et al., 2022, Jiang et al., 2024).

6. Convergence Properties and Theoretical Guarantees

Dual-embedding and primal-dual schemes enable provable convergence rates under general structural assumptions:

• PDBO achieves $G(x, \cdot)$1 complexity for strongly convex LL problems, and $G(x, \cdot)$2 for nonconvex outer levels (Sow et al., 2022).
• BLOCC delivers $G(x, \cdot)$3 complexity in generic cases, and $G(x, \cdot)$4 in affine LL constraints, with rigorous finite-time guarantees and no requirement for Hessian inversion (Jiang et al., 2024).
• BAGDC attains $G(x, \cdot)$5-type stationarity with single-loop updates and dramatically lower wall time compared to classic GBLO/IGBLO approaches (Liu et al., 2022).

A consistent finding is that embedding dual variables or constraint parameters into the optimization loop eliminates pathological error accumulation from inexact differentiation and allows robust navigation of LL nonuniqueness or flat directions.

7. Challenges, Limitations, and Directions

Current limitations of these frameworks include the need for strong convexity or local restricted secant conditions for optimal rates, careful step size management, and, in some penalty or constraint-embedded approaches, the need for large penalty parameters ($G(x, \cdot)$6). Moreover, while first-order methods are effective for large-scale BLO, nonconvex LL (e.g., deep neural nets) require additional regularization or prox-linear penalties. Extensions to stochastic or multi-level scenarios are being actively developed, with variance-reduced loops and cascade-type primal-dual strategies suggested as promising directions (Jiang et al., 2024).

Further research explores direct generalization to multi-level, stochastic, and highly nonconvex regimes, as well as enhanced dual-embedding schemes for learning representations (as in Koopman operator learning), domain-specific inductive biases, and engineering-scale optimization (Huang et al., 2023, Jiang et al., 2024).

---

References

• "BLO-SAM: Bi-level Optimization Based Overfitting-Preventing Finetuning of SAM" (Zhang et al., 2024)
• "A Primal-Dual Approach to Bilevel Optimization with Multiple Inner Minima" (Sow et al., 2022)
• "A Primal-Dual-Assisted Penalty Approach to Bilevel Optimization with Coupled Constraints" (Jiang et al., 2024)
• "Towards Extremely Fast Bilevel Optimization with Self-governed Convergence Guarantees" (Liu et al., 2022)
• "Learning Koopman Operators with Control Using Bi-level Optimization" (Huang et al., 2023)