Adaptive Bilevel Search (ABS)

• Adaptive Bilevel Search (ABS) is a collection of adaptive bilevel optimization frameworks that coordinate an outer variable with inner problem solutions.
• It integrates multiple strategies such as curvature-aware updates, first-order methods under PŁ conditions, and fragmentation-aware guidance across diverse applications.
• ABS optimizes complex tasks in neural architecture search, hyperparameter learning, and computing networks by adaptively managing descent directions, step sizes, and solution accuracy.

Adaptive Bilevel Search (ABS) designates a set of adaptive bilevel optimization frameworks rather than a single standardized algorithm. In recent arXiv literature, the label is used for an auxiliary-program approach to differentiable neural architecture search, for adaptive fully first-order algorithms for nonconvex bilevel optimization under Polyak–Łojasiewicz conditions, for an adaptively inexact first-order method for hyperparameter learning, and for a fragmentation-aware bilevel service-mapping framework in Computing Power Networks. In each usage, an outer search variable is coordinated with inner decisions or inner solves, and the adaptation operates through coupled descent directions, step-size control, accuracy control, search-support reduction, or fragmentation-aware global guidance (Shukla et al., 28 Jun 2026, Shi et al., 30 Dec 2025, Salehi et al., 2023, Xie et al., 10 Jul 2025).

1. Terminological scope and research usage

The term ABS is used across several technical contexts, and the label does not identify a universal operator set. One ABS family is curvature-aware and second-order; one is fully first-order and tuning-free under PŁ assumptions; one uses backtracking line search with inexact oracles; and one is a distributed particle-swarm framework for graph-structured service mapping. The shared feature is a bilevel architecture in which upper-level progress is made only in conjunction with lower-level structure or lower-level feasibility.

ABS usage	Outer / inner structure	Adaptive components
AP-NAS	$\alpha$ for architecture, $w$ for weights	curvature-aware coupling, trust-region or line-search, damping
AF${}^2$BA / A${}^2$F${}^2$BA	$x$ for outer variable, $y,z$ for lower-level and penalty subproblems	AdaGrad-Norm or AC-GM, fully first-order stepsizes
MAID	$\lambda$ or $\theta$ for hyperparameters, $x$ for inner state	backtracking line search, dynamic $w$0 control
CPN ABS	$w$1 for co-location proportions, local PW-kGPP and IMCF decisions	elite-guided PSO, dynamic top-$w$2 masking, fragmentation-aware fitness

This multiplicity matters methodologically. In neural architecture search and hyperparameter learning, ABS is a bilevel optimization method for continuous outer variables. In Computing Power Networks, ABS is a bilevel search framework over proportion vectors whose induced lower-level solutions are graph partitions and routing plans. The common terminology therefore indicates adaptive outer–inner coordination, not a single convergence theory or implementation template.

2. Mathematical structure of bilevel search

The canonical bilevel problem in the cited literature is

$w$3

with the corresponding hyper-objective

$w$4

The stationarity notion used in the PŁ-based work is standard: $w$5 is an $w$6-stationary point of a differentiable function $w$7 if $w$8 (Shi et al., 30 Dec 2025). In differentiable NAS, the same hierarchy appears as

$w$9

where ${}^2$0 are continuous architecture parameters and ${}^2$1 are network weights (Shukla et al., 28 Jun 2026).

When the lower level is sufficiently regular, implicit differentiation yields a hypergradient. For NAS, near a lower-level optimum,

${}^2$2

The PŁ-based theory gives parallel formulas for ${}^2$3, including a penalty-regularized hyper-objective ${}^2$4 for non-strongly convex lower levels, and an explicit expression with the Moore–Penrose inverse of ${}^2$5 (Shi et al., 30 Dec 2025). In the hyperparameter-learning setting, strong convexity of the inner problem in ${}^2$6 permits the classical implicit-differentiation formula

${}^2$7

together with computable inexact hypergradient oracles (Salehi et al., 2023).

These formulations separate ABS variants by how they handle the outer–inner coupling. Some preserve lower-level optimality explicitly through coupled directions; some approximate the hypergradient with only first-order information; and some accept inexact inner solves but regulate their errors so that outer descent remains certified.

3. ABS in neural architecture search

In "Bilevel Optimization for Neural Architecture Search" (Shukla et al., 28 Jun 2026), ABS denotes the auxiliary-program approach within Auxiliary Program-based NAS (AP-NAS), with LP-DARTS and SOCP-DARTS as concrete instantiations. The central idea is to replace a hypergradient-only update of ${}^2$8 followed by retraining of ${}^2$9 with a joint descent direction ${}^2$0 that decreases validation loss to first order while preserving lower-level optimality to first order. At a reference point ${}^2$1 with ${}^2$2, the lower-level stationarity linearization yields

${}^2$3

where ${}^2$4 and ${}^2$5. ABS then solves the auxiliary program

${}^2$6

or an LP relaxation with ${}^2$7. Eliminating ${}^2$8 gives the best-response coupling

${}^2$9

The search is adaptive in three explicit senses. First, it is curvature-aware: ${}^2$0 follows the locally correct first-order manifold of inner optima through ${}^2$1. Second, step sizes are chosen by line-search along ${}^2$2 or via trust-region acceptance tests. Third, reduced Hessians, L-BFGS or rank-limited approximations, and damping ${}^2$3 are used to manage nonconvex curvature and memory. PyTorch Hessian–vector products, conjugate gradient solves, and warm-started restoration steps are the practical mechanism by which these updates are realized.

The AP-NAS paper positions ABS against both sampling-based NAS and gradient-based DARTS-style methods. Sampling-based methods are described as less sample-efficient because they evaluate more architectures, whereas ABS uses gradients and second-order inner curvature. Relative to DARTS, ABS enforces the linearized KKT stationarity of the inner problem and updates ${}^2$4 simultaneously rather than updating only ${}^2$5 and then retraining ${}^2$6. The consolidated comparisons report that differentiable NAS drastically reduces search cost relative to RL and evolutionary methods, with DARTS at approximately 4 GPU days versus NASNet and AmoebaNet in the thousands, and that under comparable CPU-hour budgets LP-DARTS and SOCP-DARTS outperform standard DARTS on CIFAR-10 and transfer better to ImageNet. The reported figures are DARTS: CIFAR-10 82.57%, ImageNet Top-1 39.09% (5-top1 64.10%), Params 3.38M; LP-DARTS: CIFAR-10 92.56%, ImageNet Top-1 42.50% (67.55%), Params 1.66M; SOCP-DARTS: CIFAR-10 90.56%, ImageNet Top-1 44.02% (68.76%), Params 2.23M. The same auxiliary program is also extended to simultaneous hyperparameter tuning and fine-tuning, including the LiFT transformer fine-tuning setting.

4. Fully first-order adaptive ABS under PŁ conditions

"Adaptive Algorithms for Nonconvex Bilevel Optimization under PŁ Conditions" (Shi et al., 30 Dec 2025) treats ABS as a fully first-order adaptive pipeline instantiated by AF${}^2$7BA and its accelerated variant A${}^2$8F${}^2$9BA. The lower-level function $x$0 and the penalty function $x$1 are assumed to satisfy a PŁ condition with respect to $x$2, and the analysis also assumes Lipschitz gradients and Lipschitz Hessians for $x$3 and $x$4, a lower bound on $x$5, and a sufficiently small choice of $x$6, set in the paper to $x$7. The algorithms solve two inner subproblems per outer iteration: a lower-level search for $x$8 approximating $x$9, and a penalty search for $y,z$0 approximating $y,z$1. They then form the fully first-order hypergradient estimator

$y,z$2

which avoids Jacobian-vector and Hessian-vector products entirely.

Adaptivity enters at both inner and outer levels. AF$y,z$3BA uses AdaGrad-Norm in the inner searches and in the outer update

$y,z$4

A$y,z$5F$y,z$6BA accelerates the inner loops with AC-GM, an auto-conditioned gradient method that is linesearch-free and uses evolving curvature surrogates $y,z$7 and a scale parameter $y,z$8. The paper proves linear convergence for AC-GM under the PŁ regime, with the optimal choice stated as $y,z$9, and uses this to reduce the inner iteration counts to $\lambda$0.

The complexity guarantees are explicit. Both algorithms achieve $\lambda$1 outer iteration complexity for finding an $\lambda$2-stationary point of the hyper-objective. AF$\lambda$3BA has $\lambda$4 first-order oracle complexity because its inner iteration counts scale as $\lambda$5, whereas A$\lambda$6F$\lambda$7BA achieves $\lambda$8 first-order complexity, matching the best-known tuned first-order methods up to logarithmic factors. The work emphasizes that no prior knowledge of Lipschitz or PŁ constants is required, but it also states several scope conditions: the analysis is deterministic, depends on PŁ and smoothness assumptions, and does not include empirical experiments.

5. Adaptively inexact ABS for hyperparameter learning

"An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning" (Salehi et al., 2023) presents MAID, identified there as an instantiation of ABS. The setting is bilevel learning with inexact function values and inexact hypergradients, motivated by imaging, inverse problems, and multinomial logistic regression. The reduced outer objective is

$\lambda$9

and, in the paper’s notation, $\theta$0 is the hyperparameter vector and $\theta$1 is the inner minimizer of $\theta$2. The algorithm solves the inner problem only approximately, solves the associated linear system only approximately, and forms an inexact hypergradient $\theta$3. A computable a posteriori bound on the error $\theta$4 links inner tolerances $\theta$5 to outer hypergradient accuracy.

The adaptive mechanism has two parts. First, MAID enforces a descent-direction condition

$\theta$6

so that $\theta$7 is guaranteed to be a descent direction for the true outer objective. If the condition fails, the algorithm tightens the inner tolerances by $\theta$8 and $\theta$9 and recomputes the inexact hypergradient. Second, MAID performs backtracking line search with an inexact Armijo test based on computable upper and lower bounds for the exact objective value. The sufficient-decrease rule is checked through a quantity $x$0 built from $x$1, gradient norms, the inner accuracy $x$2, and the term $x$3. If no trial step is accepted within the current backtracking budget, the algorithm again tightens $x$4 and retries. This realizes the principle stated in the paper: inner work is spent only when needed to certify descent or to accept a step.

The convergence theorem establishes finite termination of the line search, a sufficient-decrease inequality,

$x$5

a uniform positive lower bound on accepted step sizes, summability of $x$6, and therefore $x$7. The empirical section reports three classes of experiments. On a synthetic quadratic bilevel problem, MAID adaptively drove $x$8 to approximately $x$9 regardless of starting accuracy and reached lower loss with less total inner work than any fixed high-accuracy run. On multinomial logistic regression for MNIST with 1440 hyperparameters, MAID achieved lower validation loss than HOAG under the same total inner budget, and the a posteriori $w$00 sequence could be fit by an approximately summable schedule $w$01 that MAID discovered automatically. In variational image denoising, MAID reached training loss approximately $w$02 for smoothed total variation versus approximately $w$03–$w$04 for DFO-LS with dynamic accuracy, with test PSNR approximately $w$05 dB versus $w$06 dB, and for convex field-of-experts it achieved approximately $w$07 dB PSNR versus approximately $w$08 dB for smoothed TV learned by MAID on the same data.

6. Fragmentation-aware ABS for service mapping in Computing Power Networks

"A Fragmentation-Aware Adaptive Bilevel Search Framework for Service Mapping in Computing Power Networks" (Xie et al., 10 Jul 2025) uses ABS in a different application domain. Here the substrate is a Computing Power Network graph $w$09 with compute capacities on nodes and bandwidth capacities on links, and each service request is a graph $w$10 whose service functions have CPU demands and whose logical links have bandwidth demands. The joint decision problem includes binary placement variables $w$11 and routing selectors $w$12 under exact-one-placement, compute-capacity, endpoint–tunnel coherence, and network-bandwidth constraints. For a fixed arriving request, the operational objective reduces to minimizing external bandwidth cost on the current residual network. The paper proves NP-hardness of this per-request problem through a polynomial-time reduction from the Graph Bisection Problem.

ABS reformulates the placement component through continuous co-location proportions $w$13 over substrate nodes. For fixed $w$14, service-function mapping becomes a Proportional Weight-constrained $w$15-way Graph Partitioning Problem, and routing of the resulting Cut-LLs becomes an Integer Multi-Commodity Flow instance over precomputed path sets. The framework is therefore bilevel: the upper level searches over $w$16, while the lower level solves PW-kGPP, then IMCF, for each candidate. The paper states that if the upper level could exhaustively enumerate all feasible $w$17 and the lower level solved PW-kGPP and IMCF to optimality, the nested approach would recover the optimum of the per-request subproblem.

The search guidance is fragmentation-aware rather than gradient-based. For a candidate mapping $w$18, ABS computes Node Resource Exhaustion Degree (NRED), Computing-to-Bandwidth Utilization Gap (CBUG), and Path Node Valuelessness Level (PNVL), then minimizes

$w$19

The reported cumulative-distribution analyses indicate that NRED correlates strongest with global performance, then CBUG, then PNVL. The optimizer is a distributed elite-guided PSO. Particles represent continuous $w$20 vectors, negative entries are clipped to zero, dynamic top-$w$21 masking retains the most promising support and normalizes it to sum to one, and successful feasibility can shrink the active support for subsequent iterations. Workers run local swarms asynchronously, maintain elite and common subsets and local archives, and communicate best particles to a controller holding a bounded global archive.

The experimental setting includes a Waxman topology with 100 nodes and 500 links, Rocketfuel AS6461 with 129 nodes and 363 links, capacities uniformly in $w$22, and 2000 service entities per run with 50–100 service functions and edge density approximately $w$23. Against heuristics, metaheuristics, and learning-based baselines adapted to allow co-location, ABS and ABS initialized with RW-BFS dominate across both topologies and metrics. The headline results include acceptance ratio $w$24 versus $w$25 on the random topology and $w$26 versus $w$27 on Rocketfuel; revenue $w$28M versus $w$29M on the random topology and $w$30M versus $w$31M on Rocketfuel; long-term average revenue $w$32 versus $w$33 on the random topology and $w$34 versus $w$35 on Rocketfuel; and profit $w$36M versus $w$37M on the random topology and $w$38 versus $w$39 on Rocketfuel. The paper emphasizes that ABS improves compute utilization by at least $w$40 on the random topology and $w$41 on Rocketfuel over the best baseline, and that its service acceptance ratio is $w$42 higher than the best baseline in the complex Rocketfuel scenario. The reported limitations are equally explicit: the implementation optimizes CPU and bandwidth only, assumes accurate instantaneous capacities and stationary request statistics within a run, relies on heuristic lower-level solvers, and adds control-plane and computation overhead relative to simple heuristics.