Bilevel Optimization for Neural Architecture Search

Abstract: Bilevel optimization has become an influential and widely adopted framework for addressing hierarchical optimization problems in machine learning, providing an effective approach to modeling the interaction between two levels of optimization, with applications such as hyperparameter tuning, meta-learning, adversarial training, and data poisoning. Neural Architecture Search (NAS), a subfield of hyperparameter optimization, is a prime example of a bilevel optimization problem, with architecture parameters optimized at the outer-level and network weights optimized at the inner level. This paper presents a structured overview of NAS through the lens of bilevel optimization. We categorize existing NAS approaches into two main classes: sampling-based methods, which search optimal architectures using different architecture samplers, and bilevel theory-based methods, which solve the architecture search problem using bilevel optimization principles. We further highlight our current research direction, wherein the bilevel NAS formulation is addressed through an auxiliary mathematical programming framework. This framework enables the systematic integration of second-order information from the model's training loss function and ensures the optimality of the model parameters while modifying architecture parameters. By simultaneously updating the architecture and model parameters along their respective optimal descent directions derived from the auxiliary mathematical program, these methods achieve more principled and theoretically consistent results. The same auxiliary program can also be used for simultaneous hyperparameter and model fine-tuning. A comparative analysis shows that bilevel theory-based approaches generally outperform sampling-based methods, both in accuracy and efficiency.

• The paper presents a bilevel optimization framework that structures NAS by treating architecture selection and weight training as nested problems.
• It introduces differentiable methods such as hypergradient-based and auxiliary program-based updates, significantly enhancing efficiency and accuracy.
• Empirical results show notable improvements in accuracy and computational cost on benchmarks like CIFAR-10 and ImageNet.

Bilevel Optimization for Neural Architecture Search: A Technical Analysis

Introduction and Motivation

Neural Architecture Search (NAS) is a core component of automated machine learning, aimed at automating the design of neural network topologies to maximize predictive performance for specific tasks. The increasing depth and complexity of modern neural networks accentuate the risks of over-parameterization and overfitting, making principled architecture selection critical. NAS is fundamentally a bi-level optimization problem: the upper-level (outer) selects architectural hyperparameters (e.g., layer types, connections), while the lower-level (inner) solves for network weights given a fixed architecture. The inherent hierarchy and coupling between these decision spaces motivates the adoption of bilevel optimization frameworks to represent the interplay between architecture choices and model parameter training.

Figure 1: Representation power and overfitting in ANNs.

Fundamentals of Bilevel Optimization

A bilevel optimization problem (BOP) consists of two nested optimization problems: the upper-level (leader) and the lower-level (follower), where the follower solves a subordinate problem conditioned on the leader's decision. Formally, given upper-level variables $x$ and lower-level variables $y$, the solution set of the follower induces a constraint on feasible leader decisions. This leads to complex, typically non-convex feasible regions and introduces sensitivity, non-differentiability, and multiple optima at the lower level.

Figure 3: Single Lower-level Optimal Solution in a BOP.

Figure 2: Interdependence between the upper- and lower-level problems.

Key mathematical properties include:

• Non-convexity and potential disconnectedness of feasible regions.
• Strong NP-hardness, even in apparently simple cases.
• Solution ambiguity in presence of multiple follower optima, prompting optimistic (cooperative) and pessimistic (adversarial) modeling paradigms.

Bilevel frameworks enable direct representation of hierarchical objectives such as hyperparameter optimization, adversarial robustness, inverse optimization, and, centrally, NAS.

Figure 4: Geometric illustration of a considered BOP.

Neural Architecture Search via Bilevel Optimization

In NAS, architecture parameters $A$ are optimized at the upper level to minimize a validation loss $\mathcal{L}_v(A,\hat{W})$, where $\hat{W}$ denotes the optimal network weights learned at the lower level by minimizing a training loss $\mathcal{L}_t(A, W)$. This leads to the canonical bilevel structure:

$$\min_A\ \mathcal{L}_v(A, \hat{W})\quad \text{subject to } \hat{W} = \operatorname{argmin}_W\ \mathcal{L}_t(A, W)$$

Figure 5: Upper- and lower-level optimization in NAS.

The nested dependency makes the evaluation of $A$ contingent on solving a non-convex training problem for each candidate architecture. This structure is critical to avoiding suboptimal architectures that may perform well with non-optimized or partially-trained weights (thus mitigating selection bias and overfitting tendencies). The performance and generalization of an architecture can only be reliably measured with appropriately trained weights, emphasizing the necessity for a bilevel view over single-level approximations.

Figure 6: Multilayer perceptron architecture search using grid search.

Figure 7: Bayesian optimization-based sampler in an NAS setting.

Figure 8: EC based samplers for NAS.

Figure 9: RL based sampler for NAS.

Figure 10: Building task-specific networks by selecting operations from a search space.

Methodological Taxonomy of NAS Approaches

Sampling-based NAS

Sampling-based approaches handle the bilevel structure primarily via heuristic or stochastic exploration of the architecture space. Principal families include:

• Grid/random search: Exhaustive or random exploration of architectural configurations; becomes intractable in high dimensions.
• Bayesian optimization (TPE, GP): Surrogate models steer exploration toward promising regions in the search space; more sample-efficient but sensitive to surrogate fidelity.
• Evolutionary algorithms: Populations of architectures evolve under mutation and selection; naturally support multi-objective trade-offs (e.g., accuracy vs. resource cost).
• RL-based controllers: Policy networks (e.g., RNNs) sequentially sample architectural motifs, refined via reward feedback from model evaluations.

While flexible and broadly applicable, sampling-based techniques are computationally expensive and often require substantial GPU resources. They tend to be outperformed by differentiable approaches in both accuracy and efficiency, especially under budget constraints.

Bilevel Theory-Based NAS

These methods leverage bilevel optimization principles and differentiability to provide principled solutions:

• KKT Reformulations: The lower-level optimality conditions (first-order or full KKT) are embedded as constraints or penalized in the upper-level problem, resulting in a (relaxed) single-level formulation amenable to standard optimizers. This approach enables joint or alternating updates, but its efficiency depends on approximating nonconvex constraints and handling large systems.
• Surrogate-based methods: Use learned or analytic surrogates for the upper-level objective, lower-level value function ($\varphi$-mapping), or response mapping ($\Psi$-mapping), substituting expensive nested optimization with tractable approximations.
• Hypergradient-based approaches: Compute upper-level gradients w.r.t. architecture parameters by differentiating through the lower-level training procedure (via implicit or unrolled differentiation), yielding hypergradients for efficient updates. The most impactful instantiation is DARTS [liu2019darts], which relaxes the discrete search space to a continuous one and performs gradient-based bilevel optimization, dramatically improving scalability.

Figure 11: A simple cell structure under continuous relaxation.

Figure 12: A cell-based neural network architecture.

• Auxiliary Program-based NAS: The latest methods frame each architecture update as the solution to an auxiliary mathematical program (e.g., LP, SOCP), guaranteeing that descent in the validation loss is always along feasible directions respecting the underlying bilevel structure. This ensures that architecture parameter changes do not violate model optimality for the weights and can be efficiently solved using structured Hessian approximations and reduced memory footprints.

  Figure 13: Determination of the optimal step-size $y$0.

Numerical Performance and Comparative Analysis

Strong empirical results are demonstrated for bilevel-theory methods. For instance, AP-NAS and DARTS variants (LP-DARTS, SOCP-DARTS) consistently achieve higher accuracy and model efficiency on benchmarks such as CIFAR-10 and ImageNet, and do so under significantly reduced computational budgets compared to RL- and evolutionary-based NAS (2606.29582).

Numerical highlights:

• SOCP-DARTS/LP-DARTS: Up to 10 percentage points higher accuracy and greater parameter efficiency versus vanilla DARTS under CPU-based, time-constrained experiments.
• Auxiliary program-based updates: Further performance gains due to guaranteed model optimality and efficient Hessian handling via techniques like GNPP and randomized subsampling.
• Meta and one-shot approaches: Methods such as Once-for-All and Autoformer showcase exceptional transferability across tasks and input modalities.

Implications and Future Directions

Theoretical implications include the formalization of NAS as a nested, non-convex, bilevel optimization problem, aligning network structure search with hyperparameter optimization, meta-learning, and adversarial robustness under a unified framework. Practically, the auxiliary mathematical programming approach enables scalable, principled, and transferable architecture search, holding direct promise for low-resource (CPU-based) and large-scale (e.g., LLM, transformer, multimodal) settings.

Research opportunities arise in:

• Efficient Hessian approximation for large-scale networks (block-diagonal, low-rank, stochastic estimation).
• Joint optimization schemes unifying multiple hyperparameters (architecture, learning rates, regularization).
• Discrete/structured search spaces with rigorous relaxation and rounding strategies.
• Zero-cost proxies and hybrid methods that combine bilevel differentiation with sampling-based or analytic predictors.
• Extension to multimodal, foundation, and hardware-constrained architectures.

Conclusion

The paper systematically demonstrates that framing NAS as a bilevel optimization problem, and explicitly respecting its hierarchical structure, enables both higher performance and superior computational tractability compared to sampling-driven methods. Auxiliary mathematical programming, differentiable relaxations, and hypergradient-based updates form the backbone of state-of-the-art approaches. Future advances in scalable bilevel solvers and hybrid optimization strategies are positioned to accelerate progress in automated deep learning and efficient foundation model deployment.