Multi-Objective Neural Architecture Search

• Multi-objective NAS is a framework that formulates neural architecture design as an optimization problem with conflicting goals such as accuracy, computational cost, and latency.
• It leverages strategies like evolutionary algorithms, surrogate-assisted models, and novelty-driven search to approximate the Pareto front effectively.
• Recent advances incorporate training-free metrics and random-weight evaluation to reduce computational expense while achieving robust performance on standard benchmarks.

Multi-Objective Neural Architecture Search (NAS) is a research paradigm that formalizes neural network architecture generation as an optimization problem over multiple, often conflicting objectives—such as predictive accuracy, computational cost, parameter count, inference latency, or additional hardware-aware metrics. Unlike classical single-objective NAS, which seeks a singular optimal solution, multi-objective NAS aims to approximate the Pareto front, the set of architectures for which no other architecture is strictly better in all objectives. Recent developments in this field establish methodological foundations, propose diverse algorithmic strategies, and address scalability and diversity challenges, with extensive empirical validation across tabular NAS benchmarks and real-world search spaces.

1. Formal Multi-Objective NAS Problem Definition

Let $\Omega$ denote the search space of candidate neural network architectures $x$. Multi-objective NAS (MONAS) is formulated as the problem of identifying an approximation of the Pareto front defined by the objective vector: $$\min_{x \in \Omega}\; \mathbf{F}(x) = [\,f_1(x),\, f_2(x),\,\dots,\,f_m(x)\,] \in \mathbb{R}^m$$ under the convention that all $f_i$ are to be minimized. Examples of objectives include validation error or top-$1$ misclassification ($f_1$), computational complexity measured as FLOPs or MAdds ($f_2$), parameter count ($f_3$), or application-specific metrics such as inference latency, hardware energy, or interpretability proxies. The Pareto optimal set is: $$\mathcal{P}_S = \{\,x \in \Omega \mid \nexists\,y \in \Omega{:}~ y \prec x\,\}$$ where Pareto dominance is

$$x \prec y ~\Leftrightarrow~ \forall\,i\,,\,f_i(x) \leq f_i(y),~\exists\,j\,,\,f_j(x) < f_j(y)$$

and $\mathcal{P}_F = F(\mathcal{P}_S)$ is the Pareto front in objective space. Multi-objective NAS thus seeks to return either $\mathcal{P}_F$ or a close empirical approximation, such that practitioners can select the most suitable trade-off for deployment constraints (Vo et al., 2024).

2. Optimization Strategies and Algorithmic Frameworks

A spectrum of multi-objective optimization strategies power modern NAS frameworks.

Evolutionary Algorithms (EAs):

Population-based methods such as NSGA-II (Lu et al., 2018), SMS-EMOA, or MOEA/D (Calisto et al., 2020, Ito et al., 2023, Garcia-Garcia et al., 2023) dominate practice. Typical workflows maintain a population $P$ of candidate architectures, apply Pareto-based non-dominated sorting, and use crossover, mutation, and crowding distance or hypervolume indicators to promote both front advancement and diversity.

Surrogate-Assisted Optimization:

Due to the computational cost of full architecture training, many frameworks (e.g., EMONAS (Calisto et al., 2020), MOEA-BUS (Xue et al., 9 Feb 2026), POPNASv2 (Falanti et al., 2022), SiamNAS (Zhou et al., 3 Jun 2025)) integrate fast model-based surrogates (e.g., Random Forests, SVMs, MLPs, Siamese networks) to estimate objectives, with querying of the true expensive evaluators triggered selectively. Multi-fidelity surrogates and Bayesian optimization components are used in large or continuous search settings (Wei et al., 2 Sep 2025, Zhao et al., 2024, Amin et al., 2024).

Novelty and Diversity-Promoting Mechanisms:

Objective-driven EAs risk rapid convergence and diversity collapse. Explicit novelty scores—quantifying population spread in a descriptor space defined by training-free proxies or architecture encodings—are maximized as secondary objectives in approaches such as MTF-PDNS (Vo et al., 2024) and NEvoNAS (Sinha et al., 2022), sometimes supplanting direct accuracy objectives. Elitist Pareto archives and hybrid scoring rules encourage broad search-space coverage.

Weight Sharing and Supernet Approaches:

Large-scale practical NAS pipelines (e.g., OFA$^2$ (Ito et al., 2023), RAM-NAS (Mao et al., 25 Sep 2025)) employ weight-sharing supernets for rapid evaluation, allowing hundreds of subnetworks to be profiled via inherited weights and high-fidelity performance predictors or direct measurements.

Other Techniques:

• Progressive/greedy cell expansion (POPNASv2 (Falanti et al., 2022))
• Policy-gradient RL with non-stationary rewards (NPG (Chen et al., 2020))
• Continuous encoding and differential-evolution-based search for flexible, high-dimensional architectures (CGP-based (Garcia-Garcia et al., 2023, Wei et al., 2 Sep 2025))
• Adaptive region partitioning (LaMOO (Zhao et al., 2024)) to focus exploration.

3. Evaluation Metrics and Performance Indicators

Empirical evaluation in multi-objective NAS is grounded in well-established multiobjective indicators and transfer experiments.

• Generalized Distance Metrics:

The Inverted Generational Distance ($\mathrm{IGD}^+$) measures the average distance from known Pareto-optimal points (reference front) to the closest returned solution, quantifying convergence (Vo et al., 2024).

• Hypervolume Indicator (HV):

HV computes the volume covered in objective space by the nondominated set with respect to a fixed (worse-than-all) reference point, directly reflecting both convergence and coverage (Lu et al., 2018, Ito et al., 2023, Xue et al., 9 Feb 2026).

• Diversity Statistics:

Entropy or novelty-based measures are used to monitor global search-space coverage and prevent population collapse (Sinha et al., 2022, Xue et al., 9 Feb 2026).

• Downstream Generalization and Transferability:

Strong multi-objective NAS solutions are expected to transfer successfully to other domains or tasks (e.g., architectures searched on CIFAR-10 transferred to ImageNet and CIFAR-100), maintaining favorability in the objective trade-off (Vo et al., 2024, Wang et al., 2020).

• Sample Efficiency and Runtime:

The ratio of real (wall-clock or GPU hour) search cost required to reach a given HV or $\mathrm{IGD}^+$ threshold, and the number of true (expensive) evaluations avoided via surrogates and initialization, are critical practical metrics (Zhao et al., 2024, Zhou et al., 3 Jun 2025).

4. Recent Advancements: Training-Free Metrics and Novelty-Driven Search

The use of training-free metrics is a significant development in multi-objective NAS. MTF-PDNS (Vo et al., 2024) replaces direct accuracy evaluation with a descriptor space based on a vector of $$[ \operatorname{synflow}(x), \operatorname{jacov}(x), \operatorname{snip}(x), \operatorname{compl}(x) ]$$, where $$\operatorname{synflow}, \operatorname{jacov}, \operatorname{snip}$$ are zero-cost performance proxies, and $\operatorname{compl}$ is a complexity metric (e.g., parameter count or FLOPs). The search does not optimize these proxies directly, but instead maximizes a novelty score: $$\eta(x) = \pm\,\frac{1}{|\mathcal{A}|}\sum_{a\in\mathcal{A}} \| \phi(x) - \phi(a) \|_2$$ with the sign determined by Pareto survival. This promotes exploration without sacrificing solution quality. Empirical results show faster $\mathrm{IGD}^+$ convergence, higher HV, stronger diversity, and lower compute cost compared to direct objective optimization (Vo et al., 2024).

In parallel, random-weight evaluation (RWE) (Hu et al., 2020, Hu et al., 2021) provides a fast, robust proxy for accuracy, training only the last layer on random features, which, when coupled with population EAs, enables near-optimal Pareto set estimation within hours rather than days.

5. Practical Impact and Quantitative Benchmarks

Contemporary multi-objective NAS methods achieve leading performance on canonical benchmarks such as NAS-Bench-101/201/1Shot1, CIFAR-10, CIFAR-100, and ImageNet. Key results include:

Method	Benchmark(s)	Metric(s)	Best Result(s)
MTF-PDNS (Vo et al., 2024)	NAS-Bench-101/201	$\mathrm{IGD}^+$, HV	33% lower $\mathrm{IGD}^+$ than NSGA-II/MTF-MOENAS; $>$0.518 HV
MOEA-BUS (Xue et al., 9 Feb 2026)	CIFAR-10/ImageNet	Top-1 Acc @ MAdds	98.39%/601M (C10); 80.03%/610M (IN)
SiamNAS (Zhou et al., 3 Jun 2025)	NAS-Bench-201	Error/Params/FLOPs	Recovery of SOTA Pareto front with $<0.01$ GPU-days
POPNASv2 (Falanti et al., 2022)	CIFAR-10/100	Accuracy/Train Time	$>3.9\times$ speedup over PNAS for same accuracy
OFA$^2$ (Ito et al., 2023)	ImageNet	Error/Latency	Dense Pareto front, ensembles outperforming random models
NEvoNAS (Sinha et al., 2022)	CIFAR-10/100	Accuracy/Novelty	97.46%/3.4M (C10, S1); 83.95%/3.9M (C100)

This level of performance is enabled by innovations in surrogate usage, population initialization (e.g., uniform sampling in MAdds (Xue et al., 9 Feb 2026)), and algorithmic efficiency, with state-of-the-art methods reducing required GPU search time by an order of magnitude or more compared to earlier approaches.

6. Extensions: Hardware Awareness, Interpretability, and Non-Conventional Objectives

State-of-the-art frameworks target new application-driven objectives and deployment environments.

• Hardware-Aware Objectives:

Search objectives include real inference latency on robot edge hardware (RAM-NAS (Mao et al., 25 Sep 2025)), energy on analog IMC chips (Amin et al., 2024), and constraints on parameter or memory budget (MOEA-BUS, (Xue et al., 9 Feb 2026)).

• Interpretability and Debuggability:

Introspectability metrics—quantifying class disentanglement in the activation space—are incorporated as objectives via surrogate minimization, leading to architectures with improved trustworthiness, calibration, and robustness against noisy or mislabeled data (Carmichael et al., 2021).

• Approximate Arithmetic and Task Co-Design:

Architectural co-design with hardware components (e.g., approximate multipliers) is jointly optimized alongside standard prediction objectives to yield energy-error trade-offs for deployment (Pinos et al., 2021).

• Partitioned Search and MCTS-based Sampling:

Learning-powered partitioning (LaMOO (Zhao et al., 2024)) accelerates search in vast or high-dimensional spaces by iteratively refining the most promising regions via support vector machines and Monte Carlo tree search, delivering up to $5\times$ sample efficiency.

7. Ongoing Challenges and Future Directions

Open challenges include (i) proper alignment and calibration of training-free proxies to true downstream objectives, (ii) Pareto front estimation in the presence of high-dimensional, asymmetric search spaces, (iii) dynamic adaptation to evolving objectives or data distributions, (iv) integration of multi-task or Set-of-Pareto-Sets search (Zhou et al., 3 Jun 2025), and (v) scaling to open-domain NAS with billions of candidates and more than three objectives. Approaches combining novelty maximization, multi-population selection, hardware-in-the-loop evaluation, and extensible surrogate-based region partitioning are currently at the forefront, with ongoing work on surrogates for non-conventional objectives (robustness, fairness, debuggability) and more widely applicable search heuristics.

Taken together, multi-objective NAS synthesizes advances from evolutionary computation, Bayesian and surrogate-based optimization, population diversity preservation, and hardware/system-level co-design, now enabling the efficient computation of high-quality trade-off frontiers across a wide spectrum of deep learning architectures and deployment constraints (Vo et al., 2024, Xue et al., 9 Feb 2026, Ito et al., 2023, Wei et al., 2 Sep 2025).