Hardware-Aware NAS

• Hardware-aware NAS is a multi-objective search that integrates device-level constraints such as latency, energy, and memory into neural network design.
• It employs methodologies like differentiable NAS, evolutionary algorithms, and analytical cost models to balance accuracy with hardware efficiency.
• The framework supports joint optimization of neural architectures and hardware configurations for efficient deployment on resource-constrained platforms.

Hardware-aware Network Architecture Search (NAS) encompasses algorithmic frameworks and methodologies that explicitly integrate hardware cost, resource constraints, and device-specific realism as first-class objectives in automated neural network design. Unlike standard accuracy-directed NAS—which often results in impractically large or slow architectures—hardware-aware NAS directly incorporates latency, energy, throughput, or memory constraints measured or modeled for a target platform, producing architectures that are not only accurate but operationally efficient or deployable on resource-constrained systems.

1. Motivations and Conceptual Framework

Deploying deep neural networks on edge devices, FPGAs, CPUs, GPUs, or custom ASICs requires not only high predictive accuracy but strict adherence to hardware budgets such as run-time latency, energy/power, or memory footprint. Conventional NAS techniques, which optimize only for accuracy, routinely generate architectures with excessive computational or memory demands when mapped onto real hardware (Benmeziane et al., 2021). This mismatch motivates hardware-aware NAS: the augmentation or reformulation of NAS objectives to incorporate device-driven metrics. In formal terms, hardware-aware NAS treats the architecture search as a multi-objective or constrained optimization problem over a joint architecture space $\mathcal{A}$ and sometimes a hardware design space $\mathcal{H}$, usually seeking to maximize accuracy and hardware efficiency or satisfy explicit cost thresholds (Jiang et al., 2019, Jiang et al., 2019).

Hardware cost metrics are diverse, with latency and energy as primary examples; additional constraints may include peak memory, area, or throughput. Critically, indirect proxies (FLOPs, parameter count) are shown to correlate poorly with real-device behavior, necessitating empirical, analytical, or learned cost models linked to actual deployment targets (Li et al., 2021).

2. Search Space and Objective Formulation

2.1 Architecture and Hardware Spaces

Typical hardware-aware NAS defines the overall space as a joint product of network configuration choices (operators, depths, widths, skip connections, kernel sizes, etc.) and, for co-design approaches, hardware mapping variables (pipeline partitions, PE allocations, quantization policies) (Jiang et al., 2019). Formally, for architecture $\alpha \in \mathcal{A}$ and hardware $h \in \mathcal{H}$: $$\max_{(\alpha, h) \in \mathcal{A} \times \mathcal{H}} \Bigl[ \operatorname{Acc}(\alpha),\; \operatorname{Eff}(h, \alpha) \Bigr] \;\text{s.t.}\; \operatorname{TP}(h, \alpha) \geq \operatorname{TS}$$ where $\operatorname{Acc}(\alpha)$ is validation accuracy and $\operatorname{Eff}(h, \alpha)$ is a composite hardware efficiency, often weighted over pipeline utilization and energy (Jiang et al., 2019).

For hardware-only aware NAS with fixed hardware, the search is bi-objective or constrained: $$\max_{\alpha \in \mathcal{A}} \operatorname{Acc}(\alpha) \quad \text{s.t.} \quad \operatorname{Cost}(\alpha) \leq C_{\rm max}$$ or equivalently, a scalarized objective: $$L(\alpha, w) = L_{\rm acc}(w, \alpha) + \lambda \cdot L_{\operatorname{hw}}(\alpha)$$ with $\lambda$ controlling hardware regularization strength (Cai et al., 2018, Chiang et al., 2024).

2.2 Joint Optimization and Pareto Frontier

Approaches vary between scalarizing objectives (using a linear or nonlinear combination with hyperparameters $\mathcal{H}$0) and explicitly returning Pareto-optimal sets, providing flexible trade-off points between accuracy and hardware cost. Pareto filtering is the norm in evolutionary approaches, while gradient-based methods typically yield one solution per run per weight (Bouzidi et al., 2024, Sinha et al., 2024).

3. Methodological Strategies

3.1 Differentiable NAS

Methods such as ProxylessNAS and FBNet employ continuous relaxation over discrete operator selection, enabling direct backpropagation through both architecture parameters and hardware metrics (Cai et al., 2018, Srinivas et al., 2019). Each operation $\mathcal{H}$1 is associated with a learned weight $\mathcal{H}$2, and the expected hardware cost (e.g., latency) is computed via

$\mathcal{H}$3

with $\mathcal{H}$4 softmax probabilities and $\mathcal{H}$5 the profiled latency for operator $\mathcal{H}$6. The loss function incorporates accuracy and hardware metrics as differentiable surrogates, supporting direct hardware-aware search (Cai et al., 2018, Xu et al., 2023).

3.2 Evolutionary and Population-based Search

Evolutionary algorithms (EAs) such as NSGA-II dominate multi-objective hardware-aware NAS, maintaining diverse populations and selection via Pareto dominance (Sinha et al., 2024, Bouzidi et al., 2024). Candidate architectures are encoded as discrete strings/vectors; at each generation, mutation and crossover generate new candidates evaluated for accuracy (or proxy) and hardware cost. Surrogate models or zero-cost predictors accelerate evaluation by predicting accuracy (e.g., via representation similarity metrics or statistical proxies) (Sinha et al., 2023, Zhu et al., 1 Oct 2025, Sinha et al., 2024).

Recent advances include self-adaptive evolutionary operators (SONATA), where parameter importance is inferred via tree-based surrogates and reinforced via policy learning, increasing search efficiency and convergence (Bouzidi et al., 2024).

3.3 Hardware/Software Co-Exploration

Full co-design approaches simultaneously optimize both hardware mappings (e.g., pipeline stage division, PE assignment) and neural architectures. Fast hardware-centric exploration prunes infeasible architectures; slower accuracy-driven controller updates proceed on the remaining candidates (Jiang et al., 2019). Integration with analytical performance models or simulators allows for constraint-aware early pruning in the search loop.

3.4 LLM-driven and Prompt-based NAS

Recently, LLMs have been used to generate candidate architectures by co-evolution with prompt engineering, partitioning the search space to avoid mode collapse, and updating design heuristics with feedback (Zhu et al., 1 Oct 2025). Zero-cost predictors or accuracy proxies (e.g., synflow, Fisher) enable rapid, large-scale exploration at minimal computational cost.

4. Hardware Cost Modeling and Prediction

4.1 Empirical, Analytical, and Learned Predictors

Direct measurement on target devices provides the gold standard for hardware metrics but is costly per evaluation (Li et al., 2021). Analytical models—based on operator-level timing or architectural simulation (e.g., layer latency LUTs, pipelining, quantization-aware modeling)—enable rapid estimation (Jiang et al., 2019, Zniber et al., 4 Dec 2025). Learned predictors (MLPs, GCNs, or Transformer encoders) trained on sampled architectures and profiling data provide fast, generalizable surrogates for latency and energy (Cai et al., 2018, Lee et al., 2021, Mih et al., 17 Feb 2026).

Advanced frameworks use meta-learning to enable cross-device latency prediction with only a few device-specific measurements, via meta-learned hardware embeddings or "signatures" (Lee et al., 2021). When hardware cost rankings are monotonic across devices, a proxy device and adaptation suffice for multi-target NAS, collapsing the need for per-device re-prediction (Lu et al., 2021).

4.2 Quantization and Mixed Precision

Quantization-aware NAS integrates hardware cost and accuracy trade-offs under reduced-precision arithmetic, searching over mixed-precision bitwidths and their placement. Energy, latency, and accuracy models are profiled for each quantization configuration, and quantization-aware training ensures fidelity under hardware constraints (Zniber et al., 4 Dec 2025, Kim et al., 2020).

5. Benchmarking, Datasets, and Reproducibility

Large public benchmarks such as HW-NAS-Bench provide ground-truth vision benchmarks (NAS-Bench-201, FBNet) across six hardware targets (commercial edge, FPGA, ASIC) with per-architecture measurements for accuracy, latency, and energy (Li et al., 2021). Lookup APIs facilitate cost evaluation for arbitrary architectures within the search space, supporting search algorithm benchmarking, fair comparisons, and reproducibility independent of hardware expertise.

Evaluation metrics include accuracy-cost Pareto fronts, hypervolume, inverted generational distance, and dominance ratio (Sinha et al., 2024, Zhu et al., 1 Oct 2025, Bouzidi et al., 2024). Analysis of cross-device correlation uncovers large Pareto front shifts between devices, reinforcing the need for device-specific search (Li et al., 2021).

6. Recent Innovations and Extensions

6.1 Hybrid Network and Transformer Search

Unified search spaces combine CNN, self-attention, convolutional, and activation choices for hybrid backbones (e.g., SCAN-Edge), leveraging block-wise device-specific profiling and evolutionary adaptation to match hardware best practices (Chiang et al., 2024).

Hardware-aware NAS has been adopted for tasks beyond classification, including super-resolution, early-exiting network design for adaptive inference, few-shot learning with meta-learning (H-Meta-NAS), and Transformer architecture specialization (Xu et al., 2023, Zniber et al., 4 Dec 2025, Zhao et al., 2021).

6.2 Multi-Objective and Diversity-Driven Search

State-of-the-art NAS frameworks seek to maximize not only accuracy and hardware efficiency but also solution diversity, ensuring broad Pareto coverage and enabling system-level choices post-search. Objectives for diversity encourage search algorithms not to collapse into a narrow cost-accuracy region (Sinha et al., 2024).

6.3 Zero-Cost and Representation Proxy-driven Methods

Accuracy proxies based on representation similarity to a high-accuracy reference model (mutual information, CKA) enable rapid and training-light accuracy estimation, permitting deployment of hardware-aware NAS on large-scale spaces with extremely low compute cost (Sinha et al., 2023).

6.4 Prompt-Driven, LLM-Co-Evolutionary NAS

Prompt-based LLM-driven approaches partition the search space according to measurable complexity (e.g., number of $\mathcal{H}$7 convolutions), enforce exploration within each niche, and merge Pareto fronts across partitions, achieving drastically reduced search cost with competitive accuracy-latency trade-offs (Zhu et al., 1 Oct 2025).

7. Challenges, Limitations, and Future Directions

7.1 Cost Modeling and Transferability

Accurate and scalable device-specific hardware cost prediction remains a persistent challenge. Proxies such as FLOPs or parameter count are poor surrogates, and high-fidelity lookup tables or predictors must be updated per device, compiler version, or operator fusion (Li et al., 2021, Sinha et al., 2023). Emerging solutions include meta-learning for cross-device hardware prediction (Lee et al., 2021), search-agnostic evaluators via LLMs (Mih et al., 17 Feb 2026), and robust adaptation strategies (Lu et al., 2021).

7.2 Search Efficiency and Scalability

Full-supernet or on-the-fly hardware measurement approaches incur high compute and wall-clock expense. Recent advances exploit zero-cost predictors, parameter-importance gating, LLM joint prompt evolution, and diversity-driven objectives to speed up search and improve coverage (Zhu et al., 1 Oct 2025, Bouzidi et al., 2024, Sinha et al., 2024).

7.3 Co-Exploration and Platform Generalization

There are open directions for unified search across both neural architectures and hardware mapping/configuration space (e.g., FPGA pipeline partitioning, ASIC resource allocation) (Jiang et al., 2019). The co-design approach remains relatively unexplored beyond FPGAs, and generalizing to simultaneous multi-platform deployment is a growing area of research focus.

7.4 Emerging Tasks and Constraints

Extensions to detection, segmentation, transformer architectures, security, fault tolerance, memory bandwidth, mixed-precision inference, and dynamic hardware adaptation are rapidly gaining traction. Integrating energy, robustness, and memory constraints as first-class primitives in NAS objectives remains an active research area (Zniber et al., 4 Dec 2025, Zhao et al., 2021, Sinha et al., 2024).

7.5 Reproducibility and Standardization

Benchmark suites such as HW-NAS-Bench democratize research by removing the hardware-reproducibility bottleneck, but discrepancies in datasets, training protocols, hardware target selection, and metric definitions remain a significant obstacle to comprehensive comparisons (Li et al., 2021).

---

In summary, hardware-aware NAS now comprises a mature, technically diverse field characterized by joint accuracy-hardware trade-off optimization, innovative predictor and search methodologies, and a growing set of reproducible benchmarks and tools. Success in reducing compute cost, improving Pareto frontier exploration, and bridging theoretical–real device gaps positions hardware-aware NAS as central to deployable AI across rapidly diversifying device landscapes (Benmeziane et al., 2021, Li et al., 2021, Cai et al., 2018, Jiang et al., 2019).