AnalogNAS-Bench: NAS for AIMC Hardware

• AnalogNAS-Bench is a standardized benchmark suite for evaluating Neural Architecture Search methods tailored to Analog In-Memory Computing hardware with inherent non-idealities.
• It employs a NAS search space inspired by NAS-Bench-201, featuring 15,625 unique architectures evaluated under simulated conditions of programming noise, read noise, quantization, and temporal drift.
• Benchmark results highlight that carefully balanced architectures with specific convolutional and skip connection patterns improve robustness, guiding future analog-aware DNN design.

AnalogNAS-Bench is a standardized, open-source benchmark suite specifically constructed for evaluating Neural Architecture Search (NAS) methodologies under the constraints unique to Analog In-Memory Computing (AIMC) hardware. AIMC accelerates deep neural networks by directly executing matrix–vector multiplications using analog resistive crossbars (e.g., PCM, RRAM), yielding substantial gains in energy efficiency and latency over conventional digital hardware. However, neural networks designed for digital platforms typically do not accommodate the device- and circuit-level non-idealities characteristic of analog hardware, such as non-deterministic programming, read noise, quantization, and temporal drift. AnalogNAS-Bench introduces a systematic platform to compare analog-aware NAS algorithms, scrutinize architectural robustness, and guide analog-centric DNN design, employing a search space and evaluation suite that comprehensively incorporates these non-idealities (Bessalah et al., 23 Jun 2025). All implementations and datasets supporting AnalogNAS-Bench are openly available at https://github.com/[IBM](https://www.emergentmind.com/topics/influence-blocking-maximization-ibm)/analog-nas/tree/main/analognasbench.

1. Motivation and Device Non-Idealities in AIMC

AIMC architectures implement DNN matrix–vector multiplication directly in crossbars of resistive memory cells, eliminating the need for repeated digital memory accesses. While this offers orders-of-magnitude efficiency gains, precision is compromised due to several distinct non-idealities:

• Programming Noise: After writing $W_{ij}$ to a synaptic cell, the actual stored conductance is

$$W^{\rm phys}_{ij} = W_{ij} + \epsilon^{\rm prog}_{ij}, \quad \epsilon^{\rm prog}_{ij} \sim \mathcal{N}(0,\sigma_p^2)$$

• Read Noise: During inference,

$$y = \sum_{i}W^{\rm phys}_{ij}\,x_i + \epsilon^{\rm out}_j, \quad \epsilon^{\rm out}_j \sim \mathcal{N}(0,\sigma_o^2)$$

• Quantization: Finite DAC/ADC resolution leads to

$$\widehat{x} = \mathrm{Q}_{b_{in}}(x), \quad \widehat{y} = \mathrm{Q}_{b_{out}}(y)$$

where $\mathrm{Q}_b$ is uniform $b$-bit quantization.

• Temporal Drift: Device conductances generally drift over time:

$$W^{\rm drift}_{ij}(t) = W^{\rm phys}_{ij}\left(1 + \delta_{ij}(t)\right), \quad \delta_{ij}(t)\sim \mathcal{N}(0,\sigma_d^2(t))$$

with $\sigma_d(t)$ growing (roughly) logarithmically with elapsed time $t$.

These effects, when neglected, can result in significant degradation of inference accuracy. Architectures must therefore be designed and evaluated with explicit awareness of these non-idealities for AIMC deployment (Bessalah et al., 23 Jun 2025).

2. AIMC-Specific NAS Search Space Specification

AnalogNAS-Bench builds upon the cell-based, DAG-structured search space of NAS-Bench-201, but ensures each candidate’s evaluation incorporates all aforementioned AIMC non-idealities. The macro-architecture consists of three “stages” with downsampling; each contains repeated 4-node cell blocks.

Key characteristics of the search space:

• Cell topology: 4-node directed acyclic graphs (nodes 0,1 as inputs, 2,3 as intermediates, node 4 as output summing all predecessors).
• Edge operators (5 choices per edge):

1. Skip connection (identity) 2. Zeroize (constant zero) 3. $3\times3$ convolution 4. $1\times1$ convolution 5. $3\times3$ average-pooling

• Branching and skips: All nodes aggregate from predecessors, supporting multi-path branching.
• Robustness mechanisms: Skip connections (identity path) are notably impactful for resilience under drift and noise.

The combinatorics yield $5^6 = 15\,625$ unique cells/architectures within the fixed macro-skeleton. This search space is small enough for exhaustive benchmarking yet sufficiently expressive to reveal key robustness patterns (Bessalah et al., 23 Jun 2025).

3. Evaluation Protocols and Benchmarking Methodology

AnalogNAS-Bench employs a multi-faceted evaluation pipeline:

• Datasets: CIFAR-10 (fully evaluated); CIFAR-100 and ImageNet16-120 (evaluations ongoing).
• Training: Each of the 15,625 unique architectures is trained for 200 epochs on CIFAR-10 using SGD with momentum, cosine annealing, and standard augmentations.
• Inference and simulation:
  • Baseline full-precision (digital) accuracy.
  • Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT) at 8-bit resolution.
  • “Noisy” analog accuracy: digital weights mapped to AIHWKit’s AIMC simulator incorporating programming/read noise, quantization, and output noise.
  • Hardware-Aware Training (HWT): explicit non-ideality injection during training, evaluated via analog inference.
  • Temporal drift: accuracy monitored at $$t\in\{60\text{\,s}, 1\text{\,h}, 1\text{\,d}, 30\text{\,d}\}$$, using a fixed PCM-like simulation configuration with 8-bit quantization and drift/noise parameters consistent with device models.
• Simulation tools: IBM AIHWKit, supporting device/circuit non-idealities and global drift compensation.

The evaluation suite quantifies digital baseline, quantized, and a hierarchy of hardware-degraded accuracies across the entire architecture set (Bessalah et al., 23 Jun 2025).

4. Core Findings and Architectural Insights

Three principal discoveries emerged from the comprehensive, controlled evaluation under AIMC constraints:

• Limitations of Standard Quantization: PTQ and QAT techniques preserve digital-quantized ranking ($\tau\approx0.95$ with baseline accuracy), but correlate poorly with actual analog accuracy under AIMC non-idealities ($\tau\approx0.34$). This results from quantization noise being nearly architecture-independent, whereas AIMC noise and drift are architecture-sensitive.
• Characteristics of Robust Architectures:
  • Fewer $1\times1$ convolutions: These occupy small crossbar areas and are disproportionately sensitive to noise-induced variance.
  • More $3\times3$ convolutions/average-pooling: Expanded receptive fields and spatial aggregation dilute the effects of device-level noise.
  • Moderate skip connections and branching: 2–3 identity edges plus 2–4 convolution/pooling layers per cell. Graph-theoretic analysis shows that nodes with high degree and convolutional connectivity (especially $3 \times 3 \to \text{skip} \to 3 \times 3$) drive analog robustness.
• Effect of Skip Connections on Temporal Drift: Increased skip connections (identity path count) lead to significantly smaller accuracy drops under drift at long time intervals. Skip paths act as analog “bypasses” for temporally drifting weights, maintaining representational fidelity even as device conductances shift.

These findings directly inform practical analog-aware NAS and motivate explicit inclusion of all non-idealities during architectural search (Bessalah et al., 23 Jun 2025).

5. Recommendations for Analog-Aware NAS and Research Practice

Practitioners and researchers should leverage several key guidelines derived from AnalogNAS-Bench:

• Incorporate all AIMC non-idealities—programming/read noise, quantization, output noise, temporal drift—into the NAS objective or surrogate model.
• Bias the search space: Favor architectures with wider/branched connectivity, moderate skip-connection density, and predominant $3\times3$ convolution.
• Use zero-cost graph metrics (e.g., node degree, recurrent operation patterns) as predictive features for analog robustness, enabling efficient triage of candidate architectures without full noise simulation.

Open-source code, datasets for 15,625 architectures, and simulation/evaluation APIs are available at https://github.com/IBM/analog-nas/tree/main/analognasbench, facilitating direct reuse, reproducibility, and extension to new datasets (e.g., CIFAR-100, ImageNet) and device models (e.g., RRAM, FeRAM), as well as potential adaptation to Transformer cell layouts (Bessalah et al., 23 Jun 2025).

6. Relationship to AnalogNAS Framework and Broader Context

AnalogNAS-Bench extends the methodology of AnalogNAS, a framework for automated analog hardware-aware DNN design, which utilizes a substantially larger ResNet-like search space ($\mathcal{S}$ with $7.3\times 10^{10}$ candidates) and surrogate-assisted evolutionary search to optimize a weighted multi-objective function (accuracy, power, latency, silicon area) under constraints on parameter count and analog drift sensitivity (Benmeziane et al., 2023). AnalogNAS search results have shown superior accuracy and longer-term stability on analog PCM hardware compared to digital-optimized baselines, using the AIHWKit simulator and actual 64-core IMC chips.

By distilling a manageable, well-characterized architectural search space and exhaustively providing analog-aware evaluation, AnalogNAS-Bench offers the scientific community a public standard for benchmarking, reproducibility, and discovery in AIMC-specific NAS. This enables systematic testing of new NAS methodologies and design strategies for robust analog DNNs, fostering transparent comparison and archiving of scientific progress in the analog AI domain (Bessalah et al., 23 Jun 2025, Benmeziane et al., 2023).