GraB-NAS: Hybrid Meta Neural Architecture Search

• GraB-NAS is a meta neural architecture search framework that models architectures as graphs and employs Bayesian optimization for global exploration.
• It integrates a gated message-passing scheme with latent gradient ascent to refine continuous embeddings and generate novel designs.
• Experiments on datasets like CIFAR-10 and CIFAR-100 demonstrate its superior performance and capability to overcome static search space limitations.

GraB-NAS is a Meta Neural Architecture Search (Meta-NAS) framework that integrates graph-based neural architecture representation, deep-kernel Bayesian optimization (BO), and latent space gradient exploration to efficiently discover high-performing neural network architectures across diverse tasks. Addressing the limitations of task-specific NAS and meta-NAS baselines—such as poor generalization, restricted search spaces, and high computational costs—GraB-NAS leverages a hybrid search algorithm capable of both global and local exploration, enabling the generation and identification of novel architectures with strong out-of-distribution performance (Sun et al., 13 Aug 2025).

1. Graph-based Neural Architecture Representations

GraB-NAS models neural architectures as directed acyclic graphs (DAGs) $G = (V, E)$, where:

• $V$ constitutes nodes, each representing a computational operation (e.g., convolution, skip connection) encoded via a one-hot vector $o_v$ for node $v \in V$;
• $E \subseteq V \times V$ is the set of directed edges $(u \rightarrow v)$ signifying data flow from node $u$ to node $v$.

The architecture graph is encoded using a gated message-passing scheme inspired by D-VAE: each node $v$ aggregates incoming messages via

$h^\text{in}_v = \sum_{u \to v} g(h_u) \odot m(h_u)$

where $V$0 is the hidden state of node $V$1, $V$2 and $V$3 are learned MLPs, and $V$4 denotes elementwise multiplication. The node states are then updated using a GRU,

$V$5

After a full forward-backward pass, the concatenated hidden states are projected to a continuous embedding $V$6.

Dataset information is summarized by a dataset encoder $V$7, producing an embedding $V$8 through two stacked Set Transformer modules (providing per-class prototype extraction and dataset-level pooling). For each dataset–architecture pair, a fused representation $V$9 is constructed.

2. Global Search Using Bayesian Optimization

GraB-NAS utilizes Bayesian Optimization (BO) in the joint dataset–architecture embedding space to guide global exploration:

• A deep-kernel Gaussian Process (GP) surrogate models the black-box mapping $o_v$0 true test accuracy, with prior

$o_v$1

where $o_v$2 is a kernel such as Matérn.

• Predictive posterior at a candidate $o_v$3 is given as

$o_v$4

with GP kernel matrix $o_v$5 computed over observed embeddings and $o_v$6 the associated accuracies.

• The acquisition function is Expected Improvement (EI),

$o_v$7

where $o_v$8, $o_v$9 is the standard normal CDF, and $v \in V$0 is its PDF. The next candidate is selected by maximizing EI among novel graphs. Its true accuracy is measured and appended to the support set.

3. Local Exploration by Gradient Ascent in Latent Space

After a BO warm-up phase of $v \in V$1 iterations, GraB-NAS incorporates local latent-space search:

• The support-set architecture $v \in V$2 with highest observed accuracy $v \in V$3 is chosen.
• Its graph embedding $v \in V$4 is updated by a gradient ascent step on the GP posterior mean:

$v \in V$5

with $v \in V$6 a step size.

• This perturbed embedding is decoded via a learned graph decoder $v \in V$7 to yield a potentially novel architecture $v \in V$8. If $v \in V$9 is not in the support set, it is evaluated and added.

The optimization objective for this stage is

$E \subseteq V \times V$0

maximized with respect to the graph-latent code $E \subseteq V \times V$1.

4. Hybrid Search Algorithm

The method alternates between global BO-guided search and local gradient-driven refinement over $E \subseteq V \times V$2 iterations:

• For each iteration $E \subseteq V \times V$3:
  • Update the GP posterior with support set $E \subseteq V \times V$4.
  • Perform a global selection using EI as the acquisition criterion.
  • If $E \subseteq V \times V$5, initiate a local refinement step on the best $E \subseteq V \times V$6 as described.

This hybridization exploits BO's exploration–exploitation tradeoff while enabling rapid convergence and discovery of architectures beyond the confines of the original search space. Decoding perturbed latent codes yields graphs that are not restricted to the original NAS-Bench-201 candidate pool.

5. Experimental Protocol and Performance

The experimental framework consists of:

• Search Space: NAS-Bench-201, modeling each cell as a 4-node DAG with edge operations in {zeroize, skip, $E \subseteq V \times V$7 conv, $E \subseteq V \times V$8 conv, $E \subseteq V \times V$9 avg pooling}. Networks are composed by stacking cells.
• Meta-training: Conducted over random subsets of ImageNet-1K.
• Meta-testing: Evaluated on previously unseen datasets, including CIFAR-10, CIFAR-100, MNIST, SVHN, FGVC Aircraft, and Oxford-Pets. All models are trained from scratch using the established NAS-Bench-201 protocols.
• Baselines: Random Search, one-shot NAS (GDAS, SETN, PC-DARTS, DrNAS), and Meta-NAS methods (MetaD2A, TNAS).

GraB-NAS achieves test accuracies (mean $(u \rightarrow v)$0 std over three runs): 94.37 on CIFAR-10 (matching MetaD2A), 73.51 on CIFAR-100 (TNAS: 73.02), 99.78 on MNIST, 96.64 on SVHN, 58.87 on Aircraft (TNAS: 56.14, $(u \rightarrow v)$1), and 43.15 on Pets (TNAS: 39.12, $(u \rightarrow v)$2). Anytime-performance analysis demonstrates more rapid ascent to high accuracy compared with baselines. Rank heatmaps indicate GraB-NAS holds first place across all six benchmarks (Sun et al., 13 Aug 2025).

6. Ablation and Analysis

Ablation studies dissect the contributions of components:

• Graph-decoder ablation: Replacing the decoder with a $(u \rightarrow v)$3-nearest neighbor lookup in the latent space ("GraB-NAS w/ KNN") already surpasses TNAS, evidencing the utility of gradient-tunable latent codes. Integrating the full graph decoder ("GraB-NAS w/ decoder") further improves results, allowing the discovery of architectures outside the enumerated search set (e.g., CIFAR-100: 73.43 using KNN vs 73.51 with graph decoder).
• Pruned space ablation: Removal of top-$(u \rightarrow v)$4 architectures from NAS-Bench-201 constrains competing methods—TNAS can at best identify the remaining optimum. GraB-NAS, by contrast, generates novel candidates via its graph decoder and exceeds the pruned search space's maximum test accuracy (e.g., $(u \rightarrow v)$51.08% on CIFAR-100 after removing the top-50 candidates).

7. Distinctive Features and Impact

GraB-NAS advances Meta-NAS by:

• Modeling architectures as continuous graph embeddings, facilitating differentiable search and enabling efficient navigation of combinatorial search spaces;
• Unifying BO's global modeling power with gradient ascent for local improvement, overcoming the limitations of finite candidate enumeration;
• Enabling the out-of-distribution generation of architectures tailored to new tasks, yielding consistently strong generalization and empirical results;
• Demonstrating, through ablation, that decoder-guided novelty is requisite for surpassing the performance limitations of meta-NAS restricted to static search spaces.

These features position GraB-NAS as an effective approach for task-adaptive neural architecture discovery under realistic meta-learning conditions (Sun et al., 13 Aug 2025).