Activation-aware Score Function

• Activation-aware Score Function is a zero-cost proxy metric that quantifies a network’s ability to separate inputs and its activation capacity using binary ReLU masks.
• It decouples traditional evaluation by leveraging the log-determinant of a normalized Hamming kernel for distinguishing ability and an activation count penalty for capacity.
• The method is enhanced by a fast training strategy that boosts ranking accuracy and has been validated across multiple NAS benchmarks.

An activation-aware score function is a class of zero-cost proxy metrics in neural architecture search (NAS) that evaluates candidate networks by directly analyzing their layer activations, quantifying both their ability to separate inputs (“distinguishing ability”) and their activation capacity, without the need for full training. The Distinguishing Activation Score (DAS) is the canonical instance of this approach, explicitly formulated to address non-atomicity in earlier activation-based proxies. DAS is defined as a function of the log-determinant of a Hamming kernel on binary ReLU activation masks, combined with an explicit activation-count penalty weighted by a tunable hyperparameter. This construction yields a lightweight, interpretable measure that correlates strongly with downstream trained accuracy and is further enhanced by a fast-training augmentation that requires only minimal epochs of partial overfitting to a single minibatch (Liu et al., 2022).

1. Foundations and Motivation

The high computational burden of evaluating candidate architectures has historically impeded the scalability and adoption of NAS. Zero-cost proxies, which attempt to infer likely post-training test accuracy from untrained or only partially trained model properties, offer a remedy. Traditional gradient-based and saliency-based metrics often lack specificity to the inductive properties of individual architectures. Activation-aware approaches, in contrast, exploit the behavior of binary ReLU activation patterns across input batches, positing a direct link between an architecture’s ability to produce diverse activation responses and its eventual discriminative capacity after training.

The original “without training” (WOT) score encapsulated this principle by computing the log-determinant of a Hamming-similarity kernel between activation masks. However, WOT entangled several underlying factors, motivating its decoupling into atomic metrics that reveal and separately measure input-distinguishability and activation unit capacity (Liu et al., 2022).

2. Atomic Metrics: Distinguishing Ability and Activation Capacity

Let $A$ denote a candidate convolutional-ReLU architecture, and $X=\{x_1, ..., x_N\}$ a sampled minibatch. For each $x_i$, the binary activation mask $c_i \in \{0,1\}^{N_A}$ records the activation state of each ReLU unit. The pairwise Hamming kernel is given by:

$K_H(i,j) = N_A - d_H(c_i, c_j)$

where $d_H$ denotes Hamming distance.

The decomposed atomic metrics are:

• Distinguishing-ability metric ($S_\text{dist}$): Normalizing $K_H$ by $N_A$ removes scale dependence,

$$NK_H = K_H / N_A = [1 - d_H(c_i, c_j)/N_A]_{i,j=1}^N$$

The metric is then $X=\{x_1, ..., x_N\}$0, corresponding to the separability of activation patterns across inputs.

• Activation-unit count ($X=\{x_1, ..., x_N\}$1): The total number of ReLU units in the network, $X=\{x_1, ..., x_N\}$2. This term, commonly included in logarithmic form, proxies network capacity and representational richness (Liu et al., 2022).

3. Distinguishing Activation Score (DAS): Formulation and Decoupling

The distinguishing activation score (DAS) is defined by a linear composition of the above atomic metrics:

$X=\{x_1, ..., x_N\}$3

where $X=\{x_1, ..., x_N\}$4 is a tunable coefficient, with $X=\{x_1, ..., x_N\}$5 (where $X=\{x_1, ..., x_N\}$6 is the batch size) providing robust performance across evaluated search spaces.

Theoretical analysis shows that WOT’s original score,

$X=\{x_1, ..., x_N\}$7

decomposes as

$X=\{x_1, ..., x_N\}$8

revealing that WOT implicitly fixes $X=\{x_1, ..., x_N\}$9. DAS generalizes this by allowing $x_i$0 to be chosen or tuned, thereby exposing and correcting for WOT’s conflation of activation count and input separability (Liu et al., 2022).

Both $x_i$1 and $x_i$2 exhibit positive empirical correlation with post-training held-out accuracy. $x_i$3 reflects intrinsic discriminative properties, while $x_i$4 tracks parameter budget and latent representational power.

4. Fast Training Strategy for Enhanced Proxy Accuracy

Untrained activation metrics, while computationally efficient, may not robustly reflect the trained generalization accuracy. Full model training, although accurate, is impractically expensive for NAS. The “fast training” strategy interpolates between these extremes by briefly overfitting the candidate architecture to a fixed minibatch-label pair $x_i$5 for a small number $x_i$6 of epochs, recording activation masks post-adaptation, and then recalculating DAS.

Algorithmically:

1. Randomly initialize network weights.
2. For each epoch $x_i$7, run forward/backward passes on $x_i$8 and update weights (e.g., via SGD).
3. After $x_i$9 epochs, extract binary activation masks from the adapted model.
4. Compute DAS with the new activations.

Empirically, $c_i \in \{0,1\}^{N_A}$0 in $c_i \in \{0,1\}^{N_A}$1 suffices for significant improvement in ranking fidelity (Kendall’s $c_i \in \{0,1\}^{N_A}$2), with $c_i \in \{0,1\}^{N_A}$3 capturing most obtainable benefit. This procedure increases compute cost over pure zero-training proxies but remains orders of magnitude cheaper than full training (e.g., $c_i \in \{0,1\}^{N_A}$4 more accurate with $c_i \in \{0,1\}^{N_A}$5 of the full cost) (Liu et al., 2022).

5. DARTS-Training-Bench (DTB): Dataset for Proxy Evaluation

The DARTS-Training-Bench (DTB) dataset constitutes a benchmark for zero-cost and partial-training proxy metrics. DTB features:

• 500 architectures sampled from the DARTS cell search space.
• For each architecture, snapshots of weights and binary activation patterns at epochs 0, 1, 2, ..., 5 of training (on CIFAR-10).
• Final test accuracy for each model after $c_i \in \{0,1\}^{N_A}$6 epochs of conventional training.

DTB uniquely enables quantitative study of how partial adaptation (few-shot training) impacts the predictive validity of zero-cost proxies, addressing the limitation in existing NAS-Bench datasets which only report fully trained outcomes (Liu et al., 2022).

6. Empirical Evaluation and Practical Implications

Evaluation across seven standard NAS search spaces, including NAS-Bench-101, Network Design Spaces (NDS-PNAS, NDS-ENAS, NDS-DARTS, NDS-Amoeba, NDS-NASNet), and DTB, demonstrates that DAS consistently improves both raw evaluation accuracy (as measured by Kendall’s $c_i \in \{0,1\}^{N_A}$7) and final searched architecture performance. Gains with DAS+fast training over pure WOT range from $c_i \in \{0,1\}^{N_A}$8 to $c_i \in \{0,1\}^{N_A}$9 in accuracy ranking.

Example outcomes for NAS-Bench-101:

Method	Cost (s)	Accuracy (%)
Random	—	$K_H(i,j) = N_A - d_H(c_i, c_j)$0
NASWOT ($K_H(i,j) = N_A - d_H(c_i, c_j)$1)	$K_H(i,j) = N_A - d_H(c_i, c_j)$2	$K_H(i,j) = N_A - d_H(c_i, c_j)$3
DAS ($K_H(i,j) = N_A - d_H(c_i, c_j)$4)	$K_H(i,j) = N_A - d_H(c_i, c_j)$5	$K_H(i,j) = N_A - d_H(c_i, c_j)$6
DAS ($K_H(i,j) = N_A - d_H(c_i, c_j)$7)	$K_H(i,j) = N_A - d_H(c_i, c_j)$8	$K_H(i,j) = N_A - d_H(c_i, c_j)$9

DAS with fast training outperforms prior activation-based proxies even when evaluated with significantly fewer architecture samples. The gain is consistent across search spaces, with especially pronounced relative improvement observed for more complex or less regular architectures (Liu et al., 2022).

7. Limitations, Parameterization, and Extensions

• The optimal choice of $d_H$0 is empirically sensitive to batch size, search space, and label distribution; practical default is $d_H$1, but grid-search over a hold-out set is recommended for maximal fidelity.
• Fast training entails modest additional compute; the trade-off between $d_H$2 and ranking precision should be adjusted to compute budget.
• Additional atomic metrics (e.g., layer count, synflow-based saliency) could be integrated into the DAS framework, though new combination weights would need to be calibrated.
• The method’s interpretability derives from its explicit decoupling: DAS exposes the contributions of separability and activation count, informing both proxy and architecture design (Liu et al., 2022).

In summary, the activation-aware score function, and specifically the distinguishing activation score (DAS), provides a robust, interpretable, and empirically validated proxy for neural architecture evaluation in NAS. Its formulation as a composite of input separability (via ReLU activation diversity) and activation capacity offers natural guidance for further research on zero-cost proxies and efficient NAS pipelines.