Zero-Cost Proxies in NAS

• Zero-cost proxies are scalar surrogates computed on untrained neural architectures to predict future test accuracy or robustness with a single forward/backward pass.
• They combine diverse metrics like Jacobian, pruning, expressivity, and topology, and have evolved to include learned and automated proxy formulas for improved fidelity.
• Integrating proxy ensembles in NAS pipelines accelerates search efficiency by up to 10¹–10⁴× while supporting robust and multi-modal evaluations.

Zero-cost proxies (ZCPs) are scalar surrogates computed on untrained neural network architectures to predict their downstream performance after full training. These proxies, typically derived via a single forward and/or backward pass on a randomly initialized network, enable highly efficient neural architecture search (NAS) by dramatically reducing the computational cost of evaluating candidate architectures. Modern ZCPs span Jacobian-based, pruning-based, expressivity-based, parameter-statistics, and topology-driven metrics. They have evolved from heuristic, manually engineered quantities to parameterized, learned, and even automatically discovered formulas, with increasing empirical fidelity for both clean accuracy and robustness. Recent advances leverage ensembles, symbolic search, and graph- or operator-embeddings, extending ZCPs to convolutional, transformer, and multi-modal search spaces.

1. Formal Definitions and Classes of Zero-Cost Proxies

A zero-cost proxy is a function $s = f(\mathcal{A})$, where $\mathcal{A}$ is a neural architecture (possibly including its DAG specification, weights, activations, and statistics from one initialization or synthetic input), and $f$ returns a scalar that aims to predict future test accuracy or robustness of $\mathcal{A}$. Table 1 provides core classes and representative mathematical definitions:

Type	Formula (at initialization)	Example Proxy
Jacobian-based	E.g., $$\text{jacov} = D_{\mathrm{KL}}(\mathcal{N}(0,I)\|\mathcal{N}(0,K))$$ with $$K_{ij}=\langle\nabla_x f_{\theta}(x_i), \nabla_x f_{\theta}(x_j)\rangle$$	jacov, nwot, epe-nas
Pruning-based	SNIP: $$\sum_i \left|\frac{\partial J}{\partial \theta_i} \cdot \theta_i\right|$$	SNIP, GraSP, SynFlow
Piecewise-linear	Zen-score: Gaussian complexity of piecewise linear regions	Zen-score
Hessian-based	$\lambda_{\max}$ of $\nabla^2_{\theta} J$	λ_max
Baselines	FLOPs, $\|\theta\|_2$, #params	params, flops
Topology-based	SED: entropy-based score from architecture graph	SED

Beyond these, sophisticated learned and parameterized ZCPs (e.g., ParZC (Dong et al., 2024), TG-NAS (Qiao et al., 2024)) integrate node-level or operator-level uncertainty, while automated search by symbolic evolution (EZNAS (Akhauri et al., 2022), GreenMachine (Cortês et al., 2024), LPZero (Dong et al., 2024)) directly emits analytics proxies optimized for maximum ranking correlation.

2. Calculation Protocols and Benchmark Evaluation

Zero-cost proxies are computed on the untrained network, generally with a single mini-batch (real or synthetic data, Gaussian, or even no data for topology-only proxies). Computational complexity is typically that of a single forward/backward pass ($\mathcal{A}$0second per candidate on GPU), except for topological proxies (e.g., SED (Wu et al., 2024)), which require no execution at all. Evaluation typically measures proxy performance by the rank correlation (Kendall’s $\mathcal{A}$1, Spearman’s $\mathcal{A}$2), precision@k (top-k identification), and sometimes regression $\mathcal{A}$3 vs. ground truth accuracy or robustness across entire NAS benchmarks such as NAS-Bench-101, 201, NDS, DARTS, and AutoFormula search spaces (Krishnakumar et al., 2022, Lukasik et al., 2023).

Proxies are quantifiably validated by their ability to:

• Predict clean accuracy (standard test accuracy after full training),
• Predict adversarial or corruption robustness (post-attack accuracy, e.g., FGSM, PGD, Square attack),
• Scale across search space types (cell-based, macro, vision transformer, NLP models).

3. Empirical Performance, Limitations, and Complementarity

Proxies such as jacov, nwot, SynFlow, and SNIP routinely achieve moderate to high global correlation with test accuracy on small to medium cell-based benchmarks: e.g., the best single ZCPs can reach Spearman $\mathcal{A}$4 and Kendall $\mathcal{A}$5 on NAS-Bench-201 (Abdelfattah et al., 2021, Lukasik et al., 2023, Krishnakumar et al., 2022). On larger or more diversified spaces, their correlations can degrade, especially when the architectures are structurally similar or the search space encompasses multiple modalities (Kadlecová et al., 2024, Akhauri et al., 2022).

Robustness prediction (performance under adversarial attacks) is substantially more challenging. While clean accuracy can often be regressed from a single ZCP, adversarial robustness ranking typically requires an ensemble of proxies. For example, in NAS-Bench-201, single ZCPs can yield negative $\mathcal{A}$6 for APGD robustness, whereas a random forest on all proxies raises robust accuracy $\mathcal{A}$7 to 0.65–0.88 ((Lukasik et al., 2023), Table 1).

Analyses consistently show that the information carried by different ZCPs is complementary. An information-theoretic analysis (NAS-Bench-Suite-Zero (Krishnakumar et al., 2022)) reveals up to 1–1.5 bits of mutual information gain from combining several proxies. Ensemble usage (feature concatenation in a tree or GNN-based surrogate) enhances both ranking accuracy and sample efficiency by up to 42% for NAS pipelines (Krishnakumar et al., 2022).

4. Automated and Parameterized Proxy Design

Recent methods transcend manual design by automatic search and learning:

• Parametric ZCPs (ParZC (Dong et al., 2024)): Parameterize the aggregation of node-level ZCP statistics via a Bayesian mixer with uncertainty modeling, using a differentiable Kendall’s tau (DiffKendall) loss to directly optimize the ranking induced by the proxy.
• Genetic Programming/Evolution (EZNAS (Akhauri et al., 2022), GreenMachine (Cortês et al., 2024), LPZero (Dong et al., 2024)): Evolve symbolic or algebraic proxies as syntax trees over primitives such as layer activations, gradients, and statistics. Fitness is defined by rank correlation with ground-truth accuracy over sampled architecture sets.
• Graph and Operator Embedding Models (TG-NAS (Qiao et al., 2024)): Employ operator-level semantic embeddings via Transformers (MiniLM) and learn a prediction function (GCN) on the architectural DAG, providing generalization to new primitives.
• Topology-Driven ZCPs (SED (Wu et al., 2024)): Use only the architecture’s graph/metainformation—with analytic entropy-based measures—to score architectures without any weight instantiation, achieving orders-of-magnitude faster proxy evaluation.

Such advances enable adaptation to new search spaces (e.g., vision transformer, NLP), robustness targets, and multi-objective scenarios.

5. Best Practices and Integration into NAS Workflows

Research and benchmarking consistently advocate the following:

• Use an ensemble of diverse ZCPs, especially Jacobian-based and pruning-based measures, as input to a nonlinear or tree-based surrogate (e.g., random forest, XGBoost) for high-fidelity performance prediction (Lukasik et al., 2023, Krishnakumar et al., 2022).
• For clean-accuracy-only NAS, a single well-correlated ZCP (e.g., jacov, SynFlow, SWAP) is sufficient for rapid candidate downselection (Abdelfattah et al., 2021, Peng et al., 2024).
• For robustness-aware or multi-objective NAS, multi-proxy ensembles are essential; corroborated by robustness R² (FGSM/APGD) rising from near-zero for single ZCPs to 0.65–0.88 when ZCPs are combined (Lukasik et al., 2023).
• Best-practice integration includes: (1) scoring large candidate pools by ZCP ensemble/surrogate, (2) using proxy-driven warmup for evolutionary or RL-based NAS, and (3) replacing or augmenting training-based predictors in Bayesian optimization, where ZCPs are especially impactful during early (data-scarce) NAS iterations (Shen et al., 2021, Krishnakumar et al., 2022).

Proxy cost reduction yields search speed-ups of $\mathcal{A}$8 compared to reduced-training or supernet-based proxies, democratizing NAS for resource-constrained or green-AI scenarios (Cortês et al., 2024, Abdelfattah et al., 2021).

6. Robustness, Generalization, and Emerging Directions

Zero-cost proxies remain imperfect surrogates for full evaluation, with several open challenges:

• Biases and Limitations: Many ZCPs are biased by architecture size, convolution count, or superficial graph properties, limiting intra-cluster discrimination (Kadlecová et al., 2024). Simple normalization schemes (e.g., dividing by parameter count plus constant) can reduce bias (Krishnakumar et al., 2022).
• Robustness: Prediction of adversarial or corruption robustness is difficult; requires combination of feature-, parameter-, and gradient-consistency proxies (e.g., CRoZe (Ha et al., 2023), multi-proxy regression (Lukasik et al., 2023)).
• Generalization: Most hand-designed ZCPs degrade when transferred to structurally novel or non-convolutional search spaces (ViT, NLP), motivating learned approaches (ParZC, L-SWAG, LPZero) (Dong et al., 2024, Casarin et al., 12 May 2025, Dong et al., 2024).
• Automated Proxy Design: Symbolic regression and genetic programming show state-of-the-art cross-dataset ranking performance, reducing reliance on heuristic design (Akhauri et al., 2022, Cortês et al., 2024).
• Operator and Graph Encoding: Operator embedding and graph-based features (GRAF (Kadlecová et al., 2024), TG-NAS (Qiao et al., 2024)) can encode topological and semantic properties unavailable to classic ZCPs, further boosting performance prediction when combined.

7. Impact and Prospects

Zero-cost proxies have transformed NAS by enabling massive architecture screening at negligible cost. Composite, automatically discovered, uncertainty-aware, and graph-operator-embedded proxies now regularly surpass handcrafted metrics, generalizing across architectures and tasks. Their integration into Bayesian, evolutionary, and differentiable NAS continues to shrink search latency, energy, and environmental impact, supporting scalable, robust, and sustainable pipeline design (Cortês et al., 2024, Lukasik et al., 2023). Future directions include multi-objective and constraint-aware design, formal theory connecting proxy metrics with optimization-dynamics and generalization, and adaptive proxy ensembles tailored to novel modalities and search conditions.