---
title: Zero-Cost Proxies for Efficient NAS
url: https://www.emergentmind.com/topics/zero-cost-proxies
type: topic
---

# Zero-Cost Proxies for Efficient NAS

Zero-cost proxies are analytical or computational metrics that estimate the performance of candidate neural network architectures without any gradient-based training or fine-tuning of model parameters. Within Neural Architecture Search (NAS), zero-cost proxies serve as highly efficient surrogates for model evaluation, replacing expensive full or partial training with cheap, architecture-level signals derived from initial network statistics or topology. These proxies are now central to training-free and zero-shot NAS pipelines, as they enable rapid ranking and selection among potentially vast search spaces, while requiring only a single forward or backward pass—or, in some approaches, purely combinatorial analysis of network graphs. Recent research has resulted in a diverse landscape of zero-cost proxies, spanning from expert-designed gradient or activation-based metrics to comprehensive data-driven and auto-discovered ensemble frameworks.

## 1. Core Concepts and Methodologies

Zero-cost proxies—also known as zero-shot or training-free proxies—are scalar functions $z(A)$ mapping a randomly-initialized, untrained architecture $A$ (or its DAG representation) to $\mathbb{R}$. Their goal is to rank architectures according to their expected generalization performance on a given task. The dominant methodology is to compute per-layer or per-block statistics using a single mini-batch (typically real or synthetic data, or even data-free all-ones input), aggregate these statistics, and use the result as a ranking signal [2101.08134][2210.03230].

A canonical zero-cost NAS routine thus comprises:
1. Generation or sampling of candidate architectures from the search space;
2. Per-candidate computation of one or several proxy scores in parallel;
3. Ranking and selection of promising candidates for full training or further evaluation.

Proxies can be categorized into:
- **Gradient-based proxies**: $\|\partial \mathcal{L}/\partial \theta\|$, SNIP, SynFlow, GraSP, Fisher, GradNorm, NWOT, Jacobian covariance (jacov), Hessian-based.
- **Activation and expressivity proxies**: Number of linear regions, NTK spectrum, Zen-NAS, NASWOT, epe-nas.
- **Structural/topological proxies**: FLOPs, number of parameters, operation counts, GRAF, SED.
- **Composite and learned proxies**: Ensemble models, genetic programming–discovered proxy formulas (EZNAS, GreenMachine), parametric models (ParZC), transformer-GCN models (TG-NAS).

## 2. Mathematical Foundations and Principal Formulas

Most zero-cost proxies reduce to a per-architecture aggregation of measurements computable at initialization. Key representatives are as follows:

| Proxy      | Formula/Computation (LaTeX syntax)                                                  | Intuition                         |
|------------|-------------------------------------------------------------------------------------|-----------------------------------|
| GradNorm   | $S_{\text{grad}}(\theta) = \sum_{i=1}^N \|\partial \mathcal{L}/\partial \theta_i\|$ | Trainability                      |
| SNIP       | $S_{\text{snip}} = \sum_{i=1}^N |\theta_i \, \partial \mathcal{L}/\partial \theta_i|$ | Sensitivity to pruning            |
| SynFlow    | $S_{\text{synflow}} = \sum_{i=1}^N |\partial \mathcal{L}_{\mathrm{sf}}/\partial \theta_i \cdot \theta_i|$ | Gradient flow in data-free mode   |
| Zen-NAS    | $S_{\text{zen}} = \log\, \mathbb{E}_x\|f_e(x) - f_e(x + \alpha\,\epsilon)\|_F$     | Expressivity under perturbations  |
| Jacov      | $S_{\text{jacov}} = -H_{\text{classes}}(\cos\text{sim}(J(x_i), J(x_j)))$            | Class-discriminative Jacobians    |
| Params     | $S_{\text{params}} = \sum_{l} |W_l|$                                                | Capacity and expressivity         |
| FLOPs      | $S_{\text{flops}} = \sum_{l} \text{FLOPs}_l$                                        | Capacity and compute, baseline    |

Composite and learned proxies, such as those found by EZNAS [2209.07413], LPZero [2410.04808], and GreenMachine [2411.15290], combine multiple such statistics with parametric or non-parametric operators to maximize empirical rank correlation with ground-truth test accuracy.

## 3. Evaluation Protocols and Benchmarks

Evaluation of proxy effectiveness is primarily conducted in terms of correlation between proxy-induced rankings and final test accuracy on large NAS benchmarks, including NAS-Bench-101, NAS-Bench-201, NATS-Bench SSS/TSS, and DARTS macro-cell spaces. Protocols include:
- **Rank-correlation metrics**: Kendall's $\tau = \frac{2}{n(n-1)}\sum_{i<j} \text{sign}(s_i-s_j)\,\text{sign}(y_i-y_j)$, Spearman's $\rho$, or, for regression models, $R^2$ on predicted accuracy [2210.03230][2307.01998][2307.09365].
- **Benchmarks**: Uniform or stratified sampling across the full architecture space to distinguish both high- and low-performing models.
- **Downstream impact**: Proxy-guided NAS loops (random search, evolution, RL, surrogate-based) are measured on the number of full-training runs to reach a fixed performance threshold and on wall-clock total search time [2101.08134][2110.10423][2404.00271].

A table of typical proxy performance (Kendall's $\tau$ on NAS-Bench-201/CIFAR-10) is below:

| Proxy        | $\tau$ |
|--------------|--------|
| Params       | 0.55   |
| SynFlow      | 0.54   |
| SNIP         | 0.41   |
| Zen-NAS      | 0.29   |
| TG-NAS       | 0.57   |
| GreenMachine | 0.89*  |
| GreenFactory | 0.90+  |

*Indicates stratified sampling or ensemble; see [2411.15290][2505.09344] for details.

## 4. Ensemble, Automated, and Learned Proxies

Limitations of individual proxies—generalization gaps across search spaces, spurious correlations with trivial statistics (e.g. params, depth), and reduced discrimination among top-10% architectures—have led to new automated and ensemble approaches:

- **Genetic Programming (GP) proxies**: EZNAS, GreenMachine, and LPZero search a symbolic program space, evolving novel zero-cost metrics that outperform hand-designed proxies and show superior transferability [2209.07413][2411.15290][2410.04808].
- **Parametric models**: ParZC utilizes a Mixer Architecture with Bayesian layers, learning to aggregate node-level proxies with uncertainty estimation and explicitly optimizing a differentiable Kendall's Tau loss [2402.02105].
- **Graph/Topology-only proxies**: SED [2411.06236], GRAF [2404.16551], and operator-level graph convolutional models (TG-NAS [2404.00271]) extract purely combinatorial or language-embedded features for performance estimation.
- **Multi-proxy ensembles**: GreenFactory [2505.09344] ensembles over 21 zero-cost proxies using a random forest regressor, achieving state-of-the-art correlations on NATS-Bench SSS/TSS.

Empirical evidence confirms that ensembles leveraging complementary proxies frequently exceed the performance of any single measure and mitigate biases due to over-reliance on architecture size or skip-connections [2210.03230][2505.09344].

## 5. Integration in NAS Workflows and Empirical Acceleration

Zero-cost proxies are now tightly integrated into the inner and outer loops of NAS algorithms. Common modes of use include:

- **Warmup**: Score large candidate pools by proxy, restrict expensive training to the top-$k$ architectures [2101.08134].
- **Move proposal**: Use proxies at each selection or mutation step in evolutionary or RL-based NAS, rapidly generating high-quality offspring [2101.08134].
- **Surrogate modeling**: Proxy scores serve as inputs to fast regression or ranking surrogates (e.g., XGBoost, random forests) that predict performance or guide sampling [2210.03230][2411.15290][2505.09344].
- **One-shot NAS augmentation**: Ranking distillation from zero-cost proxies improves ranking quality in weight-sharing supernets [2301.09850].
- **Differentiable NAS operation selection**: Zero-cost operation scoring (Zero-Cost-PT) outperforms classic DARTS $\alpha$-based selection for discrete operations in one-shot NAS, enabling scalable, high-fidelity NAS in macro and cell-based spaces [2106.06799].

Speedups are substantive: ProxyBO achieves up to $5.41\times$ fewer full-training steps than regularized evolution to hit identical test error on NAS-Bench-101; AZ-NAS and GreenFactory achieve effective search using $<$1\% of the compute of training-based methods on NATS-Bench; SED delivers a $10^3\times$ runtime reduction relative to gradient-based proxies [2110.10423][2403.19232][2411.06236][2505.09344].

## 6. Limitations, Pitfalls, and Design Best Practices

While zero-cost proxies are indispensable in training-free NAS, several limitations persist:
- **Generalization gaps**: Many proxies, especially size-based or gradient proxies, exhibit strong bias towards models with more parameters or deeper layers, failing in search spaces with non-monotonic parameter–performance relationships [2210.03230][2404.16551].
- **Discrimination among top-tiers**: Correlations typically degrade when restricted to the top-$x$\% of highest-accuracy models [2307.01998].
- **Robustness prediction**: While clean accuracy can often be regressed from one or two proxies, prediction of adversarial robustness requires aggregation of multiple (often weakly correlated) measures [2307.09365].
- **Domain transfer**: Operator-specific or search-space–tuned proxies (e.g., ViT, CNN, NLP) degrade outside their domain. Universal proxies such as TG-NAS, Auto-Prox, and LPZero aim to address this but remain an active area [2404.00271][2312.09059][2410.04808].
- **Hyperparameter tuning and proxy selection**: Performance and runtime are sensitive to proxy hyperparameters (e.g., mini-batch size, initializations), smoothing temperatures, and aggregation functions [2110.10423][2402.02105].

Best practices—substantiated by large-scale empirical studies—include:
- Ensemble several non-redundant proxies and use regression or ranking models to aggregate their signal [2210.03230][2505.09344].
- Pre-assess and bias-correct proxies with respect to trivial architecture features (cell size, op count) [2210.03230][2404.16551].
- Evaluate proxies in stratified accuracy bins, not only on randomly drawn architectures, to ensure signal across the full spectrum [2411.15290][2505.09344].
- Introduce task and hardware-features where applicable for hardware-aware NAS [2307.01998][2404.00271].

## 7. Advances, Open Challenges, and Future Directions

Recent technical progress includes:
- Automated design of universal and transfer-friendly symbolic proxies for non-CNN architectures (ViT, LLM; e.g., Auto-Prox, LPZero) [2312.09059][2410.04808].
- Genetic programming and differentiable proxy learning (GreenMachine, ParZC) for task-adaptive and uncertainty-aware zero-cost estimation [2411.15290][2402.02105].
- Purely topological, data- and parameter-free proxies for ultrafast scoring (SED, GRAF) and interpretability [2411.06236][2404.16551].
- Combined multi-objective zero-cost NAS, e.g., balancing accuracy, robustness, and hardware constraints [2307.09365][2404.00271][2307.01998].

Open challenges remain in designing proxies that generalize across architecture families, modalities (vision, NLP, speech), and resource regimes, as well as in robustly fusing proxy sets without substantial human tuning. The theoretical underpinnings—e.g., why certain proxies correlate with accuracy in deep, structured networks—are only partially elucidated. Ongoing research is expanding zero-cost proxy theory to integrate hardware-awareness, multi-objective tradeoffs, and more principled connections to optimization and generalization.

In summary, zero-cost proxies are fundamental tools for scaling and democratizing NAS, offering several orders of magnitude acceleration over classical model evaluation and expanding the feasible scope of automated architecture search [2101.08134][2110.10423][2209.07413][2210.03230][2404.00271][2411.06236][2505.09344].

Source: https://www.emergentmind.com/topics/zero-cost-proxies