NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performance

Abstract: Artificial neural networks have been shown to be state-of-the-art machine learning models in a wide variety of applications, including natural language processing and image recognition. However, building a performant neural network is a laborious task and requires substantial computing power. Neural Architecture Search (NAS) addresses this issue by an automatic selection of the optimal network from a set of potential candidates. While many NAS methods still require training of (some) neural networks, zero-cost proxies promise to identify the optimal network without training. In this work, we propose the zero-cost proxy \textit{Network Expressivity by Activation Rank} (NEAR). It is based on the effective rank of the pre- and post-activation matrix, i.e., the values of a neural network layer before and after applying its activation function. We demonstrate the cutting-edge correlation between this network score and the model accuracy on NAS-Bench-101 and NATS-Bench-SSS/TSS. In addition, we present a simple approach to estimate the optimal layer sizes in multi-layer perceptrons. Furthermore, we show that this score can be utilized to select hyperparameters such as the activation function and the neural network weight initialization scheme.

• The paper introduces NEAR, a zero-cost proxy that pre-estimates neural network performance using effective rank calculations of activation matrices.
• It computes both pre- and post-activation matrix ranks across layers to generate a NEAR score that correlates strongly with final model accuracy.
• NEAR demonstrates practical utility by streamlining architecture search, optimizing layer sizing, and aiding in activation function selection.

Review of "NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performance"

The paper "NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performance" by Raphael T. Husistein, Markus Reiher, and Marco Eckhoff introduces a novel zero-cost proxy called Network Expressivity by Activation Rank (NEAR). This technique aims to predict the performance of neural network architectures without necessitating any training, thus potentially saving considerable computational resources and time. Here, we provide a detailed overview of the methodology, results, and implications of this research.

Introduction and Motivation

The development and training of artificial neural networks, crucial for numerous applications such as natural language processing and image recognition, is resource-intensive. Neural Architecture Search (NAS) automates the selection of optimal neural network architectures but traditionally requires training part or all candidate models, which remains computationally expensive. The advent of zero-cost proxies offers a promising alternative by identifying optimal architectures without training. NEAR contributes to this growing body of work by leveraging the effective rank of pre- and post-activation matrices as a measurement of network expressivity and potential performance.

Methodology of NEAR

Key Concepts:

• Effective Rank: NEAR relies on the effective rank of matrices, a measure of their dimensionality and diversity. For a given layer, the effective rank is calculated for both the pre-activation (input to the activation function) and post-activation (output after applying the activation function) matrices of the neural network.
• Calculation Process: Inputs are passed through the network to generate pre- and post-activation matrices, for which NEAR calculates the effective rank. These ranks are summed across all layers to produce a final NEAR score, hypothesized to correlate with the network’s predictive performance.

Evaluation and Results

The efficacy of NEAR is demonstrated on standard NAS benchmarks: NAS-Bench-101, NATS-Bench-SSS, and NATS-Bench-TSS. These benchmarks cover a range of architectures and datasets, providing a robust evaluation environment.

NAS-Bench-101

• Correlation Performance: NEAR achieves superior rank correlation with final accuracy on NAS-Bench-101, surpassing other proxies such as Zen-Score, ZiCo, and MeCo\textsubscript{opt}.

NATS-Bench-SSS and NATS-Bench-TSS

• NATS-Bench-SSS: NEAR consistently outperforms the majority of proxies in terms of correlation metrics across datasets within this search space.
• NATS-Bench-TSS: While MeCo\textsubscript{opt} marginally outperforms NEAR in some instances, NEAR demonstrates a closely competitive performance.

Practical Applications

Layer Size Estimation

The paper introduces a method for estimating optimal layer sizes in multi-layer perceptrons based on the relative NEAR score across different sizes. This method is applied to two datasets, yielding efficient and high-performing network configurations with fewer parameters than traditional methods.

Activation Function and Weight Initialization

NEAR is evaluated for its ability to identify effective activation functions and weight initialization schemes. Using benchmarks on machine learning potentials and balanced MNIST datasets, NEAR effectively distinguishes between different activation functions and initialization schemes, demonstrating its utility beyond architecture selection.

Implications and Future Directions

The introduction of NEAR holds both practical and theoretical implications:

• Practical Implications: NEAR provides a robust tool for reducing the computational cost associated with neural network design, allowing for rapid prototyping and validation of architectures. This capability is particularly important in resource-constrained environments and applications requiring frequent model updates.
• Theoretical Implications: The methodology extends the applicability of zero-cost proxies by focusing on network expressivity in a generalized manner, applicable across various activation functions and independent of task-specific labels.

Conclusion

NEAR presents a significant advancement in the field of zero-cost proxies for neural architecture search. By leveraging the concept of effective rank to measure network expressivity, NEAR proposes a training-free alternative for pre-estimating neural network performance. Its ability to estimate both architectural aspects and hyperparameters efficiently marks an important contribution to the field, providing a foundation for future work towards even more generalized and powerful zero-cost proxies.

The paper references crucial foundational and contemporary works in neural network training, architecture search, and machine learning methodologies to establish the theoretical underpinnings and positioning of NEAR within the context of related research.

Overall, NEAR demonstrates notable precision and generalizability, pointing towards exciting possibilities for future AI developments and applications.