Zen-NAS: A Zero-Shot NAS for High-Performance Image Recognition
This paper presents Zen-NAS, an innovative Neural Architecture Search (NAS) framework that eliminates the need for computationally intensive accuracy predictors. Current NAS methodologies are often hindered by the burdensome costs associated with training numerous networks for evaluating their architectures. Zen-NAS addresses this challenge through the introduction of the Zen-Score, a zero-cost proxy for ranking neural networks based on expressivity.
Zen-Score is calculated by performing forward inferences on a randomly initialized network, without necessitating any parameter training. This approach significantly reduces computational demands, enabling high-performance architecture search within just 0.5 GPU days, as opposed to traditional methods that require extensive resources. Zen-NAS leverages this rapid computation, optimizing architectures under various inference constraints while maintaining competitive accuracy levels on the ImageNet dataset.
The methodology outlined in the paper proposes using the Zen-Score as a novel index reflecting network expressivity—a metric correlating positively with accuracy. The computations are performed on randomly initialized networks, employing Gaussian inputs to swiftly approximate network expressivity. Notably, Zen-Score is shown to manage scale sensitivity effectively, particularly in handling Batch Normalization layers, making it adaptable to real-world network design challenges.
Zen-NAS differentiates itself by providing a zero-shot NAS approach focused on maximizing the Zen-Score of candidate architectures. This enables the rapid identification of high-performance networks without the typical iterative learning and fine-tuning processes. The empirical outcomes demonstrate substantial enhancements in speed and efficiency across various hardware settings, including server-side and mobile GPUs.
This research brings forth significant implications. Practically, Zen-NAS can democratize access to efficient NAS by reducing the computational barriers for researchers and developers without extensive hardware resources. From a theoretical perspective, the paper challenges conventional NAS paradigms, advocating for exploration in expressivity-driven, data-free NAS methods.
Potential future trajectories of this work could involve further refinement of the Zen-Score to account for more complex network operations or its integration into hybrid NAS approaches, combining zero-shot efficiency with scalable adjustment strategies. Moreover, as neural networks continue to proliferate in varied applications, extending Zen-NAS to address domain-specific performance requirements could be instrumental in advancing the versatility and applicability of automated neural architecture design.
In summary, Zen-NAS exhibits remarkable promise in reimagining NAS by channeling theoretical insights into practical, scalable solutions. It invites the research community to rethink architecture design, grounding future explorations in the efficient evaluation of expressivity while simplifying the computational landscape of NAS.