FermiNets: Learning generative machines to generate efficient neural networks via generative synthesis (1809.05989v2)

Abstract: The tremendous potential exhibited by deep learning is often offset by architectural and computational complexity, making widespread deployment a challenge for edge scenarios such as mobile and other consumer devices. To tackle this challenge, we explore the following idea: Can we learn generative machines to automatically generate deep neural networks with efficient network architectures? In this study, we introduce the idea of generative synthesis, which is premised on the intricate interplay between a generator-inquisitor pair that work in tandem to garner insights and learn to generate highly efficient deep neural networks that best satisfies operational requirements. What is most interesting is that, once a generator has been learned through generative synthesis, it can be used to generate not just one but a large variety of different, unique highly efficient deep neural networks that satisfy operational requirements. Experimental results for image classification, semantic segmentation, and object detection tasks illustrate the efficacy of generative synthesis in producing generators that automatically generate highly efficient deep neural networks (which we nickname FermiNets) with higher model efficiency and lower computational costs (reaching >10x more efficient and fewer multiply-accumulate operations than several tested state-of-the-art networks), as well as higher energy efficiency (reaching >4x improvements in image inferences per joule consumed on a Nvidia Tegra X2 mobile processor). As such, generative synthesis can be a powerful, generalized approach for accelerating and improving the building of deep neural networks for on-device edge scenarios.

An Analytical Overview of "FermiNets: Learning Generative Machines to Generate Efficient Neural Networks via Generative Synthesis"

The pursuit of designing more efficient neural networks remains at the forefront of AI research, especially regarding deployment in constrained environments such as mobile and consumer devices. The paper "FermiNets: Learning Generative Machines to Generate Efficient Neural Networks via Generative Synthesis" introduces a novel method, generative synthesis, for automatically generating highly efficient deep neural networks, which addresses the challenges posed by the computational and architectural complexity of traditional deep learning models.

Generative Synthesis and FermiNets

The central contribution of the paper lies in the introduction of generative synthesis, a process using a generator-inquisitor pair to create neural networks that meet specific operational requirements. Generative synthesis represents a departure from existing strategies like precision reduction or model compression, focusing instead on learning a generator capable of producing optimized network architectures without manual intervention.

The paper presents strong numerical results across different tasks such as image classification, semantic segmentation, and object detection. The resulting networks, named FermiNets, exhibit notable improvements in model efficiency and computational cost compared to state-of-the-art networks. Key performance metrics include achieving efficiencies greater than ten times current models while maintaining competitive accuracy levels, with additional enhancements in energy efficiency demonstrated on NVIDIA Tegra X2 processors.

Methodology

The methodology is mathematically formalized, with the generator $\mathcal{G}$ and inquisitor $\mathcal{I}$ working in tandem to iteratively improve network designs through a defined objective function. This objective function maximizes a universal performance measure while meeting operational constraints. This iterative learning framework stands out for its ability to produce a diverse range of network configurations tailored to the computational constraints of edge devices.

Experimental Validation

The empirical validation of generative synthesis is highlighted through experiments across varying AI tasks. Notably, FermiNets outperformed models like MobileNet, ShuffleNet, and RefineNet in terms of metrics such as information density, MAC operations, and NetScore. These improvements affirm the potential of generative synthesis to balance accuracy with architectural and computational efficiencies, making neural networks viable for edge deployment.

• Image Classification: FermiNets yielded top-1 accuracies exceeding other efficient networks by up to 1.4% on CIFAR-10 datasets, with significantly reduced MAC operations, emphasizing computational efficiency.
• Semantic Segmentation: The FermiNet-SS displayed an information density over twelve times that of the baseline RefineNet, while maintaining high accuracy levels.
• Object Detection: The FermiNet-OD demonstrated substantial energy efficiency improvements, allowing mobile processors to perform more inferences per joule consumed than existing models like DetectNet.

Implications and Future Directions

The implications of the work are twofold: practically, it provides a framework for deploying deep learning models on edge devices without sacrificing performance or efficiency. Theoretically, it opens new avenues for research in generative networks and automatic architecture synthesis. The ability of FermiNets to generalize and produce a wide array of network designs has potential applications in various technological and consumer industries, potentially leading to more sustainable AI deployments.

In conclusion, the paper provides insightful advancements for the synthesis of efficient neural networks through generative processes. As the field progresses, further research could address optimizing generator-inquisitor dynamics and expanding the scope of operational requirements for broader applicability across diverse edge computing platforms.