MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (1711.06798v3)

Abstract: We present MorphNet, an approach to automate the design of neural network structures. MorphNet iteratively shrinks and expands a network, shrinking via a resource-weighted sparsifying regularizer on activations and expanding via a uniform multiplicative factor on all layers. In contrast to previous approaches, our method is scalable to large networks, adaptable to specific resource constraints (e.g. the number of floating-point operations per inference), and capable of increasing the network's performance. When applied to standard network architectures on a wide variety of datasets, our approach discovers novel structures in each domain, obtaining higher performance while respecting the resource constraint.

• The paper introduces MorphNet, a method that iteratively shrinks and expands network layers to optimize architectures under specific resource constraints.
• The paper demonstrates robust performance improvements on Inception V2 and MobileNet with consistent FLOP usage and enhanced test accuracy on ImageNet.
• The paper implies practical benefits for deploying efficient deep networks in resource-limited settings, paving the way for automated architecture design in diverse applications.

Iterative Optimization of Neural Network Architectures with MorphNet

The paper introduces MorphNet, a method that automates the design of neural network structures by iteratively optimizing the architecture through a process of shrinking and expanding network layers. In contrast to traditional manually-designed architectures, MorphNet offers a systematic approach to structure learning that accounts for computational resources, such as the number of floating-point operations (FLOPs) per inference, while striving to enhance model performance.

Methodology

MorphNet operates via two primary phases: shrinking and expanding. The shrinking step applies a sparsifying regularizer that selectively prunes neurons based on resource constraints, predominantly focusing on activations. This regularizer is resource-weighted, allowing the network to adapt to specific resource limitations inherent in various computational environments. Subsequently, the expanding phase uniformly scales the network’s layers multiplicatively to recover the model's initial computational budget, facilitating an increase in overall network capacity without breaching resource constraints.

The adaptive nature of this approach makes it scalable to larger network architectures, such as those required in industrial applications, where manual tuning can be prohibitively complex. MorphNet’s ability to accommodate a wide array of standard architectures and diverse datasets further underscores its versatility.

Key Experimental Results

The researchers validated MorphNet’s efficacy through extensive experiments, particularly demonstrating its capability on the Inception V2 and MobileNet architectures when trained on the ImageNet dataset. For Inception V2 with a FLOP regularizer, training results indicated a convergence in architectural configurations across separate experimental runs, exemplified by a relative standard deviation of 1.12% and 0.208% for FLOPs and test accuracy, respectively. This consistency highlights the robustness and stability of the architectures produced by MorphNet under identical hyperparameter settings.

Moreover, MorphNet outperformed some traditional baselines, particularly in cases involving MobileNets, where the method was able to surpass models with width multiplier baselines by producing architectures that offered a more optimal trade-off between FLOPs and accuracy.

Discussion of Implications

The practical implications of MorphNet are significant, especially in environments where computational resources are a governing factor. By optimizing neural network architectures with respect to predefined resources, MorphNet allows for the deployment of more efficient deep learning models in resource-constrained scenarios, such as mobile and edge devices. Theoretically, the introduction of a systematic approach to iterate over both architecture parameters and resource constraints could encourage the emergence of new neural network topology designs that were previously unexplored due to manual design limitations.

Furthermore, MorphNet’s framework suggests potential applications beyond mere width optimization. The paper hints at extending this iterative optimization to include other architectural aspects, such as filter dimensions and network depth, which could lead to more holistic network tuning processes for a variety of applications. Future work could involve integrating other forms of regularizers and exploring non-uniform expansion strategies to further enhance MorphNet's flexibility.

In summary, MorphNet presents a compelling methodology for automated neural network design, marrying performance improvement with resource efficiency. Its demonstrated scalability and adaptability mark it as a valuable tool for the development of optimized deep learning architectures, fostering potential advancements in both academic and practical domains of machine learning.