Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups (1605.06489v3)

Abstract: We propose a new method for creating computationally efficient and compact convolutional neural networks (CNNs) using a novel sparse connection structure that resembles a tree root. This allows a significant reduction in computational cost and number of parameters compared to state-of-the-art deep CNNs, without compromising accuracy, by exploiting the sparsity of inter-layer filter dependencies. We validate our approach by using it to train more efficient variants of state-of-the-art CNN architectures, evaluated on the CIFAR10 and ILSVRC datasets. Our results show similar or higher accuracy than the baseline architectures with much less computation, as measured by CPU and GPU timings. For example, for ResNet 50, our model has 40% fewer parameters, 45% fewer floating point operations, and is 31% (12%) faster on a CPU (GPU). For the deeper ResNet 200 our model has 25% fewer floating point operations and 44% fewer parameters, while maintaining state-of-the-art accuracy. For GoogLeNet, our model has 7% fewer parameters and is 21% (16%) faster on a CPU (GPU).

Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups

The paper "Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups" addresses the challenges of computational inefficiency in deep convolutional neural networks (CNNs) by introducing a novel architectural approach. This work focuses on leveraging hierarchical filter groups to create sparsely connected, compact CNNs that maintain or even improve upon the accuracy of existing state-of-the-art models. The proposed architecture aims to reduce computational costs and model size significantly.

The authors demonstrate that a large proportion of the learned weights in CNNs are often redundant, which can be exploited to enhance efficiency without losing accuracy. This is achieved by imposing a sparse connection structure resembling tree roots, where filters are divided into groups that process subsets of the input channels. This structural sparsity leads to a considerable decrease in both floating-point operations (FLOPS) and parameters required, thereby accelerating both training and inference processes.

Key numerical results validate the efficacy of this method: the paper reports a 31% reduction in CPU inference time and a 12% reduction in GPU inference time for a modified ResNet 50 architecture, alongside 40% fewer parameters and 45% fewer floating point operations. The approach also maintains comparable accuracy to baseline models on benchmark datasets like CIFAR10 and ILSVRC.

The implications of this research are multifaceted. Practically, the ability to deploy efficient CNNs on resource-constrained devices such as mobile phones and embedded systems is enhanced, given the reduced computational and memory requirements. Theoretically, the concept of exploiting hierarchical sparsity aligns with biological neural networks' understanding, reflecting a more efficient feature learning process. Future work could explore dynamic filter group structures or integrate this methodology with other efficiency-driven techniques like quantization or pruning.

In conclusion, this paper offers substantial contributions to the field of efficient deep learning architectures. By reducing computational burden without compromising performance, it paves the way for advances in deploying deep learning models across diverse application areas. Future developments could include exploring adaptive filter groups and broader applications beyond image classification, potentially influencing fields such as natural language processing and reinforcement learning.