Towards Principled Design of Deep Convolutional Networks: Introducing SimpNet (1802.06205v1)

Abstract: Major winning Convolutional Neural Networks (CNNs), such as VGGNet, ResNet, DenseNet, \etc, include tens to hundreds of millions of parameters, which impose considerable computation and memory overheads. This limits their practical usage in training and optimizing for real-world applications. On the contrary, light-weight architectures, such as SqueezeNet, are being proposed to address this issue. However, they mainly suffer from low accuracy, as they have compromised between the processing power and efficiency. These inefficiencies mostly stem from following an ad-hoc designing procedure. In this work, we discuss and propose several crucial design principles for an efficient architecture design and elaborate intuitions concerning different aspects of the design procedure. Furthermore, we introduce a new layer called {\it SAF-pooling} to improve the generalization power of the network while keeping it simple by choosing best features. Based on such principles, we propose a simple architecture called {\it SimpNet}. We empirically show that SimpNet provides a good trade-off between the computation/memory efficiency and the accuracy solely based on these primitive but crucial principles. SimpNet outperforms the deeper and more complex architectures such as VGGNet, ResNet, WideResidualNet \etc, on several well-known benchmarks, while having 2 to 25 times fewer number of parameters and operations. We obtain state-of-the-art results (in terms of a balance between the accuracy and the number of involved parameters) on standard datasets, such as CIFAR10, CIFAR100, MNIST and SVHN. The implementations are available at \href{url}{https://github.com/Coderx7/SimpNet}.

• The paper proposes several principles for principled deep convolutional network design, focusing on efficiency and robustness, and introduces SimpNet as an example architecture based on these guidelines.
• Empirical evaluation shows SimpNet achieves competitive accuracy on CIFAR10/100, MNIST, and SVHN datasets with significantly fewer parameters than deeper models like VGGNet, ResNet, and DenseNet.
• The introduction of SAF-Pooling combined with dropout enhances network robustness against occlusions, suggesting new directions for adaptive pooling mechanisms and architecture efficiency.

Overview of CNN Design Principles: The Introduction of SimpNet

The paper "Towards Principled Design of Deep Convolutional Networks: Introducing SimpNet" by Seyyed Hossein Hasanpour Matikolaee et al. presents a critical exploration of conventional practices in Convolutional Neural Network (CNN) design, proposing new principles for optimizing architectures with a specific focus on computation and memory efficiency. The research is an insightful addition to the dialogue surrounding efficient neural network design, extending existing paradigms with innovative methodologies.

Key Propositions and Principles

The authors outline several foundational principles that should guide the design of efficient CNN architectures:

1. Gradual Expansion and Minimum Allocation: Highlighting that deeper networks typically offer improved performance, the paper suggests gradually expanding the architecture and avoiding superfluous allocation of processing units to prevent overfitting and unnecessary computation overhead.
2. Homogeneous Groups of Layers: The paper advises designing networks in homogeneous groups of layers, contributing to a structured hierarchy and facilitating more granular tuning and inspection.
3. Local Correlation Preservation: Emphasizing the importance of capturing spatial correlations, the authors recommend using smaller kernels, particularly $3 \times 3$, to preserve locality, especially in the initial layers of the network.
4. Maximum Information Utilization: This principle aims to preserve larger feature maps and thus maintain richer information pools to enhance the network's discriminative capabilities.
5. Strided Convolution vs. Pooling: The authors argue for the continued use of pooling over strided convolution, asserting that pooling effectively provides translation invariance and reduces computational complexity.
6. Dropout and SAF-Pooling: Introducing the Simple Adaptive Feature composition Pooling (SAF-Pooling) operation, the paper advocates for combining pooling with dropout to simulate feature absence scenarios caused by occlusions or variations, thereby training networks to be more robust.

SimpNet Architecture

Based on the aforementioned principles, the authors introduce SimpNet, a CNN designed to exemplify these efficient strategies. SimpNet is characterized by its simplicity, avoiding complex multi-path designs, and demonstrating the power of fundamental architectural constructs when systematically optimized. The network's empirical evaluation demonstrates its ability to achieve a competitive balance between accuracy and parameter utilization, outperforming deeper models like VGGNet, ResNet, and DenseNet on datasets such as CIFAR10, CIFAR100, MNIST, and SVHN, while maintaining significant parameter efficiency.

Experimental Insights and Results

In various experiments, SimpNet is benchmarked against prominent architectures across standard datasets. It manages to achieve state-of-the-art results in terms of accuracy balanced by the number of parameters. Notably, SimpNet exhibits robustness in scenarios where parameter budgets are stringent, outperforming architectures with far greater complexity and depth.

Implications and Future Directions

The paper has several key implications for the community. It encourages a re-evaluation of depth-centric architecture design, promoting efficiency through systematic principles rather than ad hoc modifications. The successful application of SAF-pooling introduces a promising area for future research to explore adaptive pooling mechanisms that enhance network robustness.

The authors suggest evolving these principles to encompass broader architectural challenges and propose models that are not only efficient but adaptive to mobile and embedded systems challenges. They highlight the potential for automated architecture design, where the principles could drive evolutionary optimization techniques to craft tailored networks for specific tasks.

Overall, the insights provided by Hasanpour et al. foster a critical discourse on efficient design principles in deep learning architecture, underlining the need for a principled approach in the face of ever-growing computational demands.