Accelerating Very Deep Convolutional Networks for Classification and Detection (1505.06798v2)

Abstract: This paper aims to accelerate the test-time computation of convolutional neural networks (CNNs), especially very deep CNNs that have substantially impacted the computer vision community. Unlike previous methods that are designed for approximating linear filters or linear responses, our method takes the nonlinear units into account. We develop an effective solution to the resulting nonlinear optimization problem without the need of stochastic gradient descent (SGD). More importantly, while previous methods mainly focus on optimizing one or two layers, our nonlinear method enables an asymmetric reconstruction that reduces the rapidly accumulated error when multiple (e.g., >=10) layers are approximated. For the widely used very deep VGG-16 model, our method achieves a whole-model speedup of 4x with merely a 0.3% increase of top-5 error in ImageNet classification. Our 4x accelerated VGG-16 model also shows a graceful accuracy degradation for object detection when plugged into the Fast R-CNN detector.

Accelerating Very Deep Convolutional Networks for Classification and Detection

The paper "Accelerating Very Deep Convolutional Networks for Classification and Detection" by Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun addresses the imperative issue of reducing the computational complexity of Convolutional Neural Networks (CNNs), especially very deep models like VGG-16, during test time. Given the exponential increase in computational cost with deeper networks, this paper proposes a novel method that incorporates nonlinear transformations and low-rank approximations to achieve significant speedups without markedly degrading accuracy.

The proposed method diverges from traditional acceleration techniques that primarily focus on linear approximations. Instead, it takes into account the nonlinearity introduced by activation functions like ReLU. The method involves solving a nonlinear optimization problem without relying on stochastic gradient descent (SGD), which is prone to various issues such as sensitivity to initialization and local optima in complex tasks like ImageNet classification.

Key Contributions

1. Nonlinear Optimization Solution: The authors propose a response reconstruction method combining nonlinear neurons and low-rank constraints. This is achieved using Generalized Singular Value Decomposition (GSVD), allowing the method to account for errors accumulated through multiple layers, prominently in deeper networks.
2. Asymmetric Reconstruction: The method introduces an asymmetric reconstruction approach that addresses accumulated error by optimizing each layer sequentially. This 'asymmetric' approach uses the precise responses of previous non-approximated layers to minimize the error in subsequent approximations.
3. Rank Selection Technique: A rank selection method is incorporated to determine the optimal rank for each layer based on their redundancy. The rationale is to adaptively allocate computational resources by understanding the individual contribution of each layer to the overall accuracy, thus maximizing efficiency and maintaining high accuracy.
4. Three-Dimensional Decomposition: The authors extend their work by incorporating spatial decomposition techniques to address spatial dimensions alongside channel dimensions. This combination effectively uses both methods to better approximate layers and manage computational resources.

Experimental Results

The experimental evaluation is thorough, involving two prominent models: the SPP-10 model and the VGG-16 model. Results showcase that the proposed nonlinear and asymmetric methods significantly outperform traditional methods such as those proposed by Jaderberg et al. in terms of accuracy degradation at similar speedup ratios.

• Single-Layer Testing:

  Single-layer evaluations demonstrate that the nonlinear GSVD-based approach consistently improves accuracy compared to linear methods, especially for higher layers like Conv7, which have a higher degree of sparsity in their activations.

• Multi-Layer and Whole-Model Evaluations:

  For multi-layer approximations, the asymmetric reconstruction method shows a substantial reduction in the increase of classification error, underscoring its efficacy in managing error accumulation. In whole-model tests, the rank selection approach further optimizes overall performance by balancing the approximation precision across all layers.

• Application to Object Detection:

  The accelerated models were tested in object detection tasks using the Fast R-CNN framework on the PASCAL VOC 2007 dataset. Results indicated a graceful degradation in performance, validating the practical viability of deploying such accelerated models in real-world applications.

Implications and Future Work

From a practical standpoint, this research implies that very deep CNNs can be significantly accelerated without substantial loss of accuracy, making them more deployable in real-time applications such as mobile computing and cloud services where computational resources are constrained. The theoretical implications suggest that better optimization techniques that handle nonlinearity and minimize errors across sequences of operations can unlock more efficient use of deep learning models.

Looking forward, further work could explore optimizing training efficiency by integrating these principles, thus reducing both training and inference time. Moreover, expanding the methods to other types of layers, such as batch normalization and residual connections in even deeper networks like ResNets, could enhance the versatility and applicability of the presented techniques.

In conclusion, the method described in this paper marks a notable advancement in accelerating very deep networks, offering a balanced trade-off between speedup and accuracy—paving the way for more efficient and practical deployment of deep learning models in various domains.