Speeding up Convolutional Neural Networks with Low Rank Expansions (1405.3866v1)

Abstract: The focus of this paper is speeding up the evaluation of convolutional neural networks. While delivering impressive results across a range of computer vision and machine learning tasks, these networks are computationally demanding, limiting their deployability. Convolutional layers generally consume the bulk of the processing time, and so in this work we present two simple schemes for drastically speeding up these layers. This is achieved by exploiting cross-channel or filter redundancy to construct a low rank basis of filters that are rank-1 in the spatial domain. Our methods are architecture agnostic, and can be easily applied to existing CPU and GPU convolutional frameworks for tuneable speedup performance. We demonstrate this with a real world network designed for scene text character recognition, showing a possible 2.5x speedup with no loss in accuracy, and 4.5x speedup with less than 1% drop in accuracy, still achieving state-of-the-art on standard benchmarks.

• The paper introduces two novel low-rank approximation schemes that deliver up to 4.5x speedup with less than a 1% drop in accuracy.
• It leverages filter and data reconstruction optimizations to decompose 2D convolutions into efficient 1D operations.
• Experimental results on scene text recognition demonstrate competitive accuracy, enabling practical real-time CNN deployment.

Speeding Up Convolutional Neural Networks with Low Rank Expansions

The paper "Speeding up Convolutional Neural Networks with Low Rank Expansions" by Jaderberg, Vedaldi, and Zisserman addresses an important computational challenge in the deployment of Convolutional Neural Networks (CNNs). Specifically, the authors propose two acceleration schemes that significantly reduce the computational burden of CNNs by exploiting channel and filter redundancy to create low-rank filter approximations. This paper's core contributions lie in its innovative methods that are architecture-agnostic and can be seamlessly integrated into existing CPU and GPU frameworks, thereby offering versatile applicability.

Introduction

CNNs have revolutionized machine learning applications, particularly in computer vision, by providing state-of-the-art performance in numerous benchmarks. However, the substantial computational requirements of these networks, especially during the evaluation phase, pose a significant barrier to their widespread deployment, particularly in real-time applications. The authors highlight that convolutional layers are the primary computational bottleneck in these networks and focus on accelerating them through low-rank expansions that take advantage of filter redundancy.

Filter Approximation Schemes

The authors introduce two primary schemes for filter approximation:

1. Scheme 1 directly applies low-rank approximations to each convolutional filter. This involves decomposing the filters into a linear combination of a smaller set of separable basis filters. This approach leverages the redundancy between filters applied to different channels, aiming to achieve computational efficiency without sacrificing accuracy.
2. Scheme 2 extends the low-rank approximation concept by factoring convolutional layers into sequences of two layers with rectangular filters. This method considers both input and output channel redundancies, leading to superior performance in terms of speedup. The filters are split into 1D vertical and horizontal components, effectively transforming 2D convolutions into a series of less computationally intense 1D operations.

Optimization Methods

The paper details two optimization techniques for training these low-rank approximations:

• Filter Reconstruction Optimization minimizes the reconstruction error of original filters by approximating them with a separable basis. This involves penalizing the nuclear norm of the filters to enforce low-rank properties.
• Data Reconstruction Optimization focuses on the reconstruction error of layer outputs given actual training data. This technique is particularly effective as it aligns closely with the training data distribution, thereby potentially improving the model's end-to-end performance. The optimization is performed by constructing a mirrored CNN for the separated basis layers and training them using back-propagation.

Experimental Results

The paper empirically evaluates the proposed schemes using a CNN trained for scene text character recognition. The baseline model achieves state-of-the-art performance of 91.3% accuracy. By applying the proposed schemes, the authors demonstrate a possible 2.5x speedup without any loss in accuracy, and a 4.5x speedup with less than a 1% drop in accuracy. This results in an accuracy of 90.3%, which remains competitive with current state-of-the-art methods.

Scheme 2, in particular, shows superior actual measured speedups compared to the theoretical improvements due to the more efficient handling of the convolution operations in the Caffe framework.

Implications and Future Work

The implications of this research are far-reaching. By significantly reducing the computational load of CNNs, these methods pave the way for more efficient deployment in real-time systems where speed and resource constraints are critical. This could impact various applications, from autonomous vehicles to mobile devices, where computational efficiency and real-time processing capabilities are paramount.

Future research directions suggested by the authors include exploring other forms of separable filter arrangements and structures. Moreover, the potential to incorporate these low-rank approximations during the training phase of CNNs calls for further investigation. Understanding how to leverage these reductions without compromising discriminative performance remains an open and exciting challenge.

In summary, this paper provides valuable insights and practical methods for accelerating CNNs, making significant strides toward more efficient deep learning implementations.