More is Less: A More Complicated Network with Less Inference Complexity (1703.08651v2)

Abstract: In this paper, we present a novel and general network structure towards accelerating the inference process of convolutional neural networks, which is more complicated in network structure yet with less inference complexity. The core idea is to equip each original convolutional layer with another low-cost collaborative layer (LCCL), and the element-wise multiplication of the ReLU outputs of these two parallel layers produces the layer-wise output. The combined layer is potentially more discriminative than the original convolutional layer, and its inference is faster for two reasons: 1) the zero cells of the LCCL feature maps will remain zero after element-wise multiplication, and thus it is safe to skip the calculation of the corresponding high-cost convolution in the original convolutional layer, 2) LCCL is very fast if it is implemented as a 1*1 convolution or only a single filter shared by all channels. Extensive experiments on the CIFAR-10, CIFAR-100 and ILSCRC-2012 benchmarks show that our proposed network structure can accelerate the inference process by 32\% on average with negligible performance drop.

• The paper introduces a low-cost collaborative layer (LCCL) that dynamically skips redundant computations in CNNs to reduce inference complexity.
• The methodology exploits ReLU-induced sparsity to achieve over 32% speedup on benchmarks such as CIFAR-10, CIFAR-100, and ILSVRC-2012.
• The approach balances efficiency and accuracy, making CNNs more practical for resource-constrained applications without significant performance loss.

Analysis of "More is Less: A More Complicated Network with Less Inference Complexity"

The research paper "More is Less: A More Complicated Network with Less Inference Complexity" introduces an innovative approach to accelerating convolutional neural networks (CNNs) by enhancing their architecture with a novel layer. Specifically, the proposed model adds a low-cost collaborative layer (LCCL) to each convolutional layer, aiming for significant decreases in inference complexity while maintaining effective prediction accuracy.

Core Concept and Implementation

The pivotal aspect of this paper is the introduction of LCCL, which complements each existing convolutional layer in a CNN. The method relies on equipping the CNN with these additional low-cost layers, which work in tandem with the original layers to reduce computation. The LCCL is constructed either as a $1 \times 1$ convolution or as a single shared filter across channels. The multiplicative interaction between the ReLU-activated outputs from the original layer and the LCCL allows zero-value outputs from the LCCL to eliminate unnecessary calculations in the corresponding original layer. This process efficiently decreases computational overhead, particularly for layers with sparse activations post-ReLU.

Methodology and Technical Advancements

The LCCL's implementation intelligently leverages sparse activations, a property inherently present in ReLU layers, to skip computations of those elements in the convolutional outputs that contribute insignificantly to the final decision layer. The LCCN architecture thus marks a departure from conventional sparsity exploitation methods, which may compromise accuracy due to predefined thresholds or integration as a regularizer, by dynamically determining zero contributions during inference.

Training the LCCN requires standard methodologies such as Stochastic Gradient Descent (SGD) with backpropagation, wherein the model is optimized by using the LCCL’s output to guide the backpropagation process. Extensive experiments on standard datasets, CIFAR-10, CIFAR-100, and ILSVRC-2012, demonstrate that the LCCN architecture can accelerate inference by over 32% on average with minimal performance degradation.

Comparative Analysis and Results

This approach distinguishes itself from traditional methods like low-rank approximation, fixed-point arithmetic, and product quantization. The LCCN advances a distinct layer-based acceleration strategy, offering a pragmatic balance between computational efficiency and model performance. Notably, the comparisons show that some networks even gain accuracy due to the LCCN’s reduced risk of overfitting, facilitated by its effective sparsity exploitation.

On complex networks such as pre-activation ResNet variants and Wide Residual Networks (WRNs), the experimental evidence supports substantial FLOPs reduction. For instance, the ResNet-110, when enhanced with LCCL, achieves a 34% speedup with a negligible increase in top-1 error, thereby proving the practicality of the method in deploying CNNs on resource-constrained devices like mobile platforms.

Implications and Future Directions

The implications of this work are significant for the deployment of CNNs in environments with limited computational resources. By reducing computation without altering model accuracy considerably, LCCN offers a pathway to more computationally efficient neural architectures. Moreover, this technique is adaptable to a wide range of tasks beyond image classification, such as detection and segmentation, due to its generic applicability to convolutional operations.

For future research, improvements in realistic speedup realization and integration with other acceleration strategies like fixed-point and pruning methods provide promising directions. Also, further development might explore the automated design of LCCL structures via neural architecture search methodologies, optimizing both their placement and their collaborative function across diverse network types, thus maximizing computational savings while maintaining robustness and generalization ability.

In summary, the LCCL-concept represents a significant contribution to the field of efficient deep learning architectures by providing a flexible, computationally feasible approach to maintain high performance in high-demand applications.