Learning the Number of Neurons in Deep Networks (1611.06321v3)

Abstract: Nowadays, the number of layers and of neurons in each layer of a deep network are typically set manually. While very deep and wide networks have proven effective in general, they come at a high memory and computation cost, thus making them impractical for constrained platforms. These networks, however, are known to have many redundant parameters, and could thus, in principle, be replaced by more compact architectures. In this paper, we introduce an approach to automatically determining the number of neurons in each layer of a deep network during learning. To this end, we propose to make use of structured sparsity during learning. More precisely, we use a group sparsity regularizer on the parameters of the network, where each group is defined to act on a single neuron. Starting from an overcomplete network, we show that our approach can reduce the number of parameters by up to 80\% while retaining or even improving the network accuracy.

• The paper introduces a structured sparsity regularizer that determines optimal neuron counts during training, reducing over-parameterization.
• The method achieved up to 80% parameter reduction and a 1.6% accuracy improvement on ImageNet, alongside significant speedups on ICDAR tasks.
• Integrating neuron selection into training streamlines architecture optimization, offering a single-step process with broader potential in deep learning.

An Examination of "Learning the Number of Neurons in Deep Networks"

The paper authored by Alvarez and Salzmann introduces a novel approach to determining the optimal number of neurons within each layer of a deep neural network. Recognizing that the model's complexity—and therefore its performance—is highly dependent on the chosen architecture, which is usually heuristically determined, their method addresses the inefficiencies of over-parameterization. This is particularly relevant for deploying deep networks on platforms with constrained computational resources.

The authors propose a technique that leverages structured sparsity, applying a group sparsity regularizer to the network parameters. Each group corresponds to the parameters associated with a single neuron. By incorporating this regularizer during the training phase, the method effectively cancels out neurons with negligible influence, thereby reducing redundancy in the network. This technique differs from traditional destructive model selection methods, which typically analyze parameters individually and do not scale well to large networks.

Significant experimental results underscore the efficacy of the proposed method. Applied to several network architectures, such as variants of the widely-used VGG (VGG-B) and other networks like DecomposeMe, the authors demonstrate a parameter reduction of up to 80% while maintaining, and in some instances even improving, classification accuracy. For example, on the ImageNet dataset, they reported a 1.6% increase in accuracy for their modified BNet$^C$ network compared to the baseline while reducing the parameter count significantly.

A key aspect of the paper is the performance and functionality gains at test time. The proposed model selection method leads to a more compact, and therefore faster, network; in some instances, such as the character recognition task evaluated on the ICDAR dataset, a speedup of nearly 50% was achieved. This reduction in computational load is crucial for the practical deployment of deep learning models, particularly when memory and processing power are limited.

The research brings to focus the potential of regularizer-based techniques in addressing network complexity. Unlike post-processing strategies that require a fully trained network before pruning can take place, this approach integrates neuron selection into the network training process itself. This offers a single-step model selection and training procedure, which could potentially enhance the generalization capabilities, as indicated by a noted reduction in the gap between training and validation accuracy.

Theoretical implications suggest that the use of structured sparsity for neuron selection could be extended beyond image classification tasks. Future work may explore its applicability to other neural network applications, including regression tasks and autoencoders, as well as adaptations where whole layers might be deemed redundant.

In summation, the paper presents a comprehensive paper on optimizing deep network architectures through automatic neuron selection. By reducing the networks' size during training, the proposed method contributes to efficient deep learning models that could influence future advances in both theoretical developments and practical applications of artificial intelligence. As deep learning continues to evolve, it is critical to refine the architectural design of neural networks, and this work offers a step forward in automated architecture optimization.