Parametric Exponential Linear Unit for Deep Convolutional Neural Networks (1605.09332v4)

Abstract: Object recognition is an important task for improving the ability of visual systems to perform complex scene understanding. Recently, the Exponential Linear Unit (ELU) has been proposed as a key component for managing bias shift in Convolutional Neural Networks (CNNs), but defines a parameter that must be set by hand. In this paper, we propose learning a parameterization of ELU in order to learn the proper activation shape at each layer in the CNNs. Our results on the MNIST, CIFAR-10/100 and ImageNet datasets using the NiN, Overfeat, All-CNN and ResNet networks indicate that our proposed Parametric ELU (PELU) has better performances than the non-parametric ELU. We have observed as much as a 7.28% relative error improvement on ImageNet with the NiN network, with only 0.0003% parameter increase. Our visual examination of the non-linear behaviors adopted by Vgg using PELU shows that the network took advantage of the added flexibility by learning different activations at different layers.

• The paper introduces a parameterized ELU that learns activation shapes via back-propagation, enhancing CNN performance over the standard ELU.
• The methodology adds only 2L extra parameters, achieving up to a 7.28% error reduction on benchmarks like ImageNet with minimal overhead.
• Empirical evaluations on datasets including MNIST, CIFAR, and ImageNet reveal activation flexibility and expose the adverse effects of batch normalization with PELU.

Parametric Exponential Linear Unit for Deep Convolutional Neural Networks

The paper titled "Parametric Exponential Linear Unit for Deep Convolutional Neural Networks" explores the implementation of a parameterized variant of the Exponential Linear Unit (ELU) activation function—designated as the Parametric ELU (PELU)—to enhance the performance of Convolutional Neural Networks (CNNs) in object recognition tasks. This detailed exploration is driven by the need to address the existing limitations of the standard ELU, specifically the manual configuration of parameters that might not universally optimize performance across different network architectures and datasets. The PELU presents a novel approach by allowing the network to learn the ideal activation shape dynamically through back-propagation during the training phase.

Core Contributions

1. Parameterization Approach: The authors design a parameterized function that captures the flexibility to modify different aspects of the activation function, such as saturation point, exponential decay, and slope, through continuously adjustable parameters. This preserves differentiability and minimizes computational overhead by adding minimal extra parameters (only $2L$, where $L$ is the layer count).
2. Empirical Evaluation: The PELU was subjected to rigorous testing across a range of datasets including MNIST, CIFAR-10/100, and ImageNet, using popular CNN architectures such as ResNet, NiN, Overfeat, and Vgg. The empirical results consistently favored PELU over ELU, promoting enhanced model convergence and reduced error rates—achieving up to a 7.28% relative error improvement with a negligible parameter increase of 0.0003% on the ImageNet dataset using the NiN architecture.
3. Batch Normalization (BN) Effects: Interestingly, the research highlights that incorporating BN prior to PELU activation detrimentally impacted performance, contrasting with ReLU activation where BN proves beneficial. This observation invites a dialogue on the compatibility and impact of BN with varied activation functions, and the reasons therein.
4. Parameter Configuration Experiments: Through computational experimentation, this paper reveals that the proposed configuration $(a, 1/b)$ significantly outperforms alternative configurations (e.g., $(1/a, b)$), primarily by supporting better non-linear behavior and model convergence.
5. Activation Flexibility: The visual inspection of Vgg network progression underscores the dynamic optimation of layer activation shapes, with PELU facilitating diverse non-linear behaviors, suggesting a tailored fitment to task-specific representation demands.

Implications for Future AI Research

The introduction of a dynamically learned activation shape proffers palpable benefits in CNN efficacy, potentially influencing future research directions in neural network design, particularly around learning adaptable network components that co-train with broader model parameters. This approach can inspire further modifications and evaluations of other activation functions, driving innovation toward automated and self-optimizing neural architectures.

Furthermore, the peculiar relationship between PELU, ELU, and BN opens up new questions on internal covariate shift management and scaling invariance roles—an area ripe for further exploration to align activation function development with architectural enhancements. The exploration of how PELU interacts with other architectural techniques, particularly in deeper networks, might yield additional performance insights and optimizations in the endeavor to develop generalizable and efficient neural networks for diverse vision applications.