Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (1511.07289v5)

Abstract: We introduce the "exponential linear unit" (ELU) which speeds up learning in deep neural networks and leads to higher classification accuracies. Like rectified linear units (ReLUs), leaky ReLUs (LReLUs) and parametrized ReLUs (PReLUs), ELUs alleviate the vanishing gradient problem via the identity for positive values. However, ELUs have improved learning characteristics compared to the units with other activation functions. In contrast to ReLUs, ELUs have negative values which allows them to push mean unit activations closer to zero like batch normalization but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit natural gradient because of a reduced bias shift effect. While LReLUs and PReLUs have negative values, too, they do not ensure a noise-robust deactivation state. ELUs saturate to a negative value with smaller inputs and thereby decrease the forward propagated variation and information. Therefore, ELUs code the degree of presence of particular phenomena in the input, while they do not quantitatively model the degree of their absence. In experiments, ELUs lead not only to faster learning, but also to significantly better generalization performance than ReLUs and LReLUs on networks with more than 5 layers. On CIFAR-100 ELUs networks significantly outperform ReLU networks with batch normalization while batch normalization does not improve ELU networks. ELU networks are among the top 10 reported CIFAR-10 results and yield the best published result on CIFAR-100, without resorting to multi-view evaluation or model averaging. On ImageNet, ELU networks considerably speed up learning compared to a ReLU network with the same architecture, obtaining less than 10% classification error for a single crop, single model network.

• The paper demonstrates that ELUs enable faster convergence by reducing bias shifts and maintaining mean activations closer to zero.
• It proposes using ELUs' negative saturation to stabilize learning and enhance noise robustness in deep neural networks.
• Empirical evaluations on MNIST, CIFAR-10/100, and ImageNet showcase ELUs' superior performance with lower error rates and quicker training.

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

This paper, authored by Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter from the Institute of Bioinformatics at Johannes Kepler University, Linz, Austria, presents the Exponential Linear Unit (ELU) as an activation function aimed at addressing several critical challenges in deep neural network training. ELUs are compared favorably to the prevalent Rectified Linear Units (ReLUs), Leaky ReLUs (LReLUs), and Parametric ReLUs (PReLUs), demonstrating both faster learning and improved performance.

Introduction to Activation Functions

The dominant activation function in contemporary neural network models is the ReLU, which is defined as the identity for positive inputs and zero otherwise. ReLUs alleviate the vanishing gradient problem by ensuring the gradient is non-contracting for positive values. However, ReLUs and their variants suffer from two primary issues:

1. Non-zero Mean Activation: Positive-valued activations induce a bias shift (mean shift) in subsequent layers.
2. Information Loss: The non-negative nature of ReLUs prohibits negative activations, leading to biased representations.

ELUs propose a solution to these limitations by introducing both identity activation for positive inputs and a saturating negative value for negative inputs.

Theoretical Framework

ELUs are defined as: $$f(x) = \begin{cases} x & \text{if } x > 0 \ \alpha (\exp(x) - 1) & \text{if } x \leq 0 \end{cases}$$

Here, $\alpha$ is a hyperparameter controlling the saturation level for negative inputs. This formulation brings the following advantages:

1. Mean Activation Closer to Zero: Negative values for negative inputs help counterbalance the positive activations, thereby pushing the mean activation towards zero.
2. Reduced Bias Shift: By decreasing the bias shift effect, the standard gradient approximates the natural gradient more closely, facilitating faster learning.
3. Noise Robustness: Saturation to a negative value decreases forward propagated variation, enhancing noise-robust deactivation states.

Empirical Results

The authors conducted comprehensive experiments on several benchmark datasets, such as MNIST, CIFAR-10, CIFAR-100, and ImageNet. They compared ELUs with ReLUs, LReLUs, and ReLU-based networks augmented with batch normalization. The results consistently showed that ELUs not only sped up the learning process but also significantly improved generalization performance.

Key Results

1. Faster Convergence: On the MNIST dataset, ELUs maintained smaller median activations throughout the training process, leading to a rapid decrease in training error.
2. Superior Performance: For CIFAR-100, ELUs achieved a test error of 24.28%, outperforming the best-known results without multi-view evaluation or model averaging. On CIFAR-10, ELUs rated among the top-performing models.
3. Scalability: On ImageNet, ELU networks reached below 10% top-5 classification error with a significant reduction in required training iterations compared to ReLU networks.

Implications and Future Directions

The introduction of ELUs as an activation function has several practical and theoretical implications:

• Faster Training: ELUs reduce training time by mitigating bias shifts, which is particularly beneficial for large datasets and complex architectures.
• Improved Representations: By allowing negative activations, ELUs help in learning more balanced and less biased representations.
• Noise Robustness: Saturating negative outputs contribute to more stable and noise-resistant activation patterns.

Looking forward, ELUs hold promise for various advancements in AI:

• Optimized Implementations: Further improvements in computational efficiency for ELU functions can enhance their applicability in real-time systems.
• Extended Applicability: ELUs can be explored in domains beyond vision, such as natural language processing, where robust and efficient training is crucial.

Conclusion

The paper's comprehensive analysis and extensive empirical validation posit ELUs as a valuable tool for enhancing deep learning architectures. By addressing critical issues associated with traditional activation functions, ELUs pave the way for more efficient and effective machine learning models.