Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark (2109.14545v3)

Abstract: Neural networks have shown tremendous growth in recent years to solve numerous problems. Various types of neural networks have been introduced to deal with different types of problems. However, the main goal of any neural network is to transform the non-linearly separable input data into more linearly separable abstract features using a hierarchy of layers. These layers are combinations of linear and nonlinear functions. The most popular and common non-linearity layers are activation functions (AFs), such as Logistic Sigmoid, Tanh, ReLU, ELU, Swish and Mish. In this paper, a comprehensive overview and survey is presented for AFs in neural networks for deep learning. Different classes of AFs such as Logistic Sigmoid and Tanh based, ReLU based, ELU based, and Learning based are covered. Several characteristics of AFs such as output range, monotonicity, and smoothness are also pointed out. A performance comparison is also performed among 18 state-of-the-art AFs with different networks on different types of data. The insights of AFs are presented to benefit the researchers for doing further research and practitioners to select among different choices. The code used for experimental comparison is released at: \url{https://github.com/shivram1987/ActivationFunctions}.

• The paper systematically categorizes and benchmarks 18 activation functions to guide the selection of optimal neural network architectures.
• Empirical analyses on datasets like CIFAR10 and CIFAR100 reveal that parametric functions such as PReLU and PDELU achieve faster convergence and improved accuracy.
• The study recommends default use of ReLU for its simplicity while advocating adaptive functions for tasks that require nuanced control over non-linearity and noise robustness.

Overview of Activation Functions in Deep Learning

The paper, "Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark," offers a detailed examination of activation functions (AFs), a critical component used in neural network architectures. The aim of the work is to provide a comprehensive understanding of the role and efficacy of various AFs in transforming non-linearly separable input data into linearly separable abstract features, a necessity for efficient data classification and pattern recognition.

Detailed Classification and Evaluation

Activation functions are the nonlinear components of neural networks that allow for complex mappings. The authors categorize these functions into several classes, such as Logistic Sigmoid/Tanh, ReLU-based, ELU-based, and Learning/adaptive-based functions, each with distinct characteristics including monotonicity, smoothness, and boundedness. The survey evaluates 18 state-of-the-art AFs across a variety of neural networks and datasets, offering a nuanced appraisal of the strengths and weaknesses of each function.

1. Traditional Functions: Among the oldest are Logistic Sigmoid and Tanh functions, which, while offering smooth and monotonic outputs, suffer from vanishing gradient problems that can stymie learning in deeper architectures. Several enhancements have been proposed to address these, but most retain the inherent complexities of the original functions.
2. Rectified Linear Unit Variants: ReLU and its variants owe their popularity to simplicity and computational efficiency. However, challenges such as the utilization of negative values, limited nonlinearity, and unbounded output have led to the development of alternatives like LReLU, PReLU, and others. Experiments highlight the continued relevance of ReLU especially for networks like VGG16 and GoogleNet, while \Derived versions such as LReLU show advantages in residual network contexts.
3. Exponential and Adaptive Functions: Exponential Linear Unit (ELU) and its variants aim to better utilize negative inputs without sacrificing training stability. Meanwhile, adaptive functions such as Swish introduce learnable parameters to add flexibility, enabling the model to self-tune non-linearity based on particular dataset characteristics. These functions are a focus of recent research due to their potential to balance adaptability with model complexity.
4. Miscellaneous Functions: The paper also looks at less conventional AFs, like Softplus and probabilistic AFs, emphasizing how these approaches attempt to incorporate properties like boundedness and stochasticity to achieve specific objectives, such as robustness to noise or mission-critical efficiency in hardware implementations.

Empirical Analysis and Performance Implications

The paper's empirical analysis is particularly robust, employing datasets such as CIFAR10 and CIFAR100 for image classification, alongside text and speech data, to benchmark AF performance. A few standout observations include:

• Image Classification: While ReLU maintains high accuracy across many networks, recent AFs like Mish and PDELU provide competitive or superior performance depending on network architecture.
• Convergence Trends: AFs with learnable parameters (like PAU and PReLU) show faster convergence, benefiting high-complexity data scenarios.
• Training Efficiency: Recognizing the trade-off between training time and performance remains crucial, with functions like ELU and Softplus appearing to strike a favorable balance.

Recommendations for Practice

The authors suggest practical guidelines for selecting activation functions depending on network architecture and dataset. For instance, ReLU remains a strong default choice due to its simplicity, but parametric functions like PReLU and PDELU can offer adaptive advantages without substantially increasing computational load. In specific tasks like language translation and speech recognition, functions like SELU and PReLU are preferred.

Conclusion

The paper provides an invaluable resource for deep learning practitioners and researchers by systematically categorizing, evaluating, and comparing a comprehensive array of activation functions within neural networks. Through its detailed surveys and empirical findings, it guides informed decisions in selecting suitable AFs to match network architecture and data types, thus furthering advancements in neural network design and functionality. The integration of numerical analyses substantiates claims, making the work a critical reference for future research and application in AI development.