Activation Functions: Dive into an optimal activation function (2202.12065v1)

Abstract: Activation functions have come up as one of the essential components of neural networks. The choice of adequate activation function can impact the accuracy of these methods. In this study, we experiment for finding an optimal activation function by defining it as a weighted sum of existing activation functions and then further optimizing these weights while training the network. The study uses three activation functions, ReLU, tanh, and sin, over three popular image datasets, MNIST, FashionMNIST, and KMNIST. We observe that the ReLU activation function can easily overlook other activation functions. Also, we see that initial layers prefer to have ReLU or LeakyReLU type of activation functions, but deeper layers tend to prefer more convergent activation functions.

• The paper demonstrates that an optimal activation function can be obtained by dynamically weighting ReLU, tanh, and sin during network training.
• It employs a cyclic strategy that alternates updates between conventional network parameters and activation weight adjustment to achieve convergence.
• Experimental results reveal that early layers favor ReLU variants while deeper layers benefit from smoother functions like tanh and sin for stability.

Analyzing the Impact of Weighted Activation Functions in Neural Networks

Introduction

The research paper, "Activation Functions: Dive into an optimal activation function," explores the potential to enhance the performance of neural networks by optimizing activation functions. Activation functions are critical in introducing non-linearity into neural networks, thus enabling them to model complex patterns within data. A variety of activation functions exist, each with its own characteristics that impact the learning and performance of neural networks. This paper proposes a methodology to derive an optimal activation function through a weighted sum approach of several existing functions.

Methodology

The paper centers on leveraging a linear combination of three widely-used activation functions: ReLU, tanh, and sin. The main innovation is the dynamic optimization of the weights associated with each activation function as part of the network training process. The experiments are carried out on three benchmark image datasets: MNIST, FashionMNIST, and KMNIST, employing a convolutional neural network model with two convolutional layers and two fully connected layers.

The network training procedure is characterized by a cyclic strategy. The initial cycle involves training traditional network parameters while holding the activation function weights constant. Subsequently, the procedure involves fixing the layer parameters and optimizing the activation weights. This cycle is repeated to ensure convergence towards an optimal set of activation weights. Optimization is performed using the Adam optimizer with prescribed learning rates across three distinct phases.

Results

The conducted experiments offer insights into the preferred composition of activation functions across different network layers and datasets. The results indicate a tendency for initial network layers to favor ReLU or its variant, LeakyReLU, due to their efficient gradient propagation and sparsity characteristics. In deeper layers, however, a gradual shift towards more convergent activation functions such as tanh or sin is observed, which can be attributed to their smoothing characteristics that assist in stabilizing the learning process of deeper network sections.

The experimental findings are formalized through specific weight distributions across the activation functions per layer. For instance, the preliminary layer of MNIST allocates significant emphasis to ReLU function ($P_1 = 0.4848$), whereas in deeper layers, sin ($P_3 = 0.6283$) emerges as more dominant. Such empirical data reinforce the concept that the role of activation functions is dynamic and context-dependent within the network architecture.

Implications and Future Directions

This paper underscores the potential advantages of customizing activation functions in neural networks beyond static singular use. The methodology expands the toolbox available for deep learning practitioners, suggesting that optimal activation configurations may vary not only by network architecture but also by the nature of the tasks being addressed.

Future research can build on these findings by incorporating a broader range of activation functions into the weighted combination framework or by extending the approach to other types of neural network architectures. Moreover, exploring alternative combinations, such as multiplicative interactions of activation functions, may uncover further efficiency gains.

Conclusively, this approach presents a promising direction for making neural networks more adaptable to diverse datasets and tasks by judiciously tuning one of their core components, the activation functions, thereby optimizing their capabilities to generate robust and accurate predictive models.