Learning Activation Functions to Improve Deep Neural Networks (1412.6830v3)

Abstract: Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural network architectures composed of static rectified linear units, achieving state-of-the-art performance on CIFAR-10 (7.51%), CIFAR-100 (30.83%), and a benchmark from high-energy physics involving Higgs boson decay modes.

• The paper introduces adaptive piecewise linear (APL) activation functions that are learned through gradient descent to boost network accuracy.
• APL units reduce error rates on benchmarks like CIFAR-10 and CIFAR-100, outperforming traditional activations and models like maxout with fewer parameters.
• The study underscores both practical and theoretical benefits, highlighting improved performance and potential for future adaptive neural architectures.

Learning Activation Functions to Improve Deep Neural Networks

This paper presents a method for enhancing deep neural network performance through adaptive piecewise linear activation functions. Traditionally, neural networks use fixed activation functions like ReLU, tanh, or sigmoid, which significantly influence learning dynamics and expressiveness. The authors propose a more flexible approach by enabling these functions to be learned independently at each neuron using gradient descent.

The Adaptive Piecewise Linear Unit

The introduced adaptive piecewise linear (APL) activation unit is a composition of hinge-shaped functions, mathematically expressed as:

$$h_i(x) = \max(0,x) + \sum_{s=1}^{S}a_{i}^{s}\max(0,-x+b_{i}^{s})$$

Here, each APL unit comprises a hyperparameter $S$ (the number of hinges), and parameters $a^s_i$ and $b^s_i$, which are optimized during training. The versatility of APL units lies in their ability to represent both convex and non-convex functions, constrained to behave linearly as $x \to \infty$ or $x \to -\infty$.

Numerical Performance and Benchmarks

The introduction of APL units led to a marked improvement in several benchmarks:

• CIFAR-10 error rates decreased from 12.56% using ReLU to 11.38% with APL units.
• CIFAR-100 performance improved similarly, from 37.34% to 34.54%.
• On high-energy physics tasks, notably the Higgs boson decay dataset, APL units outperformed existing models, delivering a higher AUC and discovery significance than ensembles of traditional networks.

These improvements underscore the potential of learned activation functions to outperform fixed ones across diverse domains.

Comparative Analysis

Comparisons with other advanced activation functions, such as maxout and network-in-network (NIN), reveal that APL units achieve similar representational capabilities with substantially fewer parameters. This efficiency facilitates applications like convolutional networks, where APL units can uniquely define nonlinearities per feature map point with computational tractability.

Practical and Theoretical Implications

The theoretical foundation of this work suggests that neural networks can benefit from more diverse activation functions tailored to specific neuron-level responses. Practically, this adaptability could lead to more robust networks with heightened accuracy and efficiency.

The success of APL units opens pathways for future research into dynamically learning other network components, such as more complex or hierarchical function structures.

Conclusion

This research emphasizes the significance of adaptable activation functions in neural networks. By learning these functions via gradient descent, it improves network performance without substantially increasing complexity. Future work may explore expanding these methods to other network substructures, potentially ushering in a new class of adaptive neural architectures.