Neural Networks Fail to Learn Periodic Functions and How to Fix It (2006.08195v2)

Abstract: Previous literature offers limited clues on how to learn a periodic function using modern neural networks. We start with a study of the extrapolation properties of neural networks; we prove and demonstrate experimentally that the standard activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions. We hypothesize that this is due to their lack of a "periodic" inductive bias. As a fix of this problem, we propose a new activation, namely, $x + \sin^2(x)$, which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the ReLU-based activations. Experimentally, we apply the proposed method to temperature and financial data prediction.

• The paper demonstrates that standard activation functions lack a periodic inductive bias, leading to poor extrapolation of periodic functions.
• It introduces the Snake activation function, defined as x + sin²(x), which combines favorable optimization with periodic modeling.
• Empirical tests on temperature and financial datasets validate the new function’s superior performance in capturing periodic trends.

Insights into Learning Periodic Functions with Neural Networks

The paper "Neural Networks Fail to Learn Periodic Functions and How to Fix It" presents a thorough examination of the inability of standard neural network architectures to accurately learn and extrapolate periodic functions. The authors, Liu Ziyin, Tilman Hartwig, and Masahito Ueda, provide both theoretical insights and empirical evidence to support their claims. Notably, this paper identifies a specific weakness in conventional activation functions and suggests an innovative activation mechanism to address this deficit.

Overview

The authors begin their investigation by highlighting the significance of periodic functions across various domains such as natural sciences, human biology, and economics. Recognizing that neural networks are generally effective at interpolating data within the training range, the paper points out their deficiency in extrapolating periodic functions. This limitation stems from the absence of a natural periodic inductive bias in conventional activation functions like ReLU, tanh, and sigmoid.

Their primary contribution includes a detailed analysis of how standard activation functions fail to capture periodic patterns in data. Through both theoretical proofs and experiments, the authors demonstrate that these functions cannot extrapolate periodic behavior due to their inherent properties. ReLU networks, for instance, tend to extrapolate linearly, while tanh networks converge towards constant values outside the training zone.

Proposed Solution

The paper proposes a novel activation function, $x + \sin^2(x)$, termed "Snake," which embodies a periodic inductive bias. This function maintains the favorable optimization characteristics of ReLU while being predisposed to model periodic functions. The authors rigorously validate this function through experiments on temperature datasets and financial data predictions, showcasing superior performance in learning periodic patterns compared to traditional activation functions.

Theoretical Contributions

The authors establish two universal extrapolation theorems that formally describe the extrapolation behavior of networks with ReLU and tanh activation functions. They demonstrate that neural networks with these activations inherently fail to extrapolate periodic functions because of the asymptotic properties of the activations.

For the proposed Snake function, the paper also introduces a "Universal Extrapolation Theorem" asserting that neural networks using this activation can approximate any periodic function, offering both point-wise convergence and uniform approximation.

Experimental Validation

Experimental results validate the theoretical findings, highlighting the shortcomings of existing methods in extrapolating periodic behaviors. In practical applications, such as body temperature and financial market predictions, networks employing Snake demonstrate notable improvements in capturing periodic trends, outperforming both traditional networks and other periodic function-based approaches.

Implications and Future Directions

The proposed Snake function signifies an important development in designing neural networks capable of learning periodic functions. The findings hold substantial implications for fields requiring periodic extrapolation, such as climate modeling and economic forecasting. This work suggests that activation functions tailored to specific problem domains can significantly enhance the generalization and predictive capabilities of neural networks.

Future avenues for research include exploring other architectural adaptations or parameter tuning strategies that might further improve the capability of neural networks to model periodicity. Moreover, integrating the Snake function in recurrent architectures presents an interesting opportunity to extend its application into more complex time-dependent tasks.

In conclusion, this paper provides a compelling case for revisiting the design of activation functions in neural networks, underscoring the critical role they play in modeling extrapolative properties. The introduction of Snake offers a promising path forward in the tailored development of neural networks to meet specific functional demands.