Chebyshev Feature Neural Network for Accurate Function Approximation

Abstract: We present a new Deep Neural Network (DNN) architecture capable of approximating functions up to machine accuracy. Termed Chebyshev Feature Neural Network (CFNN), the new structure employs Chebyshev functions with learnable frequencies as the first hidden layer, followed by the standard fully connected hidden layers. The learnable frequencies of the Chebyshev layer are initialized with exponential distributions to cover a wide range of frequencies. Combined with a multi-stage training strategy, we demonstrate that this CFNN structure can achieve machine accuracy during training. A comprehensive set of numerical examples for dimensions up to $20$ are provided to demonstrate the effectiveness and scalability of the method.

• The paper's main contribution is the development of the Chebyshev Feature Neural Network (CFNN) that uses a Chebyshev feature layer and multi-stage training to achieve near-machine precision.
• The CFNN architecture integrates Chebyshev functions with learnable frequencies to cover a wide range of frequency domains for superior approximation of both smooth and discontinuous functions.
• Extensive numerical experiments in multiple dimensions demonstrate CFNN's scalability and its significant potential in high-precision scientific computing applications.

Chebyshev Feature Neural Network for Accurate Function Approximation

The presented paper introduces a novel Deep Neural Network (DNN) architecture termed Chebyshev Feature Neural Network (CFNN), which is designed to achieve highly accurate function approximation, potentially reaching machine precision. This architecture strategically incorporates Chebyshev functions with learnable frequencies as the initial hidden layer while following it up with standard fully connected layers.

Introduction to CFNN

The primary motivation behind CFNN is the evident limitations of traditional DNNs in scientific computing applications where high precision is paramount. While DNNs are theoretically capable of approximating functions with arbitrary precision—thanks to the universal approximation theorem—their practical application often falls short, plateauing at accuracy levels between $\mathcal{O}(10^{-5})$ and $\mathcal{O}(10^{-2})$. This is insufficient for numerous scientific computing tasks, driving the need for advanced architectures like CFNN.

The core innovation in CFNN is the use of Chebyshev functions in the first hidden layer, initialized with an exponential distribution of frequencies. This initialization ensures a wide coverage of frequency domains, aiding in capturing both low and high-frequency components of the target function. Moreover, CFNN employs a multi-stage training strategy, which iteratively minimizes the residuals of the function approximation, thus progressively enhancing the accuracy with each stage.

Architectural Details

CFNN is structured as follows:

1. Chebyshev Feature Layer: The first hidden layer uses generalized Chebyshev polynomials, extending their degrees from integers to positive real numbers. This generalization results in Chebyshev features, aiding in superior function approximations due to their orthogonal properties.
2. Subsequent Fully Connected Layers: The layers following the Chebyshev Feature Layer are standard feedforward layers that employ traditional activation functions such as $\tanh$.
3. Initialization Strategy: Weights in the Chebyshev layer are sampled using an exponential distribution, which varies across training stages to emphasize different frequency components at each stage.
4. Multi-stage Training: Each stage trains the network on the residuals from the previous stage, scaled appropriately to manage their magnitudes. This iterative refinement ensures the reduction of errors across training stages, pushing the precision close to machine limits.

Numerical Examples and Effectiveness

The paper presents an extensive set of numerical experiments spanning multiple dimensions (up to 20). For one-dimensional examples, CFNN achieves approximation errors as low as $\mathcal{O}(10^{-14})$ for smooth functions and around $\mathcal{O}(10^{-6})$ to $\mathcal{O}(10^{-5})$ for nonsmooth and discontinuous functions. In higher dimensions, functions defined in $[-1,1]^d$ (for $d=2, 5, 10, 20$) were tested, demonstrating CFNN's scalability and its ability to maintain high approximation accuracy.

Implications and Future Directions

The implications of CFNN are substantial for scientific computing fields that require high-precision function approximations, such as solving differential equations, inverse problems, and operator learning. CFNN's innovative architecture effectively integrates principles of classical approximation theory into modern neural networks, thus expanding the toolkit available for scientific machine learning.

Looking forward, several paths for further research and development emerge:

• Extending CFNN to Other Domains: Investigating the use of CFNN in broader classes of scientific computing problems, particularly those involving complex boundary conditions and heterogeneous domains.
• Optimization of Training Procedures: Further refinement of the multi-stage training process and exploration of other initialization strategies could yield even better convergence properties.
• Integration with Other Methods: Combining CFNN with other advanced neural network architectures and optimization techniques could lead to hybrid models with even greater accuracy and applicability.

In conclusion, CFNN represents a significant step forward in the domain of high-precision function approximation using neural networks. The architecture's efficacy in achieving near-machine precision opens new avenues for application in scientifically demanding fields, setting the stage for future advancements in both theory and practice.