The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems (1710.00211v1)

Abstract: We propose a deep learning based method, the Deep Ritz Method, for numerically solving variational problems, particularly the ones that arise from partial differential equations. The Deep Ritz method is naturally nonlinear, naturally adaptive and has the potential to work in rather high dimensions. The framework is quite simple and fits well with the stochastic gradient descent method used in deep learning. We illustrate the method on several problems including some eigenvalue problems.

• The paper introduces the Deep Ritz Method, leveraging deep neural networks to approximate trial functions for variational problems.
• It demonstrates high accuracy and parameter efficiency in solving both 2D and high-dimensional Poisson equations compared to traditional finite difference methods.
• The method employs SGD and numerical quadrature, while its transfer learning strategy highlights potential for broader applications in PDEs and eigenvalue problems.

The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems

Introduction

In their paper, Weinan E and Bing Yu propose the Deep Ritz Method, a deep learning-based numerical algorithm for solving variational problems. This method targets the numerical solution of variational problems that frequently arise in the context of partial differential equations (PDEs). The approach leverages the natural adaptivity and high-dimensional capabilities of deep neural networks (DNNs), seamlessly integrating with the stochastic gradient descent (SGD) methods ubiquitously used in deep learning.

The Framework of the Deep Ritz Method

The Deep Ritz Method consists of three core components:

1. Deep Neural Network-Based Approximation: The trial function is represented via a deep neural network. The non-linear and compositional nature of DNNs is particularly advantageous for high-dimensional function approximations.
2. Numerical Quadrature Rule for Functionals: To compute functionals, numerical quadrature is employed. This involves discretizing continuous sums over data points, which are modelled as integral points.
3. Optimization Algorithm: The method utilizes SGD for optimization, a practice widely recognized for its efficiency in machine learning-based problems.

Construction of Trial Functions

The trial function $u(x; \theta)$ is expressed via a neural network transformation $z_\theta(x)$. Each layer of the network consists of blocks incorporating linear transformations, activation functions, and residual connections, facilitating efficient training and mitigating issues like the vanishing gradient problem. The activation function $\phi(x)=\max\{x^3,0\}$ balances simplicity and accuracy.

Optimization Framework

The functional $I(u)$ is minimized using an SG-based integral approximation. The stochastic nature of SGD aligns well with the random sampling of points required in numerical quadrature, thus promoting a smoother convergence trajectory.

Numerical Experiments

Poisson Equation in Two Dimensions

For the two-dimensional Poisson equation, the Deep Ritz Method demonstrates superior accuracy with fewer parameters compared to traditional finite difference methods (FDMs). The method successfully resolves corner singularities and captures the solution's complex behavior, benefiting from the intrinsic adaptivity of neural networks. Relative $L_2$ errors highlight the efficiency of the Deep Ritz Method over FDM, especially in terms of parameter efficiency.

High-Dimensional Poisson Equations

The method's performance was evaluated on 10-dimensional Poisson equations, achieving a relative $L_2$ error of approximately 0.4% after 50,000 iterations. When extended to 100 dimensions, results indicated a relative error of about 2.2%, underscoring the method's robustness and efficacy in high-dimensional contexts.

Variational Problems with Neumann Boundary Conditions

For problems with Neumann boundary conditions, the method effectively utilized a functional formulation without penalty terms for boundary constraints, achieving promising accuracy levels across different dimensions. This flexibility is essential for addressing a variety of physical and engineering problems.

Transfer Learning

The research explored the benefits of transfer learning by reusing network weights from similar problems. This strategy significantly accelerated the training process, particularly in initial stages, demonstrating transfer learning's potential in variational problem-solving.

Eigenvalue Problems

The Deep Ritz Method was applied to eigenvalue problems arising in quantum mechanics. Utilizing a variation of the Rayleigh quotient, the method achieved low errors for both infinite potential wells and harmonic oscillators across multiple dimensions. However, the method's performance suffered as dimensionality increased, indicating areas for future improvement.

Discussion and Future Directions

The Deep Ritz Method shows considerable promise for solving high-dimensional variational problems with the following advantages:

1. Natural Adaptivity: The method adapts efficiently to the problem structure.
2. Dimensional Robustness: It demonstrates resilience to the curse of dimensionality.
3. Algorithmic Simplicity: The method is straightforward and integrates well with SGD.

However, challenges remain, including non-convexity of the variational problem, convergence rate inconsistency, and the complexity of handling boundary conditions. Future research should focus on network architecture optimization, choice of activation functions, and further exploration of minimization algorithms to enhance the method's applicability and efficacy.

In summary, the Deep Ritz Method represents a significant advancement in the application of deep learning to variational problems. Future work will undoubtedly refine and expand its utility in computational mathematics and applied disciplines.