Efficient training of physics-informed neural networks via importance sampling (2104.12325v1)

Abstract: Physics-Informed Neural Networks (PINNs) are a class of deep neural networks that are trained, using automatic differentiation, to compute the response of systems governed by partial differential equations (PDEs). The training of PINNs is simulation-free, and does not require any training dataset to be obtained from numerical PDE solvers. Instead, it only requires the physical problem description, including the governing laws of physics, domain geometry, initial/boundary conditions, and the material properties. This training usually involves solving a non-convex optimization problem using variants of the stochastic gradient descent method, with the gradient of the loss function approximated on a batch of collocation points, selected randomly in each iteration according to a uniform distribution. Despite the success of PINNs in accurately solving a wide variety of PDEs, the method still requires improvements in terms of computational efficiency. To this end, in this paper, we study the performance of an importance sampling approach for efficient training of PINNs. Using numerical examples together with theoretical evidences, we show that in each training iteration, sampling the collocation points according to a distribution proportional to the loss function will improve the convergence behavior of the PINNs training. Additionally, we show that providing a piecewise constant approximation to the loss function for faster importance sampling can further improve the training efficiency. This importance sampling approach is straightforward and easy to implement in the existing PINN codes, and also does not introduce any new hyperparameter to calibrate. The numerical examples include elasticity, diffusion and plane stress problems, through which we numerically verify the accuracy and efficiency of the importance sampling approach compared to the predominant uniform sampling approach.

• The paper proposes an importance sampling framework that targets high-error regions to speed up the training process of physics-informed neural networks.
• It employs a piecewise constant approximation of the loss function to significantly reduce the computational cost of evaluating gradients at numerous collocation points.
• Numerical validations on elasticity, transient diffusion, and stress analysis benchmarks demonstrate faster convergence and improved computational efficiency compared to uniform sampling.

Efficient Training of Physics-Informed Neural Networks via Importance Sampling

The paper at hand presents a method to improve the training efficiency of Physics-Informed Neural Networks (PINNs) through the implementation of an importance sampling framework. PINNs are a class of deep neural networks designed to integrate partial differential equations (PDEs) directly into their loss function by leveraging the concepts of automatic differentiation and stochastic gradient descent. Unlike traditional methods, PINNs do not require labeled data generated from numerical solvers but rely solely on the differential equations that govern the problem domain. This paper addresses a significant challenge in PINNs: their computational inefficiency stemming from the traditional uniform sampling of collocation points.

Key Contributions

1. Proposal of Importance Sampling Framework for PINNs: The paper introduces an importance sampling framework that adapts the distribution of collocation points based on their contribution to the total loss. Specifically, the collocation points are drawn from a distribution that is proportional to the magnitude of the loss at each point, improving convergence speed by focusing more computational effort on areas where the residuals are higher.
2. Piecewise Constant Approximation of Loss Function: As directly evaluating the gradient of the loss function at each collocation point per iteration is expensive, the authors devised a computationally efficient approximation strategy. They propose evaluating the loss at a sparse set of 'seed' points and implementing a piecewise constant approximation within the domain, allowing this modified sampling approach to significantly reduce computational costs.

Numerical Validation

The methodology was tested across a series of benchmark problems including elasticity on plate geometries, transient diffusion problems, and stress analysis in structural components, showcasing clear improvements in training efficiency. The PINNs trained with the proposed importance sampling method demonstrated faster convergence than those trained with uniform sampling, both in terms of the number of iterations and wall-clock time. For example, in the elasticity problem, this approach effectively reduced unnecessary computation by focusing on collocation points with higher loss values, subsequently narrowing in on the solution space more rapidly than traditional techniques.

Implications and Future Directions

The paper implies that importance sampling can be a powerful tool to enhance the computational efficiency of PINNs, making them a more viable option for real-world large-scale PDE problems. The introduction of a loss-guided sampling mechanism represents a significant shift from existing uniform or random sampling methods, particularly because it aligns computational resources with areas of the problem domain that drive error reduction.

Looking forward, the adoption of importance sampling could evolve to accommodate even more complex PDE scenarios, such as those that are high-dimensional or stochastic in nature. Future research could explore distributed implementations of the importance sampling approach, potentially distributed across multiple computational nodes, to further accelerate training. The integration of adaptive mechanics that automatically refine the distribution of points based on real-time convergence metrics could further enhance efficiency.

In conclusion, this paper lays down a significant methodology for the improvement of PINNs, opening avenues for their accelerated application in the engineering and physical sciences domains where PDE-based modeling plays a crucial role.