PDEBENCH: An Extensive Benchmark for Scientific Machine Learning (2210.07182v7)

Abstract: Machine learning-based modeling of physical systems has experienced increased interest in recent years. Despite some impressive progress, there is still a lack of benchmarks for Scientific ML that are easy to use but still challenging and representative of a wide range of problems. We introduce PDEBench, a benchmark suite of time-dependent simulation tasks based on Partial Differential Equations (PDEs). PDEBench comprises both code and data to benchmark the performance of novel machine learning models against both classical numerical simulations and machine learning baselines. Our proposed set of benchmark problems contribute the following unique features: (1) A much wider range of PDEs compared to existing benchmarks, ranging from relatively common examples to more realistic and difficult problems; (2) much larger ready-to-use datasets compared to prior work, comprising multiple simulation runs across a larger number of initial and boundary conditions and PDE parameters; (3) more extensible source codes with user-friendly APIs for data generation and baseline results with popular machine learning models (FNO, U-Net, PINN, Gradient-Based Inverse Method). PDEBench allows researchers to extend the benchmark freely for their own purposes using a standardized API and to compare the performance of new models to existing baseline methods. We also propose new evaluation metrics with the aim to provide a more holistic understanding of learning methods in the context of Scientific ML. With those metrics we identify tasks which are challenging for recent ML methods and propose these tasks as future challenges for the community. The code is available at https://github.com/pdebench/PDEBench.

• The paper introduces PDEBench, a versatile benchmark suite that standardizes the evaluation of SciML methods on simulations ranging from simple 1D to complex 3D PDEs.
• The paper incorporates innovative metrics such as cRMSE, bRMSE, and fRMSE to assess physical consistency, boundary adherence, and fidelity across frequency regimes.
• The paper benchmarks models like FNO, U-Net, and PINNs, highlighting FNO’s superior performance while revealing challenges in handling high-frequency and non-smooth dynamics.

PDEBench: An Extensive Benchmark for Scientific Machine Learning

The paper introduces PDEBench, a comprehensive benchmark suite designed to evaluate the performance of scientific machine learning methods on time-dependent simulation tasks governed by partial differential equations (PDEs). This benchmark addresses the current lack of standardized and challenging benchmarks in the domain of scientific machine learning (SciML).

Dataset Composition and Features

PDEBench expands on existing efforts by incorporating a diverse set of PDEs, with variations ranging from simplistic 1D equations to complex 3D systems with non-trivial boundary conditions. Notably, it includes:

1. Wide Range of PDEs: A broad spectrum of 11 different PDEs, including both time-dependent and time-independent forms.
2. Big Data Appeal: Large ready-to-use datasets facilitate experimentation across various scenarios and conditions.
3. Extensible Codebase: Researchers are equipped with user-friendly APIs to generate data and test novel models against established baselines.

Evaluative Metrics

Recognizing the limitations of classical evaluation metrics like RMSE (Root Mean Square Error), PDEBench proposes additional metrics to measure model performance in a more nuanced way. These include:

• cRMSE: Error from conserved values, assessing physical consistency.
• bRMSE: Boundary RMSE, evaluating conformance to boundary conditions.
• fRMSE: Fourier domain RMSE, divided into low, mid, and high frequencies, assessing fidelity across different frequency regimes.

Baseline Implementations

The paper benchmarks several state-of-the-art SciML models:

• Fourier Neural Operator (FNO): Exhibits strong general performance, learning PDEs in Fourier space effectively.
• U-Net Model: Evaluated with autoregressive training methods for stability.
• Physics-Informed Neural Networks (PINNs): Applied particularly to cases with analytically challenging dynamics.

Findings and Comparative Analysis

The initial evaluations demonstrate that FNO consistently provides the best performance across numerous metrics, although challenges remain in handling high frequency and non-smooth regimes, as evident in experiments with the Burgers’ equation. U-Net's autoregressive behavior required stabilization through training adjustments, and PINNs showed resilience in capturing complex features, despite middleware frequency struggles.

Implications and Future Directions

PDEBench opens opportunities for systematic assessment and improvement of SciML methods in scenarios reflective of real-world applications. It encourages further development in areas of differentiable programming and scientific computing, offering a platform for refinement and calibration of advanced models. Researchers are invited to contribute by expanding the benchmark dataset, incorporating additional dynamics, and addressing the broader challenges presented by these PDE-related tasks.

Overall, PDEBench serves as a pivotal contribution to the field, offering a robust framework for advancing SciML, with prospects for fostering innovation in the modeling of complex physical systems.