2000 character limit reached
PDEformer: Towards a Foundation Model for One-Dimensional Partial Differential Equations (2402.12652v2)
Published 20 Feb 2024 in math.NA and cs.NA
Abstract: This paper introduces PDEformer, a neural solver for partial differential equations (PDEs) capable of simultaneously addressing various types of PDEs. We propose to represent the PDE in the form of a computational graph, facilitating the seamless integration of both symbolic and numerical information inherent in a PDE. A graph Transformer and an implicit neural representation (INR) are employed to generate mesh-free predicted solutions. Following pretraining on data exhibiting a certain level of diversity, our model achieves zero-shot accuracies on benchmark datasets that is comparable to those of specifically trained expert models. Additionally, PDEformer demonstrates promising results in the inverse problem of PDE coefficient recovery.
- Dedalus: A flexible framework for numerical simulations with spectral methods. Physical Review Research, 2(2):023068, April 2020. doi: 10.1103/PhysRevResearch.2.023068.
- Transformers as meta-learners for implicit neural representations. In European Conference on Computer Vision, 2022.
- An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy.
- COIN++: Neural compression across modalities. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=NXB0rEM2Tq.
- Multiplicative filter networks. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=OmtmcPkkhT.
- Shap-e: Generating conditional 3d implicit functions, 2023.
- HyperdeepONet: learning operator with complex target function space using the limited resources via hypernetwork. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=OAw6V3ZAhSd.
- Fourier neural operator for parametric partial differential equations, 2021.
- Prose: Predicting operators and symbolic expressions using multimodal transformers, 2023.
- Physics informed token transformer, 2023.
- Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218–229, Mar 2021. ISSN 2522-5839. doi: 10.1038/s42256-021-00302-5.
- Multiple physics pretraining for physical surrogate models, 2023.
- Transformer for graphs: An overview from architecture perspective, 2022.
- Beyond periodicity: Towards a unifying framework for activations in coordinate-mlps. In Computer Vision – ECCV 2022, pp. 142–158, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-19827-4.
- U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234–241. Springer, 2015.
- Wire: Wavelet implicit neural representations. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18507–18516, 2023. doi: 10.1109/CVPR52729.2023.01775.
- Polynomial implicit neural representations for large diverse datasets. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2041–2051, 2023. doi: 10.1109/CVPR52729.2023.00203.
- Implicit neural representations with periodic activation functions. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 7462–7473. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/53c04118df112c13a8c34b38343b9c10-Paper.pdf.
- Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=zANxvzflMl.
- Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35:1596–1611, 2022.
- Particle swarm optimization algorithm: an overview. Soft computing, 22:387–408, 2018.
- Pde generalization of in-context operator networks: A study on 1d scalar nonlinear conservation laws, 2024.
- In-context operator learning with data prompts for differential equation problems. Proceedings of the National Academy of Sciences, 120(39), September 2023. ISSN 1091-6490. doi: 10.1073/pnas.2310142120. URL http://dx.doi.org/10.1073/pnas.2310142120.
- Fine-tune language models as multi-modal differential equation solvers, 2024.
- Continuous PDE dynamics forecasting with implicit neural representations. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=B73niNjbPs.
- Do transformers really perform badly for graph representation? In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=OeWooOxFwDa.