Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks (2307.11833v3)

Published 21 Jul 2023 in cs.CE and cs.LG

Abstract: Physics-Informed Neural Networks (PINNs) have emerged as a promising deep learning framework for approximating numerical solutions to partial differential equations (PDEs). However, conventional PINNs, relying on multilayer perceptrons (MLP), neglect the crucial temporal dependencies inherent in practical physics systems and thus fail to propagate the initial condition constraints globally and accurately capture the true solutions under various scenarios. In this paper, we introduce a novel Transformer-based framework, termed PINNsFormer, designed to address this limitation. PINNsFormer can accurately approximate PDE solutions by utilizing multi-head attention mechanisms to capture temporal dependencies. PINNsFormer transforms point-wise inputs into pseudo sequences and replaces point-wise PINNs loss with a sequential loss. Additionally, it incorporates a novel activation function, Wavelet, which anticipates Fourier decomposition through deep neural networks. Empirical results demonstrate that PINNsFormer achieves superior generalization ability and accuracy across various scenarios, including PINNs failure modes and high-dimensional PDEs. Moreover, PINNsFormer offers flexibility in integrating existing learning schemes for PINNs, further enhancing its performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Klaus-Jürgen Bathe. Finite element method. Wiley encyclopedia of computer science and engineering, pp.  1–12, 2007.
  2. Quadratic residual networks: A new class of neural networks for solving forward and inverse problems in physics involving pdes. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), pp.  675–683. SIAM, 2021.
  3. Machine learning and the physical sciences. Reviews of Modern Physics, 91(4):045002, 2019.
  4. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
  5. Physics-informed learning of governing equations from scarce data. Nature communications, 12(1):6136, 2021.
  6. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.
  7. George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303–314, 1989.
  8. Rethinking the importance of sampling in physics-informed neural networks. arXiv preprint arXiv:2207.02338, 2022.
  9. Assessing physics informed neural networks in ocean modelling and climate change applications. In AI: Modeling Oceans and Climate Change Workshop at ICLR 2021, 2021.
  10. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  11. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  12. Bengt Fornberg. A practical guide to pseudospectral methods. Number 1. Cambridge university press, 1998.
  13. Limitations of physics informed machine learning for nonlinear two-phase transport in porous media. Journal of Machine Learning for Modeling and Computing, 1(1), 2020.
  14. Convolutional sequence to sequence learning. In International conference on machine learning, pp.  1243–1252. PMLR, 2017.
  15. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp.  315–323. JMLR Workshop and Conference Proceedings, 2011.
  16. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020.
  17. A physics-informed deep learning framework for inversion and surrogate modeling in solid mechanics. Computer Methods in Applied Mechanics and Engineering, 379:113741, 2021.
  18. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505–8510, 2018.
  19. Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2):251–257, 1991.
  20. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Computational Physics, 404:109136, 2020.
  21. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34:26548–26560, 2021.
  22. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 9(5):987–1000, 1998.
  23. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020.
  24. Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. Journal of Fluid Mechanics, 807:155–166, 2016.
  25. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  10012–10022, 2021.
  26. Physics-informed neural networks for solving forward and inverse flow problems via the boltzmann-bgk formulation. Journal of Computational Physics, 447:110676, 2021.
  27. Physics-informed neural networks with hard constraints for inverse design. SIAM Journal on Scientific Computing, 43(6):B1105–B1132, 2021.
  28. Physics-informed neural networks for high-speed flows. Computer Methods in Applied Mechanics and Engineering, 360:112789, 2020.
  29. Self-adaptive physics-informed neural networks using a soft attention mechanism. arXiv preprint arXiv:2009.04544, 2020.
  30. Lagrangian pinns: A causality-conforming solution to failure modes of physics-informed neural networks. arXiv preprint arXiv:2205.02902, 2022.
  31. How do vision transformers work? arXiv preprint arXiv:2202.06709, 2022.
  32. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  33. Improving language understanding by generative pre-training. 2018.
  34. Maziar Raissi. Deep hidden physics models: Deep learning of nonlinear partial differential equations. The Journal of Machine Learning Research, 19(1):932–955, 2018.
  35. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.10561, 2017.
  36. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019.
  37. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
  38. A versatile framework to solve the helmholtz equation using physics-informed neural networks. Geophysical Journal International, 228(3):1750–1762, 2022.
  39. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  40. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing, 43(5):A3055–A3081, 2021.
  41. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449:110768, 2022a.
  42. Auto-pinn: Understanding and optimizing physics-informed neural architecture. arXiv preprint arXiv:2205.13748, 2022b.
  43. Learning in sinusoidal spaces with physics-informed neural networks. IEEE Transactions on Artificial Intelligence, 2022.
  44. Physics-informed generative adversarial networks for stochastic differential equations. SIAM Journal on Scientific Computing, 42(1):A292–A317, 2020.
  45. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pp.  581–590. IEEE, 2020.
  46. Physics informed machine learning with misspecified priors:\\\backslash\\\\backslash\an analysis of turning operation in lathe machines. In AAAI 2022 Workshop on AI for Design and Manufacturing (ADAM), 2022.
  47. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp.  11106–11115, 2021.
  48. Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data. Journal of Computational Physics, 394:56–81, 2019.
Citations (14)
View on Semantic Scholar

Summary

  • The paper introduces PINNsFormer, a transformer-based framework that leverages sequence modeling to capture temporal dependencies in solving PDEs.
  • It incorporates a novel Wavelet activation function as a universal approximator, enhancing the model's ability to address high-dimensional, complex PDEs.
  • Empirical results show faster convergence, improved generalization, and lower error rates, especially for high-frequency and multiscale challenges.

Insights into PINNsFormer: A Transformer-Based Framework for Physics-Informed Neural Networks

The paper "PINNsFormer: A Transformer-Based Framework for Physics-Informed Neural Networks" presents an innovative adaptation of Physics-Informed Neural Networks (PINNs) to utilize Transformer models. Conventional PINNs, which typically employ multilayer perceptrons (MLPs), are limited by their inability to effectively capture temporal dependencies, which are crucial in accurately modeling dynamic physics systems. The introduction of PINNsFormer provides a sequence-to-sequence model based on Transformers that enhances the generalization and accuracy of solution approximations to partial differential equations (PDEs).

Key Contributions

This work offers several significant contributions to the field:

  1. Transformer-Based Framework: PINNsFormer is the first framework that explicitly focuses on leveraging Transformers to model temporal dependencies in PDE solutions. It replaces the point-wise predictions of conventional PINNs with sequential outputs that allow the model to learn and propagate temporal information effectively.
  2. Novel Activation Function: The introduction of the Wavelet activation function acts as a universal approximator by anticipating Fourier decomposition through neural networks. This function enhances the function approximation capabilities of the model across various deep learning tasks.
  3. Enhanced Flexibility and Performance: PINNsFormer shows superior performance compared to traditional PINNs, demonstrating flexibility in incorporating different learning schemes and effectively addressing high-dimensional PDEs. This makes PINNsFormer a versatile approach for a wider array of applications.

Empirical Results

The empirical evaluations underscore the superior performance of PINNsFormer over traditional PINNs and their variants, especially in scenarios known to challenge PINNs' efficacy like high-frequency or multiscale PDEs. The results demonstrate that PINNsFormer not only improves generalization but also shows faster convergence on complex, multidimensional PDEs, significantly outperforming baseline models in terms of both training loss and validation accuracy. For instance, in the evaluation of high-dimensional Navier-Stokes equations, PINNsFormer managed to capture the correct dynamics with a much lower error rate than traditional approaches.

Implications and Future Directions

The application of Transformer models in PINNs opens new avenues for enhancing the accuracy and generalization of deep learning models in computational physics. PINNsFormer’s ability to leverage temporal dependencies offers a promising direction for solving complex PDEs that arise in practical physics and engineering problems.

Further research could explore the scalability of PINNsFormer to even more complex systems and its integration with other machine learning methods to enhance robustness and efficiency. Moreover, the Wavelet activation function's potential in different neural network structures warrants further investigation given its effectiveness in this framework.

In summary, PINNsFormer represents a significant step forward in addressing the limitations of traditional PINNs by introducing a robust Transformer-based architecture and a novel activation function suited for deep scientific computing. It invites future exploration and development to extend its applicability across various scientific domains where PDEs are crucial.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - AdityaLab/pinnsformer (137 stars)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets