A transformer-based neural operator for large-eddy simulation of turbulence
Abstract: Predicting the large-scale dynamics of three-dimensional (3D) turbulence is challenging for machine learning approaches. This paper introduces a transformer-based neural operator (TNO) to achieve precise and efficient predictions in the large-eddy simulation (LES) of 3D turbulence. The performance of the proposed TNO model is systematically tested and compared with LES using classical sub-grid scale (SGS) models, including the dynamic Smagorinsky model (DSM) and the dynamic mixed model (DMM), as well as the original Fourier neural operator (FNO) model, in homogeneous isotropic turbulence (HIT) and free-shear turbulent mixing layer. The numerical simulations comprehensively evaluate the performance of these models on a variety of flow statistics, including the velocity spectrum, the probability density functions (PDFs) of vorticity, the PDFs of velocity increments, the evolution of turbulent kinetic energy, and the iso-surface of the Q-criterion. The results indicate that the accuracy of the TNO model is comparable to the LES with DSM model, and outperforms the FNO model and LES using DMM in HIT. In the free-shear turbulence, the TNO model exhibits superior accuracy compared to other models. Moreover, the TNO model has fewer parameters than the FNO model and enables long-term stable predictions, which the FNO model cannot achieve. The well-trained TNO model is significantly faster than traditional LES with DSM and DMM models, and can be generalized to higher Taylor-Reynolds number cases, indicating its strong potential for 3D nonlinear engineering applications.
- J. Smagorinsky, “General circulation experiments with the primitive equations: I. The basic experiment,” Mon Weather Rev 91, 99–164 (1963).
- D. K. Lilly, “The representation of small-scale turbulence in numerical simulation experiments,” IBM Form , 195–210 (1967).
- M. Germano, “Turbulence: the filtering approach,” J. Fluid Mech. 238, 325–336 (1992).
- J. W. Deardorff, “A numerical study of three-dimensional turbulent channel flow at large Reynolds numbers,” J. Fluid Mech. 41, 453–480 (1970).
- P. A. Durbin, “Some recent developments in turbulence closure modeling,” Annu. Rev. Fluid Mech. 50, 77–103 (2018).
- S. B. Pope, “A more general effective-viscosity hypothesis,” J. Fluid Mech. 72, 331–340 (1975).
- H. Choi and P. Moin, “Grid-point requirements for large eddy simulation: Chapman’s estimates revisited,” Phys. Fluids 24 (2012).
- X. I. Yang and K. P. Griffin, ‘‘Grid-point and time-step requirements for direct numerical simulation and large-eddy simulation,” Phys. Fluids 33 (2021).
- Y. Bin, G. Huang, R. Kunz, and X. I. Yang, “Constrained Recalibration of Reynolds-Averaged Navier–Stokes Models,” AIAA Journal , 1–13 (2023).
- C. Meneveau and J. Katz, “Scale-invariance and turbulence models for large-eddy simulation,” Annu. Rev. Fluid Mech. 32, 1–32 (2000).
- B. Fan, Z. Yuan, Y. Wang, and J. Wang, “Eddy viscosity enhanced temporal direct deconvolution models for temporal large-eddy simulation of turbulence,” Phys. Fluids 35 (2023).
- S. Chen, Z. Xia, S. Pei, J. Wang, Y. Yang, Z. Xiao, and Y. Shi, “Reynolds-stress-constrained large-eddy simulation of wall-bounded turbulent flows,” J. Fluid Mech. 703, 1–28 (2012).
- P. Moin, K. Squires, W. Cabot, and S. Lee, “A dynamic subgrid-scale model for compressible turbulence and scalar transport,” Phys. Fluids A: Fluid Dynamics 3, 2746–2757 (1991).
- M. Germano, U. Piomelli, P. Moin, and W. H. Cabot, “A dynamic subgrid-scale eddy viscosity model,” Phys. Fluids A 3, 1760–1765 (1991).
- Y. Shi, Z. Xiao, and S. Chen, “Constrained subgrid-scale stress model for large eddy simulation,” Phys. Fluids 20 (2008).
- G. Erlebacher, M. Y. Hussaini, C. G. Speziale, and T. A. Zang, “Toward the large-eddy simulation of compressible turbulent flows,” J. Fluid Mech. 238, 155–185 (1992).
- R. A. Clark, J. H. Ferziger, and W. C. Reynolds, “Evaluation of subgrid-scale models using an accurately simulated turbulent flow,” J. Fluid Mech. 91, 1–16 (1979).
- R. Maulik, O. San, A. Rasheed, and P. Vedula, “Data-driven deconvolution for large eddy simulations of Kraichnan turbulence,” Phys. Fluids 30, 125109 (2018).
- Z. Yuan, Y. Wang, C. Xie, and J. Wang, “Dynamic iterative approximate deconvolution models for large-eddy simulation of turbulence,” Phys. Fluids 33, 085125 (2021).
- S. L. Brunton, B. R. Noack, and P. Koumoutsakos, “Machine learning for fluid mechanics,” Annu. Rev. Fluid Mech. 52, 477–508 (2020).
- K. Duraisamy, G. Iaccarino, and H. Xiao, “Turbulence modeling in the age of data,” Annu. Rev. Fluid Mech. 51, 357–377 (2019).
- M. Lienen, J. Hansen-Palmus, D. Lüdke, and S. Günnemann, “Generative Diffusion for 3D Turbulent Flows,” arXiv preprint arXiv:2306.01776 (2023).
- Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature 521, 436–444 (2015).
- Y. Wang, Z. Yuan, C. Xie, and J. Wang, “Artificial neural network-based spatial gradient models for large-eddy simulation of turbulence,” AIP Adv. 11, 055216 (2021).
- A. Beck, D. Flad, and C.-D. Munz, “Deep neural networks for data-driven turbulence models,” in Proceedings of the APS Division of Fluid Dynamics Meeting Abstracts (2019) p. G16.
- C. Xie, J. Wang, K. Li, and C. Ma, “Artificial neural network approach to large-eddy simulation of compressible isotropic turbulence,” Phys. Rev. E 99, 053113 (2019).
- M. Gamahara and Y. Hattori, “Searching for turbulence models by artificial neural network,” Phys. Rev. Fluid 2, 054604 (2017).
- Z. Zhou, G. He, S. Wang, and G. Jin, “Subgrid-scale model for large-eddy simulation of isotropic turbulent flows using an artificial neural network,” Comput Fluids 195, 104319 (2019).
- H. Li, Y. Zhao, J. Wang, and R. D. Sandberg, “Data-driven model development for large-eddy simulation of turbulence using gene-expression programing,” Phys. Fluids 33, 125127 (2021a).
- J. Ling, A. Kurzawski, and J. Templeton, “Reynolds averaged turbulence modelling using deep neural networks with embedded invariance,” J. Fluid Mech. 807, 155–166 (2016).
- Y. Guan, A. Chattopadhyay, A. Subel, and P. Hassanzadeh, “Stable a posteriori LES of 2D turbulence using convolutional neural networks: Backscattering analysis and generalization to higher Re via transfer learning,” J. Comput. Phys. 458, 111090 (2022).
- R. Han, Y. Wang, Y. Zhang, and G. Chen, “A novel spatial-temporal prediction method for unsteady wake flows based on hybrid deep neural network,” Phys. Fluids 31 (2019).
- S. Cai, Z. Mao, Z. Wang, M. Yin, and G. E. Karniadakis, “Physics-informed neural networks (PINNs) for fluid mechanics: A review,” Acta Mech Sin 37, 1727–1738 (2021).
- S. Lanthaler, S. Mishra, and G. E. Karniadakis, “Error estimates for deeponets: A deep learning framework in infinite dimensions,” Transactions of Mathematics and Its Applications 6, tnac001 (2022).
- G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang, “Physics-informed machine learning,” Nat. Rev. Phys. 3, 422–440 (2021).
- M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations,” arXiv preprint arXiv:1711.10561 (2017).
- X. Yang, S. Zafar, J.-X. Wang, and H. Xiao, “Predictive large-eddy-simulation wall modeling via physics-informed neural networks,” Phys. Rev. Fluid 4, 034602 (2019).
- M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations,” J. Comput. Phys. 378, 686–707 (2019).
- Y. Chen, D. Huang, D. Zhang, J. Zeng, N. Wang, H. Zhang, and J. Yan, “Theory-guided hard constraint projection (HCP): A knowledge-based data-driven scientific machine learning method,” J. Comput. Phys. 445, 110624 (2021).
- X. Jin, S. Cai, H. Li, and G. E. Karniadakis, “NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations,” J. Comput. Phys. 426, 109951 (2021).
- K. Wu and D. Xiu, “Data-driven deep learning of partial differential equations in modal space,” J. Comput. Phys. 408, 109307 (2020).
- H. Xu, D. Zhang, and J. Zeng, “Deep-learning of parametric partial differential equations from sparse and noisy data,” Phys. Fluids 33, 037132 (2021).
- L. Lu, X. Meng, S. Cai, Z. Mao, S. Goswami, Z. Zhang, and G. E. Karniadakis, “A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data,” Comput Methods Appl Mech Eng 393, 114778 (2022).
- N. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. Stuart, and A. Anandkumar, “Neural Operator: Learning Maps Between Function Spaces With Applications to PDEs,” J Mach Learn Res 24, 1–97 (2023).
- S. Goswami, K. Kontolati, M. D. Shields, and G. E. Karniadakis, “Deep transfer learning for partial differential equations under conditional shift with DeepONet,” arXiv preprint arXiv:2204.09810 (2022).
- Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar, “Fourier neural operator for parametric partial differential equations,” arXiv preprint arXiv:2010.08895 (2020a).
- J. Chen, J. Viquerat, and E. Hachem, “U-net architectures for fast prediction of incompressible laminar flows,” arXiv preprint arXiv:1910.13532 (2019).
- R. Wang, K. Kashinath, M. Mustafa, A. Albert, and R. Yu, “Towards physics-informed deep learning for turbulent flow prediction,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (2020) pp. 1457–1466.
- K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition (2016) pp. 770–778.
- G. Wen, Z. Li, K. Azizzadenesheli, A. Anandkumar, and S. M. Benson, “U-FNO—An enhanced Fourier neural operator-based deep-learning model for multiphase flow,” Adv Water Resour 163, 104180 (2022).
- A. Choubineh, J. Chen, D. A. Wood, F. Coenen, and F. Ma, “Fourier neural operator for fluid flow in small-shape 2D simulated porous media dataset,” Algorithms 16, 24 (2023).
- W. Peng, Z. Yuan, and J. Wang, “Attention-enhanced neural network models for turbulence simulation,” Phys. Fluids 34, 025111 (2022).
- Z. Li, D. Z. Huang, B. Liu, and A. Anandkumar, “Fourier neural operator with learned deformations for pdes on general geometries,” arXiv preprint arXiv:2207.05209 (2022a).
- Z. Jiang, M. Zhu, D. Li, Q. Li, Y. O. Yuan, and L. Lu, “Fourier-MIONet: Fourier-enhanced multiple-input neural operators for multiphase modeling of geological carbon sequestration,” arXiv preprint arXiv:2303.04778 (2023a).
- A. Tran, A. Mathews, L. Xie, and C. S. Ong, “Factorized Fourier neural operators,” arXiv preprint arXiv:2111.13802 (2021).
- P. I. Renn, C. Wang, S. Lale, Z. Li, A. Anandkumar, and M. Gharib, “Forecasting subcritical cylinder wakes with Fourier Neural Operators,” arXiv preprint arXiv:2301.08290 (2023).
- J. Guibas, M. Mardani, Z. Li, A. Tao, A. Anandkumar, and B. Catanzaro, “Adaptive Fourier neural operators: Efficient token mixers for transformers,” arXiv preprint arXiv:2111.13587 (2021).
- Z. Hao, C. Ying, Z. Wang, H. Su, Y. Dong, S. Liu, Z. Cheng, J. Zhu, and J. Song, “GNOT: A General Neural Operator Transformer for Operator Learning,” arXiv preprint arXiv:2302.14376 (2023).
- J. A. L. Benitez, T. Furuya, F. Faucher, X. Tricoche, and M. V. de Hoop, “Fine-tuning Neural-Operator architectures for training and generalization,” arXiv preprint arXiv:2301.11509 (2023).
- D. Meng, Y. Zhu, J. Wang, and Y. Shi, “Fast flow prediction of airfoil dynamic stall based on Fourier neural operator,” Phys. Fluids 35 (2023).
- Z. Deng, H. Liu, B. Shi, Z. Wang, F. Yu, Z. Liu, and G. Chen, “Temporal predictions of periodic flows using a mesh transformation and deep learning-based strategy,” Aerosp Sci Technol 134, 108081 (2023).
- M. Momenifar, E. Diao, V. Tarokh, and A. D. Bragg, “Dimension reduced turbulent flow data from deep vector quantisers,” J. Turbul. 23, 232–264 (2022).
- A. T. Mohan, D. Tretiak, M. Chertkov, and D. Livescu, “Spatio-temporal deep learning models of 3D turbulence with physics informed diagnostics,” J. Turbul. 21, 484–524 (2020).
- T. Nakamura, K. Fukami, K. Hasegawa, Y. Nabae, and K. Fukagata, “Convolutional neural network and long short-term memory based reduced order surrogate for minimal turbulent channel flow,” Phys. Fluids 33, 025116 (2021).
- Z. Li, W. Peng, Z. Yuan, and J. Wang, “Fourier neural operator approach to large eddy simulation of three-dimensional turbulence,” Theor. App. Mech. Lett. 12, 100389 (2022b).
- W. Peng, Z. Yuan, Z. Li, and J. Wang, “Linear attention coupled Fourier neural operator for simulation of three-dimensional turbulence,” Phys. Fluids 35, 015106 (2023).
- Z. Li, W. Peng, Z. Yuan, and J. Wang, “Long-term predictions of turbulence by implicit U-Net enhanced Fourier neural operator,” Phys. Fluids 35 (2023a).
- Z. Li, N. B. Kovachki, C. Choy, B. Li, J. Kossaifi, S. P. Otta, M. A. Nabian, M. Stadler, C. Hundt, K. Azizzadenesheli, et al., “Geometry-informed neural operator for large-scale 3d pdes,” arXiv preprint arXiv:2309.00583 (2023b).
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Proc. Adv. Neural Inf. Process. Syst. 30 (2017).
- N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher, “Ctrl: A conditional transformer language model for controllable generation,” arXiv preprint arXiv:1909.05858 (2019).
- Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov, “Transformer-xl: Attentive language models beyond a fixed-length context,” arXiv preprint arXiv:1901.02860 (2019).
- Z. Li, K. Meidani, and A. B. Farimani, “Transformer for partial differential equations’ operator learning,” arXiv preprint arXiv:2205.13671 (2022).
- D. Drikakis, I. W. Kokkinakis, D. Fung, and S. M. Spottswood, “Generalizability of transformer-based deep learning for multidimensional turbulent flow data,” Physics of Fluids 36 (2024).
- Q. Xu, Z. Zhuang, Y. Pan, and B. Wen, “Super-resolution reconstruction of turbulent flows with a transformer-based deep learning framework,” Phys. Fluids 35 (2023).
- M. Momenifar, E. Diao, V. Tarokh, and A. D. Bragg, “Emulating Spatio-Temporal Realizations of Three-Dimensional Isotropic Turbulence via Deep Sequence Learning Models,” arXiv preprint arXiv:2112.03469 (2021).
- K. Bi, L. Xie, H. Zhang, X. Chen, X. Gu, and Q. Tian, “Pangu-weather: A 3d high-resolution model for fast and accurate global weather forecast,” arXiv preprint arXiv:2211.02556 (2022).
- Y. Wang, A. Solera-Rico, C. S. Vila, and R. Vinuesa, “Towards optimal β𝛽\betaitalic_β-variational autoencoders combined with transformers for reduced-order modelling of turbulent flows,” Int J Heat Fluid Flow 105, 109254 (2024).
- A. Patil, J. Viquerat, and E. Hachem, “Autoregressive transformers for data-driven spatio-temporal learning of turbulent flows,” arXiv preprint arXiv:2209.08052 (2022).
- Y. Dang, Z. Hu, M. Cranmer, M. Eickenberg, and S. Ho, “TNT: Vision transformer for turbulence simulations,” arXiv preprint arXiv:2207.04616 (2022).
- Z. Li, D. Shu, and A. B. Farimani, “Scalable Transformer for PDE Surrogate Modeling,” arXiv preprint arXiv:2305.17560 (2023).
- J. Jiang, G. Li, Y. Jiang, L. Zhang, and X. Deng, “TransCFD: A transformer-based decoder for flow field prediction,” Eng Appl Artif Intell 123, 106340 (2023b).
- S. B. Pope and S. B. Pope, Turbulent flows (Cambridge university press, 2000).
- T. Ishihara, T. Gotoh, and Y. Kaneda, “Study of high–Reynolds number isotropic turbulence by direct numerical simulation,” Annu. Rev. Fluid Mech. 41, 165–180 (2009).
- Y. Wang, Z. Yuan, X. Wang, and J. Wang, “Constant-coefficient spatial gradient models for the sub-grid scale closure in large-eddy simulation of turbulence,” Phys. Fluids 34, 095108 (2022).
- P. Sagaut, Large eddy simulation for incompressible flows: an introduction (Springer Science & Business Media, 2006).
- R. D. Moser, S. W. Haering, and G. R. Yalla, “Statistical properties of subgrid-scale turbulence models,” Annu. Rev. Fluid Mech. 53, 255–286 (2021).
- P. L. Johnson, “A physics-inspired alternative to spatial filtering for large-eddy simulations of turbulent flows,” J. Fluid Mech. 934, A30 (2022).
- D. K. Lilly, “A proposed modification of the Germano subgrid-scale closure method,” Phys. Fluids A 4, 633–635 (1992).
- S. Liu, C. Meneveau, and J. Katz, “On the properties of similarity subgrid-scale models as deduced from measurements in a turbulent jet,” J. Fluid Mech. 275, 83–119 (1994).
- Z. Yuan, C. Xie, and J. Wang, “Deconvolutional artificial neural network models for large eddy simulation of turbulence,” Phys. Fluids 32, 115106 (2020).
- B. Beauzamy, Introduction to Banach spaces and their geometry (Elsevier, 2011).
- V. N. Vapnik, “An overview of statistical learning theory,” IEEE Trans Neural Netw 10, 988–999 (1999).
- Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar, “Neural operator: Graph kernel network for partial differential equations,” arXiv preprint arXiv:2003.03485 (2020b).
- M. M. Rashid, T. Pittie, S. Chakraborty, and N. A. Krishnan, “Learning the stress-strain fields in digital composites using Fourier neural operator,” Iscience 25, 105452 (2022).
- J. Pathak, S. Subramanian, P. Harrington, S. Raja, A. Chattopadhyay, M. Mardani, T. Kurth, D. Hall, Z. Li, K. Azizzadenesheli, et al., “Fourcastnet: A global data-driven high-resolution weather model using adaptive Fourier neural operators,” arXiv preprint arXiv:2202.11214 (2022).
- J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu, “Roformer: Enhanced transformer with rotary position embedding,” Neurocomputing 568, 127063 (2024).
- M. Y. Hussaini and T. A. Zang, “Spectral methods in fluid dynamics,” Annu. Rev. Fluid Mech. 19, 339–367 (1987).
- N. Chang, Z. Yuan, and J. Wang, “The effect of sub-filter scale dynamics in large eddy simulation of turbulence,” Phys. Fluids 34, 095104 (2022).
- Z. Zhuang, M. Liu, A. Cutkosky, and F. Orabona, “Understanding adamw through proximal methods and scale-freeness,” Transactions on Machine Learning Research (2022).
- I. Loshchilov and F. Hutter, “Fixing weight decay regularization in adam,” (2018).
- D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” arXiv preprint arXiv:1606.08415 (2016).
- C. Xie, J. Wang, H. Li, M. Wan, and S. Chen, “A modified optimal LES model for highly compressible isotropic turbulence,” Phys. Fluids 30, 065108 (2018).
- C. Xie, J. Wang, H. Li, M. Wan, and S. Chen, “An approximate second-order closure model for large-eddy simulation of compressible isotropic turbulence,” Commun Comput Phys 27, 775–808 (2020).
- X. Wang, J. Wang, and S. Chen, “Compressibility effects on statistics and coherent structures of compressible turbulent mixing layers,” J. Fluid Mech. 947, A38 (2022).
- N. Sharan, G. Matheou, and P. E. Dimotakis, “Turbulent shear-layer mixing: initial conditions, and direct-numerical and large-eddy simulations,” J. Fluid Mech. 877, 35–81 (2019).
- Z. Yuan, Y. Wang, X. Wang, and J. Wang, “Adjoint-based variational optimal mixed models for large-eddy simulation of turbulence,” Phys. Fluids 35, 075105 (2023).
- M. M. Rogers and R. D. Moser, “Direct simulation of a self-similar turbulent mixing layer,” Phys. Fluids 6, 903–923 (1994).
- Y. Dubief and F. Delcayre, “On coherent-vortex identification in turbulence,” J. Turbul. 1, 011 (2000).
- J. Zhan, Y. Li, W. O. Wai, and W. Hu, “Comparison between the Q criterion and Rortex in the application of an in-stream structure,” Phys. Fluids 31, 121701 (2019).
- T. Kurth, S. Subramanian, P. Harrington, J. Pathak, M. Mardani, D. Hall, A. Miele, K. Kashinath, and A. Anandkumar, “Fourcastnet: Accelerating global high-resolution weather forecasting using adaptive Fourier neural operators,” arXiv preprint arXiv:2208.05419 (2022).
- L. Z. Zhao, X. Ding, and B. A. Prakash, “Pinnsformer: A transformer-based framework for physics-informed neural networks,” arXiv preprint arXiv:2307.11833 (2023).
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.