Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 83 tok/s
Gemini 2.5 Pro 42 tok/s Pro
GPT-5 Medium 30 tok/s Pro
GPT-5 High 36 tok/s Pro
GPT-4o 108 tok/s Pro
Kimi K2 220 tok/s Pro
GPT OSS 120B 473 tok/s Pro
Claude Sonnet 4 40 tok/s Pro
2000 character limit reached

Real-time Inference and Extrapolation via a Diffusion-inspired Temporal Transformer Operator (DiTTO) (2307.09072v2)

Published 18 Jul 2023 in cs.LG, cs.AI, cs.CE, cs.NA, and math.NA

Abstract: Extrapolation remains a grand challenge in deep neural networks across all application domains. We propose an operator learning method to solve time-dependent partial differential equations (PDEs) continuously and with extrapolation in time without any temporal discretization. The proposed method, named Diffusion-inspired Temporal Transformer Operator (DiTTO), is inspired by latent diffusion models and their conditioning mechanism, which we use to incorporate the temporal evolution of the PDE, in combination with elements from the transformer architecture to improve its capabilities. Upon training, DiTTO can make inferences in real-time. We demonstrate its extrapolation capability on a climate problem by estimating the temperature around the globe for several years, and also in modeling hypersonic flows around a double-cone. We propose different training strategies involving temporal-bundling and sub-sampling and demonstrate performance improvements for several benchmarks, performing extrapolation for long time intervals as well as zero-shot super-resolution in time.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019.
  2. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218–229, 2021.
  3. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020.
  4. Pde-net: Learning pdes from data. In International conference on machine learning, pages 3208–3216. PMLR, 2018.
  5. Dl-pde: Deep-learning based data-driven discovery of partial differential equations from discrete and noisy data. arXiv preprint arXiv:1908.04463, 2019.
  6. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):82–97, 2012.
  7. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 2012.
  8. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  9. Thomas JR Hughes. The Finite Element Method: Linear Static and Dynamic Finite Element Analysis. Courier Corporation, 2012.
  10. Sergei K Godunov and I Bohachevsky. Finite difference method for numerical computation of discontinuous solutions of the equations of fluid dynamics. Matematičeskij Sbornik, 47(3):271–306, 1959.
  11. Finite volume methods. Handbook of Numerical Analysis, 7:713–1018, 2000.
  12. Spectral/HP Element Methods for Computational Fluid Dynamics. Oxford University Press, USA, 2005.
  13. Tribikram Kundu. Acoustic source localization. Ultrasonics, 54(1):25–38, 2014.
  14. Beyond the courant-friedrichs-lewy condition: Numerical methods for the wave problem using deep learning. Journal of Computational Physics, 442:110493, 2021.
  15. Physics-informed neural networks (pinns) for fluid mechanics: A review. Acta Mechanica Sinica, 37(12):1727–1738, 2021.
  16. A hybrid iterative numerical transferable solver (hints) for pdes based on deep operator network and relaxation methods. arXiv preprint arXiv:2208.13273, 2022.
  17. Benchmark problems for the mesoscale multiphysics phase field simulator (memphis). Technical report, Sandia National Lab.(SNL-NM), Albuquerque, NM (United States), 2020.
  18. Learning two-phase microstructure evolution using neural operators and autoencoder architectures. npj Computational Materials, 8(1):190, 2022.
  19. Anthony T Patera. A spectral element method for fluid dynamics: laminar flow in a channel expansion. Journal of Computational Physics, 54(3):468–488, 1984.
  20. Turbulence statistics in fully developed channel flow at low Reynolds number. Journal of Fluid Mechanics, 177:133–166, 1987.
  21. Robust automatic p-phase picking: an on-line implementation in the analysis of broadband seismogram recordings. Physics of the earth and planetary interiors, 113(1-4):265–275, 1999.
  22. Wavelet neural operator for solving parametric partial differential equations in computational mechanics problems. Computer Methods in Applied Mechanics and Engineering, 404:115783, 2023.
  23. Lno: Laplace neural operator for solving differential equations. arXiv preprint arXiv:2303.10528, 2023.
  24. Vito: Vision transformer-operator. arXiv preprint arXiv:2303.08891, 2023.
  25. U-no: U-shaped neural operators. arXiv preprint arXiv:2204.11127, 2022.
  26. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35:1596–1611, 2022.
  27. Towards multi-spatiotemporal-scale generalized pde modeling. arXiv preprint arXiv:2209.15616, 2022.
  28. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  29. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  30. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
  31. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  32. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517, 2017.
  33. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
  34. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  35. Message passing neural pde solvers. arXiv preprint arXiv:2202.03376, 2022.
  36. The community earth system model: a framework for collaborative research. Bulletin of the American Meteorological Society, 94(9):1339–1360, 2013.
  37. Learning bias corrections for climate models using deep neural operators. arXiv preprint arXiv:2302.03173, 2023.
  38. Learning operators with coupled attention. The Journal of Machine Learning Research, 23(1):9636–9698, 2022.
  39. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202.11214, 2022.
  40. Graphcast: Learning skillful medium-range global weather forecasting. arXiv preprint arXiv:2212.12794, 2022.
  41. Climax: A foundation model for weather and climate. arXiv preprint arXiv:2301.10343, 2023.
  42. The ncep/ncar 40-year reanalysis project. In Renewable Energy, pages Vol1_146–Vol1_194. Routledge, 2018.
  43. Assessment of cfd capability for hypersonic shock wave laminar boundary layer interactions. Aerospace, 4(2):25, 2017.
  44. Lawrence C Evans. Partial differential equations, volume 19. American Mathematical Society, 2022.
  45. Jürgen Jost. Partial differential equations, volume 214. Springer Science & Business Media, 2012.
  46. Jacques Hadamard. Sur les problèmes aux dérivées partielles et leur signification physique. Princeton university bulletin, pages 49–52, 1902.
  47. Neural operator: Learning maps between function spaces. CoRR, abs/2108.08481, 2021.
  48. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393:114778, 2022.
  49. Transformers in time series: A survey. arXiv preprint arXiv:2202.07125, 2022.
  50. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1):87–110, 2022.
  51. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  52. Transformer for partial differential equations’ operator learning. arXiv preprint arXiv:2205.13671, 2022.
  53. Ht-net: Hierarchical transformer based operator learning model for multiscale pdes. arXiv preprint arXiv:2210.10890, 2022.
  54. Gnot: A general neural operator transformer for operator learning. arXiv preprint arXiv:2302.14376, 2023.
  55. Gnot: A general neural operator transformer for operator learning. In International Conference on Machine Learning, pages 12556–12569. PMLR, 2023.
  56. Shuhao Cao. Choose a transformer: Fourier or galerkin. Advances in neural information processing systems, 34:24924–24940, 2021.
  57. Transformer meets boundary value inverse problems. arXiv preprint arXiv:2209.14977, 2022.
  58. Crunchgpt: A chatgpt assisted framework for scientific machine learning. arXiv preprint arXiv:2306.15551, 2023.
  59. Generative diffusion learning for parametric partial differential equations. arXiv preprint arXiv:2305.14703, 2023.
  60. A physics-informed diffusion model for high-fidelity flow field reconstruction. Journal of Computational Physics, 478:111972, 2023.
  61. Generative artificial intelligence and its applications in materials science: Current situation and future perspectives. Journal of Materiomics, 2023.
  62. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162–8171. PMLR, 2021.
  63. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
  64. Freeu: Free lunch in diffusion u-net. arXiv preprint arXiv:2309.11497, 2023.
  65. Trixi.jl: Adaptive high-order numerical simulations of hyperbolic PDEs in Julia. https://github.com/trixi-framework/Trixi.jl, 09 2021.
  66. Adaptive numerical simulations with Trixi.jl: A case study of Julia for scientific computing. Proceedings of the JuliaCon Conferences, 1(1):77, 2022.
  67. A purely hyperbolic discontinuous Galerkin approach for self-gravitating gas dynamics. Journal of Computational Physics, 442:110467, 06 2021.
  68. High-order methods for hypersonic flows with strong shocks and real chemistry. Journal of Computational Physics, 490:112310, 2023.
  69. The MathWorks Inc. Matlab version: 9.13.0 (r2022b), 2022.
Citations (11)
View on Semantic Scholar
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-Up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube