Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Precipitation Downscaling with Spatiotemporal Video Diffusion (2312.06071v3)

Published 11 Dec 2023 in cs.CV, cs.LG, physics.ao-ph, and stat.ML

Abstract: In climate science and meteorology, high-resolution local precipitation (rain and snowfall) predictions are limited by the computational costs of simulation-based methods. Statistical downscaling, or super-resolution, is a common workaround where a low-resolution prediction is improved using statistical approaches. Unlike traditional computer vision tasks, weather and climate applications require capturing the accurate conditional distribution of high-resolution given low-resolution patterns to assure reliable ensemble averages and unbiased estimates of extreme events, such as heavy rain. This work extends recent video diffusion models to precipitation super-resolution, employing a deterministic downscaler followed by a temporally-conditioned diffusion model to capture noise characteristics and high-frequency patterns. We test our approach on FV3GFS output, an established large-scale global atmosphere model, and compare it against six state-of-the-art baselines. Our analysis, capturing CRPS, MSE, precipitation distributions, and qualitative aspects using California and the Himalayas as examples, establishes our method as a new standard for data-driven precipitation downscaling.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8503–8512, 2020.
  2. SUPERVEGAN: Super resolution video enhancement GAN for perceptually improving low bitrate streams. IEEE Access, 9:91160–91174, 2021.
  3. Motion deblurring and super-resolution from an image sequence. In Computer Vision—ECCV’96: 4th European Conference on Computer Vision Cambridge, UK, April 15–18, 1996 Proceedings Volume II 4, pages 571–582. Springer, 1996.
  4. Basicvsr: The search for essential components in video super-resolution and beyond. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4947–4956, 2021.
  5. Impact of warmer sea surface temperature on the global pattern of intense convection: insights from a global storm resolving model. Geophysical Research Letters, 49(16):e2022GL099796, 2022.
  6. Second-order attention network for single image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11065–11074, 2019.
  7. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  8. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2):295–307, 2015.
  9. Fast and robust multiframe super resolution. IEEE transactions on image processing, 13(10):1327–1344, 2004.
  10. Efficient video super-resolution through recurrent latent space propagation. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 3476–3485. IEEE, 2019.
  11. Enhancing spatial variability representation of radar nowcasting with generative adversarial networks. Remote Sensing, 15(13):3306, 2023.
  12. Generating physically-consistent high-resolution climate data with hard-constrained neural networks. arXiv preprint arXiv:2208.05424, 2022.
  13. Gfdl shield: A unified system for weather-to-seasonal prediction. Journal of Advances in Modeling Earth Systems, 12(10):e2020MS002223, 2020.
  14. A generative deep learning approach to stochastic downscaling of precipitation forecasts. Journal of Advances in Modeling Earth Systems, 14(10):e2022MS003120, 2022.
  15. Flexible diffusion modeling of long videos. In Advances in Neural Information Processing Systems, 2022.
  16. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  17. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  18. Bidirectional recurrent convolutional networks for multi-frame super-resolution. Advances in neural information processing systems, 28, 2015.
  19. Video super-resolution with convolutional neural networks. IEEE transactions on computational imaging, 2(2):109–122, 2016.
  20. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1646–1654, 2016.
  21. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  22. A field guide to dynamical recurrent networks. John Wiley & Sons, 2001.
  23. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761, 2020.
  24. Graphcast: Learning skillful medium-range global weather forecasting. arXiv preprint arXiv:2212.12794, 2022.
  25. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681–4690, 2017.
  26. Stochastic super-resolution for downscaling time-evolving atmospheric fields with a generative adversarial network. IEEE Transactions on Geoscience and Remote Sensing, 59(9):7211–7223, 2020.
  27. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1833–1844, 2021.
  28. Vrt: A video restoration transformer. arXiv preprint arXiv:2201.12288, 2022a.
  29. Recurrent video restoration transformer with guided deformable attention. In Advances in Neural Information Processing Systems, 2022b.
  30. Video super-resolution based on deep learning: a comprehensive survey. Artificial Intelligence Review, 55(8):5981–6035, 2022.
  31. Dvc: An end-to-end deep video compression framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11006–11015, 2019.
  32. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2837–2845, 2021.
  33. Generative residual diffusion modeling for km-scale atmospheric downscaling. arXiv preprint arXiv:2309.15214, 2023.
  34. Pulse: Self-supervised photo upsampling via latent space exploration of generative models. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 2437–2445, 2020.
  35. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202.11214, 2022.
  36. Increasing the accuracy and resolution of precipitation forecasts using deep generative models. In International conference on artificial intelligence and statistics, pages 10555–10571. PMLR, 2022.
  37. Micro-batch training with batch-channel normalization and weight standardization. arXiv preprint arXiv:1903.10520, 2019.
  38. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125, 2022.
  39. Skilful precipitation nowcasting using deep generative models of radar. Nature, 597(7878):672–677, 2021.
  40. Elf-vc: Efficient learned flexible-rate video coding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14479–14488, 2021.
  41. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  42. Video restoration based on deep learning: a comprehensive survey. Artificial Intelligence Review, pages 1–48, 2022.
  43. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, 2022a.
  44. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–14, 2022b.
  45. Frame-recurrent video super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6626–6634, 2018.
  46. A high-resolution climate model for the us pacific northwest: Mesoscale feedbacks and local responses to climate change. Journal of climate, 21(21):5708–5726, 2008.
  47. Progressive distillation for fast sampling of diffusion models. ArXiv, abs/2202.00512, 2022.
  48. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256–2265, 2015.
  49. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
  50. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
  51. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  52. Dyamond: the dynamics of the atmospheric general circulation modeled on non-hydrostatic domains. Progress in Earth and Planetary Science, 6(1):1–17, 2019.
  53. Statistical downscaling and dynamical downscaling of regional climate in china: Present climate evaluations and future climate projections. Journal of Geophysical Research: Atmospheres, 121(5):2110–2129, 2016.
  54. Physics-informed deep learning framework to model intense precipitation events at super resolution. Geoscience Letters, 10(1):19, 2023.
  55. Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661–1674, 2011.
  56. Deep learning for downscaling tropical cyclone rainfall to hazard-relevant spatial scales. Journal of Geophysical Research: Atmospheres, page e2022JD038163, 2023.
  57. Deep learning for image super-resolution: A survey. IEEE transactions on pattern analysis and machine intelligence, 43(10):3365–3387, 2020.
  58. Fourier neural operators for arbitrary resolution climate data downscaling. arXiv preprint arXiv:2305.14452, 2023.
  59. Hierarchical autoregressive modeling for neural video compression. In International Conference on Learning Representations, 2021a.
  60. Insights from generative modeling for neural video compression. arXiv preprint arXiv:2107.13136, 2021b.
  61. Diffusion probabilistic modeling for video generation. arXiv preprint arXiv:2203.09481, 2022.
  62. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European conference on computer vision (ECCV), pages 286–301, 2018.
  63. Towards deeper understanding of variational autoencoding models. arXiv preprint arXiv:1702.08658, 2017.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Prakhar Srivastava (4 papers)
  2. Ruihan Yang (43 papers)
  3. Gavin Kerrigan (9 papers)
  4. Gideon Dresdner (9 papers)
  5. Jeremy McGibbon (9 papers)
  6. Christopher Bretherton (3 papers)
  7. Stephan Mandt (100 papers)
Citations (1)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets