Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Turb-Seg-Res: A Segment-then-Restore Pipeline for Dynamic Videos with Atmospheric Turbulence (2404.13605v1)

Published 21 Apr 2024 in cs.CV and eess.IV

Abstract: Tackling image degradation due to atmospheric turbulence, particularly in dynamic environment, remains a challenge for long-range imaging systems. Existing techniques have been primarily designed for static scenes or scenes with small motion. This paper presents the first segment-then-restore pipeline for restoring the videos of dynamic scenes in turbulent environment. We leverage mean optical flow with an unsupervised motion segmentation method to separate dynamic and static scene components prior to restoration. After camera shake compensation and segmentation, we introduce foreground/background enhancement leveraging the statistics of turbulence strength and a transformer model trained on a novel noise-based procedural turbulence generator for fast dataset augmentation. Benchmarked against existing restoration methods, our approach restores most of the geometric distortion and enhances sharpness for videos. We make our code, simulator, and data publicly available to advance the field of video restoration from turbulence: riponcs.github.io/TurbSegRes

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Nantheera Anantrasirichai. Atmospheric turbulence removal with complex-valued convolutional neural network. Pattern Recognition Letters, 171:69–75, 2023.
  2. Atmospheric turbulence mitigation using complex wavelet-based fusion. IEEE Transactions on Image Processing, 22(6):2398–2408, 2013.
  3. Automated video enhancement from a stream of atmospherically-distorted images: the lucky-region fusion approach. Atmospheric Optics: Models, Measurements, and Target-in-the-Loop Propagation III, 7463:104–113, 2009.
  4. Measurement of the probability of getting a lucky short-exposure image through turbulence. JOSA, 71(9):1138–1139, 1981.
  5. Technique for simulating anisoplanatic image formation over long horizontal paths. Optical Engineering, 51(10):101704–101704, 2012.
  6. Stanley H Chan. Tilt-then-blur or blur-then-tilt? clarifying the atmospheric turbulence model. IEEE Signal Processing Letters, 29:1833–1837, 2022.
  7. Detecting and tracking moving objects in long-distance imaging through turbulent medium. Applied Optics, 53(6):1181–1190, 2014.
  8. Simulating anisoplanatic turbulence by sampling intermodal and spatially correlated zernike coefficients. Optical Engineering, 59(8):083101, 2020.
  9. Real-time dense field phase-to-space simulation of imaging through atmospheric turbulence. IEEE Transactions on Computational Imaging, 8:1159–1169, 2022.
  10. Treating motion as option to reduce motion dependency in unsupervised video object segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5140–5149, 2023.
  11. Effects of turbulence on imaging through the atmosphere. In Optical Properties of the Atmosphere, pages 130–134. SPIE, 1978.
  12. Accurate semantic segmentation in turbulence media. IEEE Access, 7:166749–166761, 2019.
  13. Atmospheric turbulence removal in long-range imaging using a data-driven-based approach. International Journal of Computer Vision, 130(4):1031–1049, 2022.
  14. David L Fried. Statistics of a geometric representation of wavefront distortion. JOSA, 55(11):1427–1435, 1965.
  15. David L Fried. Probability of getting a lucky short-exposure image through turbulence. JOSA, 68(12):1651–1658, 1978.
  16. David L Fried. Anisoplanatism in adaptive optics. JOSA, 72(1):52–61, 1982.
  17. Restoring atmospheric-turbulence-degraded images. Applied Optics, 55(19):5082–5090, 2016.
  18. Progress in the restoration of image sequences degraded by atmospheric turbulence. Pattern Recognition Letters, 48:8–14, 2014.
  19. Open turbulent image set (otis). Pattern Recognition Letters, 86:38–41, 2017.
  20. R Gray. Zernikecalc: a matlab function to work with zernike polynomials over circular and non-circular pupils, 2019.
  21. Simulation of atmospheric phenomena. Computers & Graphics, 30(6):994–1010, 2006.
  22. Simulation of anisoplanatic imaging through optical turbulence using numerical wave propagation with new validation analysis. Optical Engineering, 56(7):071502–071502, 2017.
  23. Synthesis of atmospheric turbulence point spread functions by sparse and redundant representations. Optical Engineering, 57(2):024101, 2018.
  24. Eikonal rendering: Efficient light transport in refractive objects. ACM Transactions on Graphics (TOG), 26(3):59–es, 2007.
  25. A. Ishimaru. Wave Propagation and Scattering in Random Media. Wiley, 1999.
  26. Nert: Implicit neural representations for unsupervised atmospheric turbulence mitigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4235–4242, 2023.
  27. Andrey Nikolaevich Kolmogorov. Dissipation of Energy in Locally Isotropic Turbulence. Akademiia Nauk SSSR Doklady, 32:16, 1941.
  28. Andrei Nikolaevich Kolmogorov. The local structure of turbulence in incompressible viscous fluid for very large reynolds numbers. Proceedings of the Royal Society of London. Series A: Mathematical and Physical Sciences, 434(1890):9–13, 1991.
  29. N.S. Kopeika. A System Engineering Approach to Imaging. SPIE Optical Engineering Press, 1998.
  30. A survey of procedural noise functions. Computer Graphics Forum, 29(8):2579–2600, 2010.
  31. Restoration of atmospheric turbulence-distorted images via rpca and quasiconformal maps. Inverse Problems, 35(7):074002, 2019.
  32. Simulation of atmospheric turbulence effects and mitigation algorithms on stand-off automatic facial recognition. In Optics and Photonics for Counterterrorism, Crime Fighting, and Defence VIII, pages 182–198. SPIE, 2012.
  33. Atmospheric turbulence-degraded image restoration using principal components analysis. IEEE Geoscience and Remote Sensing Letters, 4(3):340–344, 2007.
  34. Unsupervised non-rigid image distortion removal via grid deformation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2522–2532, 2021.
  35. Ds-net: Dynamic spatiotemporal network for video salient object detection. Digital Signal Processing, 130:103700, 2022.
  36. Image reconstruction of static and dynamic scenes through anisoplanatic turbulence. IEEE Transactions on Computational Imaging, 6:1415–1428, 2020.
  37. Accelerating atmospheric turbulence simulation via learned phase-to-space transform. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14759–14768, 2021.
  38. Single frame atmospheric turbulence mitigation: A benchmark study and a new physics-inspired transformer model. In European Conference on Computer Vision, pages 430–446. Springer, 2022.
  39. Robust detection of lines using the progressive probabilistic hough transform. Comput. Vis. Image Underst., 78:119–137, 2000.
  40. Real-time shading languages. SIGGRAPH Course Notes, 2002.
  41. Poisson image editing. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 577–582. 2023.
  42. Ken Perlin. An image synthesizer. In Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques, page 287–296, New York, NY, USA, 1985. Association for Computing Machinery.
  43. A parametric model for simulating turbulence effects on imaging systems. DRDC Valcartier TR 2006, 787, 2007.
  44. A simple physical model for simulating turbulent imaging. In Infrared Imaging Systems: Design, Analysis, Modeling, and Testing XXII, pages 323–335. SPIE, 2011.
  45. Unsupervised region-growing network for object segmentation in atmospheric turbulence. arXiv:2311.03572, 2023.
  46. Efficient physics-based predictive 3d image modeling and simulation of optical atmospheric refraction phenomena. In Laser Communication and Propagation through the Atmosphere and Oceans V, page 99790S. International Society for Optics and Photonics, 2016.
  47. Computer simulation of image degradations by atmospheric turbulence for horizontal views. In Infrared Imaging Systems: Design, Analysis, Modeling, and Testing XXII, page 80140U. International Society for Optics and Photonics, 2011.
  48. Survey on digital video stabilization: Concepts, methods, and challenges. ACM Comput. Surv., 55(3), 2022.
  49. Imaging Through Turbulence. Taylor & Francis, 1996.
  50. Michael C Roggemann. Optical performance of fully and partially compensated adaptive optics systems using least-squares and minimum variance phase reconstructors. Computers & Electrical Engineering, 18(6):451–466, 1992.
  51. Turbulence strength cn2superscriptsubscript𝑐𝑛2c_{n}^{2}italic_c start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT estimation from video using physics-based deep learning. Optics Express, 30(22):40854–40870, 2022.
  52. Turbulence-induced 2d correlated image distortion. In 2017 IEEE International Conference on Computational Photography (ICCP), pages 1–13. IEEE, 2017.
  53. Valerian Ilich Tatarski. Wave propagation in a turbulent medium. Courier Dover Publications, 2016.
  54. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pages 402–419. Springer, 2020.
  55. Removing turbulence effect via hybrid total variation and deformation-guided kernel regression. IEEE Transactions on Image Processing, 25(10):4943–4958, 2016.
  56. Learning to restore images degraded by atmospheric turbulence using uncertainty. In 2021 IEEE International Conference on Image Processing (ICIP), pages 1694–1698. IEEE, 2021.
  57. Deformable sprites for unsupervised video decomposition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2657–2666, 2022.
  58. Turbulence strength estimation from an arbitrary set of atmospherically degraded images. JOSA A, 23(12):3106–3113, 2006.
  59. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5728–5739, 2022.
  60. Imaging through the atmosphere using turbulence mitigation transformer. IEEE Transactions on Computational Imaging, 2024.
  61. Removing atmospheric turbulence via space-invariant deconvolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1):157–170, 2012.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now