Papers
Topics
Authors
Recent
Search
2000 character limit reached

RSHazeDiff: A Unified Fourier-aware Diffusion Model for Remote Sensing Image Dehazing

Published 15 May 2024 in cs.CV | (2405.09083v2)

Abstract: Haze severely degrades the visual quality of remote sensing images and hampers the performance of road extraction, vehicle detection, and traffic flow monitoring. The emerging denoising diffusion probabilistic model (DDPM) exhibits the significant potential for dense haze removal with its strong generation ability. Since remote sensing images contain extensive small-scale texture structures, it is important to effectively restore image details from hazy images. However, current wisdom of DDPM fails to preserve image details and color fidelity well, limiting its dehazing capacity for remote sensing images. In this paper, we propose a novel unified Fourier-aware diffusion model for remote sensing image dehazing, termed RSHazeDiff. From a new perspective, RSHazeDiff explores the conditional DDPM to improve image quality in dense hazy scenarios, and it makes three key contributions. First, RSHazeDiff refines the training phase of diffusion process by performing noise estimation and reconstruction constraints in a coarse-to-fine fashion. Thus, it remedies the unpleasing results caused by the simple noise estimation constraint in DDPM. Second, by taking the frequency information as important prior knowledge during iterative sampling steps, RSHazeDiff can preserve more texture details and color fidelity in dehazed images. Third, we design a global compensated learning module to utilize the Fourier transform to capture the global dependency features of input images, which can effectively mitigate the effects of boundary artifacts when processing fixed-size patches. Experiments on both synthetic and real-world benchmarks validate the favorable performance of RSHazeDiff over state-of-the-art methods. Source code will be released at https://github.com/jm-xiong/RSHazeDiff.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. S. Ren and Q. Liu, “Small target augmentation for urban remote sensing image real-time segmentation,” IEEE Transactions on Intelligent Transportation Systems, 2023.
  2. H. Guan, Y. Yu, D. Li, and H. Wang, “Roadcapsfpn: Capsule feature pyramid network for road extraction from vhr optical remote sensing imagery,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 8, pp. 11 041–11 051, 2021.
  3. K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 12, pp. 2341–2353, Sep. 2010.
  4. Q. Zhu, J. Mai, and L. Shao, “Single image dehazing using color attenuation prior.” in BMVC, 2014.
  5. J. Li, Q. Hu, and M. Ai, “Haze and thin cloud removal via sphere model improved dark channel prior,” IEEE Geoscience and Remote Sensing Letters, vol. 16, no. 3, pp. 472–476, 2018.
  6. L. Xu, D. Zhao, Y. Yan, S. Kwong, J. Chen, and L.-Y. Duan, “Iders: Iterative dehazing method for single remote sensing image,” Information Sciences, vol. 489, pp. 50–62, 2019.
  7. H. Shen, C. Zhang, H. Li, Q. Yuan, and L. Zhang, “A spatial–spectral adaptive haze removal method for visible remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 9, pp. 6168–6180, 2020.
  8. G. Bi, G. Si, Y. Zhao, B. Qi, and H. Lv, “Haze removal for a single remote sensing image using low-rank and sparse prior,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–13, 2021.
  9. Y. Han, M. Yin, P. Duan, and P. Ghamisi, “Edge-preserving filtering-based dehazing for remote sensing images,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2021.
  10. W. Ren, S. Liu, H. Zhang, J. Pan, X. Cao, and M.-H. Yang, “Single image dehazing via multi-scale convolutional neural networks,” in Proc. Eur. Conf. Comput. Vis. (ECCV), Oct. 2016, pp. 154–169.
  11. B. Cai, X. Xu, K. Jia, C. Qing, and D. Tao, “Dehazenet: An end-to-end system for single image haze removal,” IEEE transactions on image processing, vol. 25, no. 11, pp. 5187–5198, 2016.
  12. B. Li, X. Peng, Z. Wang, J. Xu, and D. Feng, “Aod-net: All-in-one dehazing network,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Oct. 2017, pp. 4770–4778.
  13. S. G. Narasimhan and S. K. Nayar, “Chromatic framework for vision in bad weather,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), vol. 1, Jun. 2000, pp. 598–605.
  14. X. Liu, Y. Ma, Z. Shi, and J. Chen, “Griddehazenet: Attention-based multi-scale network for image dehazing,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 7314–7323.
  15. X. Qin, Z. Wang, Y. Bai, X. Xie, and H. Jia, “Ffa-net: Feature fusion attention network for single image dehazing,” in Proc. AAAI Conf. Artif. Intell. (AAAI), vol. 34, no. 07, Feb. 2020, pp. 11 908–11 915.
  16. Z. Chen, Y. Wang, Y. Yang, and D. Liu, “Psd: Principled synthetic-to-real dehazing guided by physical priors,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 7180–7189.
  17. Y. Cui, W. Ren, X. Cao, and A. Knoll, “Focal network for image restoration,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 13 001–13 011.
  18. Y. Li and X. Chen, “A coarse-to-fine two-stage attentive network for haze removal of remote sensing images,” IEEE Geoscience and Remote Sensing Letters, vol. 18, no. 10, pp. 1751–1755, 2020.
  19. L. Zhang and S. Wang, “Dense haze removal based on dynamic collaborative inference learning for remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–16, 2022.
  20. W. Ren, L. Ma, J. Zhang, J. Pan, X. Cao, W. Liu, and M.-H. Yang, “Gated fusion network for single image dehazing,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3253–3261.
  21. H. Zhang and V. M. Patel, “Densely connected pyramid dehazing network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3194–3203.
  22. D. Engin, A. Genç, and H. Kemal Ekenel, “Cycle-dehaze: Enhanced cyclegan for single image dehazing,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2018, pp. 825–833.
  23. Y. Dong, Y. Liu, H. Zhang, S. Chen, and Y. Qiao, “Fd-gan: Generative adversarial networks with fusion-discriminator for single image dehazing,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 07, 2020, pp. 10 729–10 736.
  24. Y. Wang, X. Yan, D. Guan, M. Wei, Y. Chen, X.-P. Zhang, and J. Li, “Cycle-snspgan: Towards real-world image dehazing via cycle spectral normalized soft likelihood estimation patch gan,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 11, pp. 20 368–20 382, 2022.
  25. Y. Zheng, J. Su, S. Zhang, M. Tao, and L. Wang, “Dehaze-aggan: Unpaired remote sensing image dehazing using enhanced attention-guide generative adversarial networks,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–13, 2022.
  26. C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 4, pp. 4713–4726, 2022.
  27. C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi, “Palette: Image-to-image diffusion models,” in ACM SIGGRAPH 2022 Conference Proceedings, 2022, pp. 1–10.
  28. L. Guo, C. Wang, W. Yang, S. Huang, Y. Wang, H. Pfister, and B. Wen, “Shadowdiffusion: When degradation prior meets diffusion model for shadow removal,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 14 049–14 058.
  29. J. Whang, M. Delbracio, H. Talebi, C. Saharia, A. G. Dimakis, and P. Milanfar, “Deblurring via stochastic refinement,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16 293–16 303.
  30. J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020.
  31. O. Özdenizci and R. Legenstein, “Restoring vision in adverse weather conditions with patch-based denoising diffusion models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  32. L. Jiao, C. Hu, L. Huo, and P. Tang, “Guided-pix2pix: End-to-end inference and refinement network for image dehazing,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 14, pp. 3052–3069, 2021.
  33. Y. Huang and X. Chen, “Single remote sensing image dehazing using a dual-step cascaded residual dense network,” in 2021 IEEE International Conference on Image Processing (ICIP).   IEEE, 2021, pp. 3852–3856.
  34. X. Liu, Z. Shi, Z. Wu, J. Chen, and G. Zhai, “Griddehazenet+: An enhanced multi-scale network with intra-task knowledge transfer for single image dehazing,” IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 1, pp. 870–884, 2022.
  35. K. Chi, Y. Yuan, and Q. Wang, “Trinity-net: Gradient-guided swin transformer-based remote sensing image dehazing and beyond,” IEEE Transactions on Geoscience and Remote Sensing, 2023.
  36. Y. Wang, J. Xiong, X. Yan, and M. Wei, “Uscformer: unified transformer with semantically contrastive learning for image dehazing,” IEEE Transactions on Intelligent Transportation Systems, 2023.
  37. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022.
  38. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in International conference on machine learning.   PMLR, 2015, pp. 2256–2265.
  39. A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion probabilistic models,” in International Conference on Machine Learning.   PMLR, 2021, pp. 8162–8171.
  40. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 684–10 695.
  41. K. Xu, M. Qin, F. Sun, Y. Wang, Y.-K. Chen, and F. Ren, “Learning in the frequency domain,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 1740–1749.
  42. L. Chi, B. Jiang, and Y. Mu, “Fast fourier convolution,” Advances in Neural Information Processing Systems, vol. 33, pp. 4479–4488, 2020.
  43. H. Wang, X. Wu, Z. Huang, and E. P. Xing, “High-frequency component helps explain the generalization of convolutional neural networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 8684–8694.
  44. M. Zhou, H. Yu, J. Huang, F. Zhao, J. Gu, C. C. Loy, D. Meng, and C. Li, “Deep fourier up-sampling,” arXiv preprint arXiv:2210.05171, 2022.
  45. Y. Yang, D. Lao, G. Sundaramoorthi, and S. Soatto, “Phase consistent ecological domain adaptation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9011–9020.
  46. Y. Yang and S. Soatto, “Fda: Fourier domain adaptation for semantic segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 4085–4095.
  47. D. Fuoli, L. Van Gool, and R. Timofte, “Fourier space losses for efficient perceptual image super-resolution,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2360–2369.
  48. H. Yu, N. Zheng, M. Zhou, J. Huang, Z. Xiao, and F. Zhao, “Frequency and spatial dual guidance for image dehazing,” in European Conference on Computer Vision.   Springer, 2022, pp. 181–198.
  49. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18.   Springer, 2015, pp. 234–241.
  50. Q. Xu, R. Zhang, Y. Zhang, Y. Wang, and Q. Tian, “A fourier-based framework for domain generalization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 14 383–14 392.
  51. E. Prince and E. Prince, “The fast fourier transform,” Mathematical Techniques in Crystallography and Materials Science, pp. 140–156, 1994.
  52. J. I. Richards, “Review: Yitzhak katznelson, an introduction to harmonic analysis,” Bulletin of the American Mathematical Society, vol. 75, no. 4, p. 709–713. [Online]. Available: http://dx.doi.org/10.1090/s0002-9904-1969-12253-6
  53. H. Dong, J. Pan, L. Xiang, Z. Hu, X. Zhang, F. Wang, and M.-H. Yang, “Multi-scale boosted dehazing network with dense feature fusion,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 2157–2167.
  54. H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for image restoration with neural networks,” IEEE Transactions on computational imaging, vol. 3, no. 1, pp. 47–57, 2016.
  55. Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural similarity for image quality assessment,” in Proc. 37th Asilomar Conf. Signals, Syst. Comput., vol. 2, Nov. 2003, pp. 1398–1402.
  56. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  57. D. Lin, G. Xu, X. Wang, Y. Wang, X. Sun, and K. Fu, “A remote sensing image dataset for cloud removal,” arXiv preprint arXiv:1901.00600, 2019.
  58. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019.
  59. J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” arXiv preprint arXiv:2010.02502, 2020.
  60. Y. Song and S. Ermon, “Improved techniques for training score-based generative models,” Advances in neural information processing systems, vol. 33, pp. 12 438–12 448, 2020.
  61. I. Q. Assessment, “From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, p. 93, Apr. 2004.
  62. G. Sharma, W. Wu, and E. N. Dalal, “The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations,” Color Res. Appl., vol. 30, no. 1, pp. 21–30, Dec. 2004.
  63. R. H. Yuhas, A. F. Goetz, and J. W. Boardman, “Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm,” in JPL, Summaries of the Third Annual JPL Airborne Geoscience Workshop. Volume 1: AVIRIS Workshop, 1992.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free