Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Driving-Video Dehazing with Non-Aligned Regularization for Safety Assistance (2405.09996v1)

Published 16 May 2024 in cs.CV

Abstract: Real driving-video dehazing poses a significant challenge due to the inherent difficulty in acquiring precisely aligned hazy/clear video pairs for effective model training, especially in dynamic driving scenarios with unpredictable weather conditions. In this paper, we propose a pioneering approach that addresses this challenge through a nonaligned regularization strategy. Our core concept involves identifying clear frames that closely match hazy frames, serving as references to supervise a video dehazing network. Our approach comprises two key components: reference matching and video dehazing. Firstly, we introduce a non-aligned reference frame matching module, leveraging an adaptive sliding window to match high-quality reference frames from clear videos. Video dehazing incorporates flow-guided cosine attention sampler and deformable cosine attention fusion modules to enhance spatial multiframe alignment and fuse their improved information. To validate our approach, we collect a GoProHazy dataset captured effortlessly with GoPro cameras in diverse rural and urban road environments. Extensive experiments demonstrate the superiority of the proposed method over current state-of-the-art methods in the challenging task of real driving-video dehazing. Project page.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (75)
  1. Nh-haze: An image dehazing benchmark with non-homogeneous hazy and haze-free images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pages 444–445, 2020.
  2. Non-local image dehazing. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 1674–1682, 2016.
  3. Basicvsr++: Improving video super-resolution with enhanced propagation and alignment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 5972–5981, 2022.
  4. Robust image and video dehazing with visual artifact suppression via gradient residual minimization. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 576–591. Springer, 2016.
  5. Robust video content alignment and compensation for rain removal in a cnn framework. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 6286–6295, 2018.
  6. Unpaired deep image dehazing using contrastive disentanglement learning. In European Conference on Computer Vision (ECCV), pages 632–648. Springer, 2022.
  7. Psd: Principled synthetic-to-real dehazing guided by physical priors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 7180–7189, 2021.
  8. Referenceless prediction of perceptual fog density and perceptual image defogging. IEEE Transactions on Image Processing (TIP), 24(11):3888–3901, 2015.
  9. Discrete haze level dehazing network. In Proceedings of the 28th ACM International Conference on Multimedia (ACMMM), pages 1828–1836, 2020.
  10. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision (ICCV), pages 764–773, 2017.
  11. Video demoireing with relation-based temporal consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17622–17631, 2022.
  12. Hardgan: A haze-aware representation distillation gan for single image dehazing. In European conference on computer vision (ECCV), pages 722–738. Springer, 2020.
  13. Deep multi-model fusion for single-image dehazing. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pages 2453–2462, 2019.
  14. Fd-gan: Generative adversarial networks with fusion-discriminator for single image dehazing. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 10729–10736, 2020.
  15. Non-aligned supervision for real image dehazing. arXiv preprint arXiv:2303.04940, 2023.
  16. Raanan Fattal. Dehazing using color-lines. ACM transactions on graphics (TOG), 34(1):1–14, 2014.
  17. Generative adversarial nets. Advances in neural information processing systems (NeurIPS), 27, 2014.
  18. Image dehazing transformer with transmission-aware 3d position embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5812–5820, 2022.
  19. Fog simulation on real lidar point clouds for 3d object detection in adverse weather. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15283–15292, 2021.
  20. Single image haze removal using dark channel prior. IEEE transactions on pattern analysis and machine intelligence (TPAMI), 33(12):2341–2353, 2010.
  21. Neural compression-based feature learning for video restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5872–5881, 2022.
  22. Spatio-temporal transformer network for video restoration. In Proceedings of the European conference on computer vision (ECCV), pages 106–122, 2018.
  23. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  24. End-to-end united video dehazing and detection. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2018a.
  25. Benchmarking single-image dehazing and beyond. IEEE Transactions on Image Processing (TIP), 28(1):492–505, 2018b.
  26. Zero-shot image dehazing. IEEE Transactions on Image Processing (TIP), 29:8457–8466, 2020.
  27. You only look yourself: Unsupervised and untrained single image dehazing neural network. International Journal of Computer Vision (ICCV), 129:1754–1767, 2021.
  28. Domain adaptive object detection for autonomous driving under foggy weather. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 612–622, 2023.
  29. Single image dehazing via conditional generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 8202–8211, 2018c.
  30. Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 1633–1642, 2019a.
  31. Lap-net: Level-aware progressive network for image dehazing. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pages 3276–3285, 2019b.
  32. Simultaneous video defogging and stereo reconstruction. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 4988–4997, 2015.
  33. Recurrent video restoration transformer with guided deformable attention. Advances in Neural Information Processing Systems (NeurIPS), 35:378–393, 2022.
  34. Towards multi-domain single image dehazing via test-time training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5831–5840, 2022a.
  35. Erase or fill? deep joint recurrent rain removal and reconstruction in videos. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 3233–3242, 2018.
  36. Griddehazenet: Attention-based multi-scale network for image dehazing. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pages 7314–7323, 2019a.
  37. Learning deep priors for image dehazing. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pages 2492–2500, 2019b.
  38. Phase-based memory network for video dehazing. In Proceedings of the 30th ACM International Conference on Multimedia (ACMMM), pages 5427–5435, 2022b.
  39. Earl J McCartney. Optics of the atmosphere: scattering by molecules and particles. New York, 1976.
  40. The contextual loss for image transformation with non-aligned data. In Proceedings of the European conference on computer vision (ECCV), pages 768–783, 2018.
  41. Making a “completely blind” image quality analyzer. IEEE Signal processing letters, 20(3):209–212, 2012.
  42. Image dehazing by joint estimation of transmittance and airlight using bi-directional consistency loss minimized fcn. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (CVPRW), pages 920–928, 2018.
  43. Vision and the atmosphere. International journal of computer vision (IJCV), 48:233–254, 2002.
  44. Bidnet: Binocular image dehazing without explicit disparity estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 5931–5940, 2020.
  45. Multi-weather image restoration via domain translation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 21696–21705, 2023.
  46. Ffa-net: Feature fusion attention network for single image dehazing. In Proceedings of the AAAI conference on artificial intelligence (AAAI), pages 11908–11915, 2020.
  47. Mb-taylorformer: Multi-branch efficient transformer expanded by taylor formula for image dehazing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12802–12813, 2023.
  48. Enhanced pix2pix dehazing network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 8160–8168, 2019.
  49. Optical flow estimation using a spatial pyramid network. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 4161–4170, 2017.
  50. Deep video dehazing with semantic segmentation. IEEE transactions on image processing (TIP), 28(4):1895–1908, 2018.
  51. Domain adaptation for image dehazing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 2808–2817, 2020.
  52. Towards domain invariant single image dehazing. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 9657–9665, 2021.
  53. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  54. Simultaneous deep stereo matching and dehazing with feature attention. International Journal of Computer Vision (IJCV), 128:799–817, 2020.
  55. Tdan: Temporally-deformable alignment network for video super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 3360–3369, 2020.
  56. Transweather: Transformer-based restoration of images degraded by adverse weather conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2353–2363, 2022.
  57. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pages 0–0, 2019.
  58. Accurate transmission estimation for removing haze and noise from a single image. IEEE transactions on image processing (TIP), 29:2583–2597, 2019.
  59. Ridcp: Revitalizing real image dehazing via high-quality codebook priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22282–22291, 2023.
  60. Video dehazing via a multi-range temporal alignment network with physical prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18053–18062, 2023.
  61. Frame-consistent recurrent video deraining with dual-level flow. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 1661–1670, 2019.
  62. Towards perceptual image dehazing by physics-based disentanglement and adversarial training. In Proceedings of the AAAI conference on artificial intelligence (AAAI), 2018.
  63. Self-augmented unpaired image dehazing via density and depth decomposition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 2037–2046, 2022.
  64. Video adverse-weather-component suppression network via weather messenger and adversarial backpropagation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 13200–13210, 2023.
  65. Perceiving and modeling density for image dehazing. In European Conference on Computer Vision (ECCV), pages 130–145. Springer, 2022.
  66. Source-free domain adaptation for real-world image dehazing. In Proceedings of the 30th ACM International Conference on Multimedia (ACMMM), pages 6645–6654, 2022a.
  67. Memory-augmented non-local attention for video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17834–17843, 2022b.
  68. Densely connected pyramid dehazing network. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 3194–3203, 2018.
  69. Joint transmission map estimation and dehazing using deep networks. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 30(7):1975–1986, 2019.
  70. Spatio-temporal deformable attention network for video deblurring. In European Conference on Computer Vision (ECCV), pages 581–596. Springer, 2022.
  71. Learning to restore hazy video: A new real-world dataset and a new method. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9239–9248, 2021.
  72. Refinednet: A weakly supervised refinement framework for single image dehazing. IEEE Transactions on Image Processing (TIP), 30:3391–3404, 2021.
  73. Fast segment anything. arXiv preprint arXiv:2306.12156, 2023.
  74. Spatio-temporal filter adaptive network for video deblurring. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pages 2482–2491, 2019.
  75. A fast single image haze removal algorithm using color attenuation prior. IEEE transactions on image processing (TIP), 24(11):3522–3533, 2015.
Citations (4)
View on Semantic Scholar

Summary

  • The paper introduces a non-aligned regularization strategy combining NRFM, FCAS, and DCAF modules to improve dehazing in dynamic driving scenarios.
  • It employs an adaptive sliding window and optical flow-guided cosine attention to align multi-frame features without strict ground-truth pairs.
  • Experiments on the GoProHazy dataset reveal superior dehazing performance with enhanced FADE and NIQE scores, promising improved safety for autonomous vehicles.

An Examination of Video Dehazing Techniques in Driving Scenarios

The paper "Driving-Video Dehazing with Non-Aligned Regularization for Safety Assistance" introduces a sophisticated framework for addressing the video dehazing challenges prevalent in real-world driving environments. This work is particularly significant considering the adverse impacts of haze, such as reduced visibility and contrast, which pose safety risks for autonomous driving systems. The authors tackle the intrinsic difficulty of obtaining accurately aligned hazy/clear video pairs necessary for model training, which is especially challenging in dynamic driving scenarios with unpredictable weather conditions.

Methodology Overview

The proposed dehazing method is centered around a novel non-aligned regularization strategy. This approach consists of two primary components: non-aligned reference frame matching and video dehazing. The authors introduced a Non-aligned Reference Frame Matching (NRFM) module that employs an adaptive sliding window to match high-quality clear reference frames from non-aligned videos. This strategy relaxes the requirement for precisely aligned ground truth during training.

For the video dehazing component, the paper introduces a Flow-guided Cosine Attention Sampler (FCAS) and a Deformable Cosine Attention Fusion (DCAF) module. FCAS improves multi-frame alignment by employing coarse optical flow to guide multi-scale cosine attention sampling. This enables the model to tackle large motion challenges in driving videos by improving spatial alignment. In contrast, DCAF leverages deformable convolution and cosine similarity for better multi-frame feature fusion, thereby enhancing dehazing performance.

Dataset and Results

The authors curated a real-world video dataset named GoProHazy, captured using GoPro cameras in various rural and urban road settings. This dataset was used to validate the proposed methodologies under realistic and diverse conditions. Through extensive experimentation, the proposed method showcased superiority over contemporary state-of-the-art techniques. For instance, the FADE and NIQE scores, which are common metrics for evaluating dehazing quality, demonstrated the method's enhanced performance in real-world video dehazing tasks without requiring strict temporal and spatial alignment of training data.

Implications and Future Directions

The presented approach promises improved visibility in autonomous vehicle systems, particularly under hazy conditions, ultimately enhancing safety. The framework's ability to operate effectively even with non-aligned video input is particularly notable, as it suggests potential applications in other domains where acquiring aligned data is difficult.

The paper's contributions open several avenues for future research. These include exploring the integration of this framework with other adverse weather conditions like rain or snow, extending its applicability across various sensor types beyond RGB cameras, and refining the model to operate in real-time scenarios. Moreover, the techniques developed could be further investigated to improve robustness and adapt to different haze intensities and dynamic driving environments.

Overall, this research makes a significant contribution to the field of computer vision and autonomous systems, addressing complex challenges with innovative solutions and demonstrating practical applicability through its superior results in real-world settings.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube