Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal (2403.07684v1)

Published 12 Mar 2024 in cs.CV

Abstract: Real-world vision tasks frequently suffer from the appearance of unexpected adverse weather conditions, including rain, haze, snow, and raindrops. In the last decade, convolutional neural networks and vision transformers have yielded outstanding results in single-weather video removal. However, due to the absence of appropriate adaptation, most of them fail to generalize to other weather conditions. Although ViWS-Net is proposed to remove adverse weather conditions in videos with a single set of pre-trained weights, it is seriously blinded by seen weather at train-time and degenerates when coming to unseen weather during test-time. In this work, we introduce test-time adaptation into adverse weather removal in videos, and propose the first framework that integrates test-time adaptation into the iterative diffusion reverse process. Specifically, we devise a diffusion-based network with a novel temporal noise model to efficiently explore frame-correlated information in degraded video clips at training stage. During inference stage, we introduce a proxy task named Diffusion Tubelet Self-Calibration to learn the primer distribution of test video stream and optimize the model by approximating the temporal noise model for online adaptation. Experimental results, on benchmark datasets, demonstrate that our Test-Time Adaptation method with Diffusion-based network(Diff-TTA) outperforms state-of-the-art methods in terms of restoring videos degraded by seen weather conditions. Its generalizable capability is also validated with unseen weather conditions in both synthesized and real-world videos.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Parameter-free online test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8344–8353, 2022.
  2. George Box. Box and jenkins: time series analysis, forecasting and control. In A Very British Affair: Six Britons and the Development of Time Series Analysis During the 20th Century, pages 161–215. Springer, 2013.
  3. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 295–305, 2022a.
  4. Snow removal in video: A new dataset and a novel method. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13211–13222, 2023a.
  5. Robust video content alignment and compensation for rain removal in a cnn framework. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6286–6295, 2018.
  6. Simple baselines for image restoration. In European Conference on Computer Vision, pages 17–33. Springer, 2022b.
  7. Cplformer: Cross-scale prototype learning transformer for image snow removal. In Proceedings of the 31st ACM International Conference on Multimedia, pages 4228–4239, 2023b.
  8. Uncertainty-driven dynamic degradation perceiving and background modeling for efficient single image desnowing. In Proceedings of the 31st ACM International Conference on Multimedia, pages 4269–4280, 2023c.
  9. Jstasr: Joint size and transparency-aware snow removal algorithm based on modified partial convolution and veiling effect removal. In European Conference on Computer Vision, pages 754–770. Springer, 2020.
  10. All snow removed: Single image desnowing algorithm using hierarchical dual-tree complex wavelet representation and contradict channel loss. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4196–4205, 2021.
  11. Learning multiple adverse weather removal via two-stage knowledge learning and multi-contrastive regularization: Toward a unified model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17653–17662, 2022c.
  12. Symbolic discovery of optimization algorithms. arXiv preprint arXiv:2302.06675, 2023d.
  13. Snow mask guided adaptive residual network for image snow removal. arXiv preprint arXiv:2207.04754, 2022.
  14. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  15. Multi-scale boosted dehazing network with dense feature fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2157–2167, 2020.
  16. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  17. Fully test-time adaptation for image segmentation. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part III 24, pages 251–260. Springer, 2021.
  18. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems, 34:2427–2440, 2021.
  19. Unified multi-weather visibility restoration. IEEE Transactions on Multimedia, 2022.
  20. Video rain streak removal by multiscale convolutional sparse coding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6644–6653, 2018.
  21. All in one bad weather removal using architectural search. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3175–3185, 2020.
  22. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International conference on machine learning, pages 6028–6039. PMLR, 2020.
  23. Recurrent video restoration transformer with guided deformable attention. arXiv preprint arXiv:2206.02146, 2022.
  24. Erase or fill? deep joint recurrent rain removal and reconstruction in videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3233–3242, 2018a.
  25. Single-domain generalization in medical image segmentation via test-time adaptation from shape dictionary. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1756–1764, 2022a.
  26. Griddehazenet: Attention-based multi-scale network for image dehazing. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7314–7323, 2019.
  27. More control for free! image synthesis with semantic diffusion guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 289–299, 2023.
  28. Phase-based memory network for video dehazing. In Proceedings of the 30th ACM International Conference on Multimedia, pages 5427–5435, 2022b.
  29. Desnownet: Context-aware deep network for snow removal. IEEE Transactions on Image Processing, 27(6):3064–3073, 2018b.
  30. Image restoration with mean-reverting stochastic differential equations. arXiv preprint arXiv:2301.11699, 2023a.
  31. Refusion: Enabling large-size realistic image restoration with latent-space diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1680–1691, 2023b.
  32. Tipi: Test time adaptation with transformation invariance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24162–24171, 2023.
  33. Efficient test-time model adaptation without forgetting. In International conference on machine learning, pages 16888–16905. PMLR, 2022.
  34. Restoring vision in adverse weather conditions with patch-based denoising diffusion models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  35. Progressive image deraining networks: A better and simpler baseline. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3937–3946, 2019.
  36. Deep video dehazing with semantic segmentation. IEEE transactions on image processing, 28(4):1895–1908, 2018.
  37. High-fidelity guided image synthesis with latent diffusion models. arXiv preprint arXiv:2211.17084, 2022.
  38. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
  39. Transweather: Transformer-based restoration of images degraded by adverse weather conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2353–2363, 2022.
  40. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
  41. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726, 2020.
  42. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7201–7211, 2022a.
  43. Rethinking video rain streak removal: A new synthesis model and a deraining network with video rain prior. In European Conference on Computer Vision, pages 565–582. Springer, 2022b.
  44. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  45. Mask-guided progressive network for joint raindrop and rain streak removal in videos. In Proceedings of the 31st ACM International Conference on Multimedia, pages 7216–7225, 2023.
  46. Diffir: Efficient diffusion model for image restoration. arXiv preprint arXiv:2303.09472, 2023.
  47. Video dehazing via a multi-range temporal alignment network with physical prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18053–18062, 2023.
  48. Self-aligned video deraining with transmission-depth consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11966–11976, 2021.
  49. Dltta: Dynamic learning rate for test-time adaptation on cross-domain medical images. IEEE Transactions on Medical Imaging, 41(12):3575–3586, 2022.
  50. Frame-consistent recurrent video deraining with dual-level flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1661–1670, 2019.
  51. Self-learning video rain streak removal: When cyclic consistency meets temporal correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1720–1729, 2020.
  52. Video adverse-weather-component suppression network via weather messenger and adversarial backpropagation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13200–13210, 2023a.
  53. Diffmic: Dual-guidance diffusion network for medical image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 95–105. Springer, 2023b.
  54. Adverse weather removal with codebook priors. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12653–12664, 2023.
  55. Semi-supervised video deraining with dynamical rain generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 642–652, 2021.
  56. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14821–14831, 2021.
  57. Domainadaptor: A novel approach to test-time adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18971–18981, 2023a.
  58. Deep dense multi-scale network for snow removal using semantic and depth priors. IEEE Transactions on Image Processing, 30:7419–7431, 2021a.
  59. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023b.
  60. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 35:38629–38642, 2022.
  61. Learning to restore hazy video: A new real-world dataset and a new method. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9239–9248, 2021b.
  62. A unified conditional framework for diffusion-based image restoration. arXiv preprint arXiv:2305.20049, 2023c.
  63. Revisiting temporal alignment for video restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6053–6062, 2022.
  64. Learning weather-general and weather-specific features for image restoration under multiple adverse weather conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21747–21758, 2023.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now