2000 character limit reached
FreqMamba: Viewing Mamba from a Frequency Perspective for Image Deraining (2404.09476v2)
Published 15 Apr 2024 in cs.CV
Abstract: Images corrupted by rain streaks often lose vital frequency information for perception, and image deraining aims to solve this issue which relies on global and local degradation modeling. Recent studies have witnessed the effectiveness and efficiency of Mamba for perceiving global and local information based on its exploiting local correlation among patches, however, rarely attempts have been explored to extend it with frequency analysis for image deraining, limiting its ability to perceive global degradation that is relevant to frequency modeling (e.g. Fourier transform). In this paper, we propose FreqMamba, an effective and efficient paradigm that leverages the complementary between Mamba and frequency analysis for image deraining. The core of our method lies in extending Mamba with frequency analysis from two perspectives: extending it with frequency-band for exploiting frequency correlation, and connecting it with Fourier transform for global degradation modeling. Specifically, FreqMamba introduces complementary triple interaction structures including spatial Mamba, frequency band Mamba, and Fourier global modeling. Frequency band Mamba decomposes the image into sub-bands of different frequencies to allow 2D scanning from the frequency dimension. Furthermore, leveraging Mamba's unique data-dependent properties, we use rainy images at different scales to provide degradation priors to the network, thereby facilitating efficient training. Extensive experiments show that our method outperforms state-of-the-art methods both visually and quantitatively.
- Dense-haze: A benchmark for image dehazing with dense-haze and haze-free images. In 2019 IEEE international conference on image processing (ICIP). IEEE, 1014–1018.
- NH-HAZE: An image dehazing benchmark with non-homogeneous hazy and haze-free images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. 444–445.
- Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling. arXiv preprint arXiv:2402.10211 (2024).
- Dehazenet: An end-to-end system for single image haze removal. IEEE transactions on image processing 25, 11 (2016), 5187–5198.
- Multi-scale boosted dehazing network with dense feature fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2157–2167.
- Clearing the skies: A deep network architecture for single-image rain removal. IEEE TIP (2017).
- Removing rain from single images via a deep detail network. In CVPR.
- Lightweight Pyramid Networks for Image Deraining. IEEE Transactions on Neural Networks and Learning Systems 31, 6 (2020), 1794–1807. https://doi.org/10.1109/TNNLS.2019.2926481
- nnmamba: 3d biomedical image segmentation, classification and landmark detection with state space model. arXiv preprint arXiv:2402.03526 (2024).
- Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023).
- Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 (2021).
- Zero-reference deep curve estimation for low-light image enhancement. In IEEE Conf. Comput. Vis. Pattern Recog. 1780–1789.
- MambaIR: A Simple Baseline for Image Restoration with State-Space Model. arXiv preprint arXiv:2402.15648 (2024).
- Exploring Fourier Prior for Single Image Rain Removal.. In IJCAI. 935–941.
- Single image haze removal using dark channel prior. IEEE transactions on pattern analysis and machine intelligence 33, 12 (2010), 2341–2353.
- Exposure Normalization and Compensation for Multiple-Exposure Correction. In IEEE Conf. Comput. Vis. Pattern Recog. 6043–6052.
- Deep Fourier-Based Exposure Correction Network with Spatial-Frequency Interaction. In Eur. Conf. Comput. Vis. Springer, 163–180.
- LocalMamba: Visual State Space Model with Windowed Selective Scan. arXiv preprint arXiv:2403.09338 (2024).
- Multi-scale progressive fusion network for single image deraining. In CVPR.
- Automatic single-image-based rain streaks removal via image decomposition. IEEE transactions on image processing 21, 4 (2011), 1742–1755.
- Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.
- Single-image depth estimation based on Fourier domain analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 330–339.
- Embedding fourier for ultra-high-definition low-light image enhancement. arXiv preprint arXiv:2302.11831 (2023).
- Recurrent squeeze-and-excitation context aggregation net for single image deraining. In ECCV.
- Griddehazenet: Attention-based multi-scale network for image dehazing. In Proceedings of the IEEE/CVF international conference on computer vision. 7314–7323.
- Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166 (2024).
- Removing rain from a single image via discriminative sparse coding. In Proceedings of the IEEE international conference on computer vision. 3397–3405.
- Image Restoration with Mean-Reverting Stochastic Differential Equations. International Conference on Machine Learning (2023).
- U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.04722 (2024).
- A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, Vol. 2. IEEE, 416–423.
- Long range language modeling via gated state spaces. arXiv preprint arXiv:2206.13947 (2022).
- FCNN: Fourier Convolutional Neural Networks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 786–798.
- Spatially-adaptive image restoration using distortion-guided networks. In ICCV.
- VL-Mamba: Exploring State Space Models for Multimodal Learning. arXiv preprint arXiv:2403.13600 (2024).
- Progressive image deraining networks: A better and simpler baseline. In CVPR.
- VmambaIR: Visual State Space Model for Image Restoration. arXiv preprint arXiv:2403.11423 (2024).
- Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933 (2022).
- Transweather: Transformer-based restoration of images degraded by adverse weather conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2353–2363.
- Graph-mamba: Towards long-range graph sequence modeling with selective state spaces. arXiv preprint arXiv:2402.00789 (2024).
- Uformer: A general u-shaped transformer for image restoration. In CVPR.
- Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv:1808.04560 (2018).
- Contrastive learning for compact single image dehazing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10551–10560.
- Uretinex-net: Retinex-based deep unfolding network for low-light image enhancement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5901–5910.
- Image de-raining transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).
- A fourier-based framework for domain generalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 14383–14392.
- SNR-aware low-light image enhancement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 17714–17724.
- Deep joint rain detection and removal from a single image. In CVPR.
- Yanchao Yang and Stefano Soatto. 2020. FDA: Fourier Domain Adaptation for Semantic Segmentation.
- Rajeev Yasarla and Vishal M Patel. 2019. Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In CVPR.
- Frequency and spatial dual guidance for image dehazing. In European Conference on Computer Vision. Springer, 181–198.
- Restormer: Efficient transformer for high-resolution image restoration. In CVPR.
- Multi-stage progressive image restoration. In CVPR.
- He Zhang and Vishal M Patel. 2018. Density-aware single image de-raining using a multi-stream dense network. In CVPR.
- Beyond brightening low-light images. International Journal of Computer Vision 129 (2021), 1013–1037.
- Kindling the darkness: A practical low-light image enhancer. In ACM Int. Conf. Multimedia. 1632–1640.
- Fourmer: An efficient global modeling paradigm for image restoration. In International Conference on Machine Learning. PMLR, 42589–42601.
- Adaptively learning low-high frequency information integration for pan-sharpening. In Proceedings of the 30th ACM International Conference on Multimedia. 3375–3384.
Sponsored by Paperpile, the PDF & BibTeX manager trusted by top AI labs.
Get 30 days freePaper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.