Enhancing RAW-to-sRGB with Decoupled Style Structure in Fourier Domain (2401.02161v1)
Abstract: RAW to sRGB mapping, which aims to convert RAW images from smartphones into RGB form equivalent to that of Digital Single-Lens Reflex (DSLR) cameras, has become an important area of research. However, current methods often ignore the difference between cell phone RAW images and DSLR camera RGB images, a difference that goes beyond the color matrix and extends to spatial structure due to resolution variations. Recent methods directly rebuild color mapping and spatial structure via shared deep representation, limiting optimal performance. Inspired by Image Signal Processing (ISP) pipeline, which distinguishes image restoration and enhancement, we present a novel Neural ISP framework, named FourierISP. This approach breaks the image down into style and structure within the frequency domain, allowing for independent optimization. FourierISP is comprised of three subnetworks: Phase Enhance Subnet for structural refinement, Amplitude Refine Subnet for color learning, and Color Adaptation Subnet for blending them in a smooth manner. This approach sharpens both color and structure, and extensive evaluations across varied datasets confirm that our approach realizes state-of-the-art results. Code will be available at ~\url{https://github.com/alexhe101/FourierISP}.
- Histogan: Controlling colors of gan-generated and real images via color histograms. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 7941–7950.
- LW-ISP: A Lightweight Model with ISP and Deep Learning. In 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022, 148. BMVA Press.
- Hinet: Half instance normalization network for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 182–192.
- Nbnet: Noise basis learning for image denoising with subspace projection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 4896–4906.
- Fast fourier convolution. Advances in Neural Information Processing Systems, 33: 4479–4488.
- Awnet: Attentive wavelet network for image isp. In Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, 185–201. Springer.
- Joint multi-scale tone mapping and denoising for HDR image enhancement. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 729–738.
- Learned smartphone isp on mobile npus with deep learning, mobile ai 2021 challenge: Report. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2503–2514.
- Aim 2020 challenge on learned image signal processing pipeline. In Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, 152–170. Springer.
- Replacing mobile camera isp with a single deep learning model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 536–537.
- Perceptual losses for real-time style transfer and super-resolution. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, 694–711. Springer.
- Embedding Fourier for Ultra-High-Definition Low-Light Image Enhancement. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
- Cameranet: A two-stage framework for effective camera isp learning. IEEE Transactions on Image Processing, 30: 2248–2262.
- Joint demosaicing and denoising with self guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2240–2249.
- Multi-level wavelet-CNN for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 773–782.
- Color image processing pipeline. IEEE Signal Processing Magazine, 22(1): 34–43.
- Deepisp: Toward learning an end-to-end image processing pipeline. IEEE Transactions on Image Processing, 28(2): 912–923.
- Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition, 8934–8943.
- Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2149–2159.
- Recovering realistic texture in image super-resolution by deep spatial feature transform. In Proceedings of the IEEE conference on computer vision and pattern recognition, 606–615.
- Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, 1398–1402. Ieee.
- Learning to adapt to light. International Journal of Computer Vision, 131(4): 1022–1041.
- The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, 586–595.
- Learning raw-to-srgb mappings with inaccurately aligned supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4348–4358.
- Image pipeline tuning for digital cameras. In 2007 IEEE International Symposium on Consumer Electronics, 1–4. IEEE.
- Fourmer: An Efficient Global Modeling Paradigm for Image Restoration. In International Conference on Machine Learning, 42589–42601. PMLR.
- Spatial-frequency domain information integration for pan-sharpening. In European Conference on Computer Vision, 274–291. Springer.