BSRAW: Improving Blind RAW Image Super-Resolution (2312.15487v1)
Abstract: In smartphones and compact cameras, the Image Signal Processor (ISP) transforms the RAW sensor image into a human-readable sRGB image. Most popular super-resolution methods depart from a sRGB image and upscale it further, improving its quality. However, modeling the degradations in the sRGB domain is complicated because of the non-linear ISP transformations. Despite this known issue, only a few methods work directly with RAW images and tackle real-world sensor degradations. We tackle blind image super-resolution in the RAW domain. We design a realistic degradation pipeline tailored specifically for training models with raw sensor data. Our approach considers sensor noise, defocus, exposure, and other common issues. Our BSRAW models trained with our pipeline can upscale real-scene RAW images and improve their quality. As part of this effort, we also present a new DSLM dataset and benchmark for this task.
- A high-quality denoising dataset for smartphone cameras. In CVPR, pages 1692–1700, 2018.
- Ntire 2019 challenge on real image denoising: Methods and results. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), June 2019.
- Improving single-image defocus deblurring: How dual-pixel images help through multi-task learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1231–1239, 2022.
- Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 126–135, 2017.
- Deep burst super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9209–9218, 2021.
- Unprocessing images for learned raw denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11036–11045, 2019.
- To learn image super-resolution, use a gan to learn how to do image degradation first. In Proceedings of the European conference on computer vision (ECCV), pages 185–200, 2018.
- Learning photographic global tonal adjustment with a database of input / output image pairs. In CVPR, 2011.
- Toward real-world single image super-resolution: A new benchmark and a new model. In IEEE Conference on International Conference on Computer Vision, pages 3086–3095, 2019.
- Practical real video denoising with realistic degradation model. arXiv preprint arXiv:2208.11803, 2022.
- Low-light image restoration with short-and long-exposure raw pairs. IEEE Transactions on Multimedia, 24:702–714, 2021.
- Learning to see in the dark. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3291–3300, 2018.
- Simple baselines for image restoration. arXiv preprint arXiv:2204.04676, 2022.
- Swin2SR: Swinv2 transformer for compressed image super-resolution and restoration. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2022.
- Model-based image signal processors via learnable dictionaries. Proceedings of the AAAI Conference on Artificial Intelligence, 36(1):481–489, Jun. 2022.
- Perceptual image enhancement for smartphone real-time applications. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1848–1858, 2023.
- Mobile computational photography: A tour. arXiv preprint arXiv:2102.09000, 2021.
- Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision, pages 184–199, 2014.
- Accurate blur models vs. image priors in single image super-resolution. In IEEE Conference on International Conference on Computer Vision, pages 2832–2839, 2013.
- Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Transactions on Image Processing, 6(12):1646–1658, 1997.
- Samuel W Hasinoff. Photon, poisson noise. Computer Vision, A Reference Guide, 4, 2014.
- Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions on Graphics (Proc. SIGGRAPH Asia), 35(6), 2016.
- Flexisp: A flexible camera image processing framework. ACM Transactions on Graphics (ToG), 33(6):1–13, 2014.
- A multi-hypothesis approach to color constancy. In VPR, 2020.
- Convolutional deblurring for natural imaging. IEEE Transactions on Image Processing, 29:250–264, 2019.
- Towards low light enhancement with raw images. IEEE Transactions on Image Processing, 31:1391–1405, 2022.
- Learned smartphone isp on mobile npus with deep learning, mobile ai 2021 challenge: Report. In CVPR Workshops, pages 2503–2514, 2021.
- Replacing mobile camera isp with a single deep learning model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 536–537, 2020.
- Real-world super-resolution via kernel estimation and noise injection. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 466–467, 2020.
- A software platform for manipulating the camera imaging pipeline. In ECCV, pages 429–444, 2016.
- Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- Understanding and evaluating blind deconvolution algorithms. In 2009 IEEE conference on computer vision and pattern recognition, pages 1964–1971. IEEE, 2009.
- Raw image deblurring. IEEE Transactions on Multimedia, 2020.
- Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1833–1844, 2021.
- Cameranet: A two-stage framework for effective camera isp learning. IEEE Transactions on Image Processing, 30:2248–2262, 2021.
- Learning raw image denoising with bayer pattern unification and bayer preserving augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
- Deep-flexisp: A three-stage framework for night photography rendering. In CVPR, pages 1211–1220, 2022.
- Unsupervised learning for real-world super-resolution. In IEEE Conference on International Conference on Computer Vision Workshops, pages 3408–3416. IEEE, 2019.
- Elmformer: Efficient raw image restoration with a locally multiplicative transformer. In Proceedings of the 30th ACM International Conference on Multimedia, pages 5842–5852, 2022.
- Burst denoising with kernel prediction networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2502–2510, 2018.
- Blind image deblurring using dark channel prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1628–1636, 2016.
- Benchmarking denoising algorithms with real photographs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1586–1595, 2017.
- Day-to-night image synthesis for training nighttime neural isps. In CVPR, pages 10769–10778, 2022.
- Rethinking the pipeline of demosaicing, denoising and super-resolution. arXiv preprint arXiv:1905.02538, 2019.
- Neural blind deconvolution using deep priors. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3341–3350, 2020.
- A machine learning approach for non-blind image deconvolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1067–1074, 2013.
- Deepisp: Toward learning an end-to-end image processing pipeline. IEEE Transactions on Image Processing, 28(2):912–923, 2018.
- A+: Adjusted anchored neighborhood regression for fast super-resolution. In ACCV, 2014.
- Attention is all you need. arXiv preprint arXiv:1706.03762, 2017.
- Real-esrgan: Training real-world blind super-resolution with pure synthetic data. arXiv preprint arXiv:2107.10833, 2021.
- Esrgan: Enhanced super-resolution generative adversarial networks. In European Conference on Computer Vision Workshops, pages 701–710, 2018.
- Practical deep raw image denoising on mobile devices. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI, pages 1–16. Springer, 2020.
- Uformer: A general u-shaped transformer for image restoration. arXiv preprint arXiv:2106.03106, 2021.
- Invertible image signal processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6287–6296, 2021.
- Two-phase kernel estimation for robust motion deblurring. In Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part I 11, pages 157–170. Springer, 2010.
- Towards real scene super-resolution with raw images. In CVPR, 2019.
- Exploiting raw images for real-scene super-resolution. IEEE transactions on pattern analysis and machine intelligence, 44(4):1905–1921, 2020.
- Motion blur kernel estimation via deep learning. IEEE Transactions on Image Processing, 27(1):194–205, 2017.
- Image super-resolution via sparse representation. IEEE transactions on image processing, 19(11):2861–2873, 2010.
- Rawgment: noise-accounted raw augmentation enables recognition in a wide variety of environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14007–14017, 2023.
- Image deblurring with blurred/noisy image pairs. In ACM SIGGRAPH 2007 papers, pages 1–es. 2007.
- Supervised raw video denoising with a benchmark dataset on dynamic scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2301–2310, 2020.
- Real-rawvsr: Real-world raw video super-resolution with a benchmark dataset. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VI, pages 608–624. Springer, 2022.
- Towards real-time 4k image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1522–1532, 2023.
- Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5728–5739, 2022.
- Cycleisp: Real image restoration via improved data synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2696–2705, 2020.
- Deep unfolding network for image super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3217–3226, 2020.
- Practical blind denoising via swin-conv-unet and data synthesis. arXiv preprint arXiv:2203.13278, 2022.
- Designing a practical degradation model for deep blind image super-resolution. In IEEE Conference on International Conference on Computer Vision, 2021.
- Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26(7):3142–3155, 2017.
- Zoom to learn, learn to zoom. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3762–3770, 2019.
- Low light raw image enhancement using paired fast fourier convolution and transformer. In 2022 IEEE International Conference on Visual Communications and Image Processing (VCIP), pages 1–5. IEEE, 2022.
- Rethinking noise synthesis and modeling in raw denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4593–4601, 2021.
- Residual dense network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2472–2481, 2018.
- Deconvolving psfs for a better motion deblurring using multiple images. In Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 636–647. Springer, 2012.