Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning Degradation-Independent Representations for Camera ISP Pipelines (2307.00761v3)

Published 3 Jul 2023 in cs.CV

Abstract: Image signal processing (ISP) pipeline plays a fundamental role in digital cameras, which converts raw Bayer sensor data to RGB images. However, ISP-generated images usually suffer from imperfections due to the compounded degradations that stem from sensor noises, demosaicing noises, compression artifacts, and possibly adverse effects of erroneous ISP hyperparameter settings such as ISO and gamma values. In a general sense, these ISP imperfections can be considered as degradations. The highly complex mechanisms of ISP degradations, some of which are even unknown, pose great challenges to the generalization capability of deep neural networks (DNN) for image restoration and to their adaptability to downstream tasks. To tackle the issues, we propose a novel DNN approach to learn degradation-independent representations (DiR) through the refinement of a self-supervised learned baseline representation. The proposed DiR learning technique has remarkable domain generalization capability and consequently, it outperforms state-of-the-art methods across various downstream tasks, including blind image restoration, object detection, and instance segmentation, as verified in our experiments.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
  2. Learning to see in the dark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3291–3300, 2018.
  3. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.
  4. Multitask aet with orthogonal tangent regularity for dark object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2553–2562, October 2021.
  5. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on image processing, 16(8):2080–2095, 2007.
  6. Learning invariant representation for unsupervised image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14483–14492, 2020.
  7. Learning robust representations via multi-view information bottleneck. In International Conference on Learning Representations, 2019.
  8. Toward convolutional blind denoising of real photographs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  9. A pipeline neural network for low-light image enhancement. IEEE Access, 7:13737–13744, 2019.
  10. Self-supervised face image restoration with a one-shot reference. International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024.
  11. Data acquisition and preparation for dual-reference deep learning of image super-resolution. IEEE Transactions on Image Processing, 31:4393–4404, 2022.
  12. Deep multi-modality soft-decoding of very low bit-rate face videos. Proceedings of the 28th ACM International Conference on Multimedia, Oct 2020.
  13. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pages 2961–2969, 2017.
  14. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.
  15. Variational interaction information maximization for cross-domain disentanglement. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 22479–22491. Curran Associates, Inc., 2020.
  16. Replacing mobile camera isp with a single deep learning model. arXiv preprint arXiv:2002.05509, 2020.
  17. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1125–1134, 2017.
  18. Learning attribute and class-specific representation duet for fine-grained fashion analysis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11050–11059, 2023.
  19. Fine-grained fashion representation learning by online deep clustering. In European Conference on Computer Vision, pages 19–35. Springer, 2022.
  20. Dnf: Decouple and feedback network for seeing in the dark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18135–18144, 2023.
  21. A new in-camera imaging model for color computer vision and its application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(12):2289–2302, 2012.
  22. The noise clinic: a blind image denoising algorithm. Image Processing On Line, 5:1–54, 2015. https://doi.org/10.5201/ipol.2015.125.
  23. Understanding and evaluating blind deconvolution algorithms. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1964–1971, 2009.
  24. Learning distortion invariant representation for image restoration from a causality perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1714–1724, 2023.
  25. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740–755. Springer, 2014.
  26. Unsupervised image denoising in real-world scenarios via self-collaboration parallel generative adversarial branches. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
  27. Functional neural networks for parametric image restoration problems. In Thirty-fifth Annual Conference on Neural Information Processing Systems (NeurIPS), volume 34, pages 6762–6775, 2021.
  28. And: Adversarial neural degradation for learning blind image super-resolution. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
  29. Learning a no-reference quality metric for single-image super-resolution, 2016.
  30. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision (ICCV), volume 2, pages 416–423, July 2001.
  31. Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters, 20(3):209–212, 2013.
  32. Hardware-in-the-loop end-to-end optimization of camera image processing pipelines. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  33. A holistic approach to cross-channel image noise modeling and its application to image denoising. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1683–1691, 2016.
  34. Automatic isp image quality tuning using nonlinear optimization. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 2471–2475. IEEE, 2018.
  35. f-gan: Training generative neural samplers using variational divergence minimization. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 271–279, 2016.
  36. Random sub-samples generation for self-supervised real image denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12150–12159, October 2023.
  37. Benchmarking denoising algorithms with real photographs, 2017.
  38. Attention-aware learning for hyperparameter prediction in image processing pipelines. In European Conference on Computer Vision, pages 271–287. Springer, 2022.
  39. Learning to exploit the sequence-specific prior knowledge for image processing pipelines optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22314–22323, 2023.
  40. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.
  41. Deepisp: Toward learning an end-to-end image processing pipeline. IEEE Transactions on Image Processing, 28(2):912–923, 2018.
  42. Blindly assess image quality in the wild guided by a self-adaptive hyper network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  43. The information bottleneck method, 2000.
  44. Hyperparameter optimization in black-box image processing using differentiable proxies. ACM Transactions on Graphics (TOG), 38(4), 7 2019.
  45. Representation learning with contrastive predictive coding. arXiv e-prints, pages arXiv–1807, 2018.
  46. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008.
  47. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pages 1905–1914, 2021.
  48. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17683–17693, June 2022.
  49. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34, 2021.
  50. Real-world noisy image denoising: A new benchmark, 2018.
  51. Rawgment: noise-accounted raw augmentation enables recognition in a wide variety of environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14007–14017, 2023.
  52. Reconfigisp: Reconfigurable camera image processing pipeline. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4248–4257, 2021.
  53. Multi-stage progressive image restoration. In CVPR, 2021.
  54. Infovae: Information maximizing variational autoencoders. arXiv preprint arXiv:1706.02262, 2017.
  55. Scene parsing through ade20k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5122–5130, 2017.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now