Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
133 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SeNM-VAE: Semi-Supervised Noise Modeling with Hierarchical Variational Autoencoder (2403.17502v1)

Published 26 Mar 2024 in cs.CV

Abstract: The data bottleneck has emerged as a fundamental challenge in learning based image restoration methods. Researchers have attempted to generate synthesized training data using paired or unpaired samples to address this challenge. This study proposes SeNM-VAE, a semi-supervised noise modeling method that leverages both paired and unpaired datasets to generate realistic degraded data. Our approach is based on modeling the conditional distribution of degraded and clean images with a specially designed graphical model. Under the variational inference framework, we develop an objective function for handling both paired and unpaired data. We employ our method to generate paired training samples for real-world image denoising and super-resolution tasks. Our approach excels in the quality of synthetic degraded images compared to other unpaired and paired noise modeling methods. Furthermore, our approach demonstrates remarkable performance in downstream image restoration tasks, even with limited paired data. With more paired data, our method achieves the best performance on the SIDD dataset.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (72)
  1. A high-quality denoising dataset for smartphone cameras. In CVPR, pages 1692–1700, 2018.
  2. Noise flow: Noise modeling with conditional normalizing flows. In CVPR, pages 3165–3173, 2019.
  3. Noise2self: Blind denoising by self-supervision. In ICML, pages 524–533. PMLR, 2019.
  4. Generating sentences from a continuous space. arXiv:1511.06349, 2015.
  5. Natural image noise dataset. In CVPRW, pages 0–0, 2019.
  6. A non-local algorithm for image denoising. In CVPR, pages 60–65. Ieee, 2005.
  7. To learn image super-resolution, use a gan to learn how to do image degradation first. In ECCV, pages 185–200, 2018.
  8. Simple baselines for image restoration. In ECCV, pages 17–33. Springer, 2022.
  9. Score priors guided deep variational inference for unsupervised real-world single image denoising. In ICCV, pages 12937–12948, 2023.
  10. Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. arXiv:2011.10650, 2020.
  11. Semi-supervised image deraining using knowledge distillation. IEEE Trans Circuits Syst Video Technol, 32(12):8327–8341, 2022.
  12. Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space. In ICIP, pages I–313. IEEE, 2007.
  13. Learning a deep convolutional network for image super-resolution. In ECCV, pages 184–199. Springer, 2014.
  14. Accelerating the super-resolution convolutional neural network. In ECCV, pages 391–407. Springer, 2016.
  15. Nonlocal image restoration with bilateral variance estimation: a low-rank approach. IEEE Trans Image Process, 22(2):700–711, 2012.
  16. Frequency separation for real-world super-resolution. In ICCVW, pages 3599–3608. IEEE, 2019.
  17. srgb real noise synthesizing with neighboring correlation-aware noise model. In CVPR, pages 1683–1691, 2023.
  18. A comprehensive tool for modeling cmos image-sensor-noise performance. IEEE Trans Electron Devices, 54(6):1321–1329, 2007.
  19. Weighted nuclear norm minimization with application to image denoising. In CVPR, pages 2862–2869, 2014.
  20. Toward convolutional blind denoising of real photographs. In CVPR, pages 1712–1722, 2019.
  21. Noise-optimal capture for high dynamic range photography. In CVPR, pages 553–560. IEEE, 2010.
  22. Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell., 33(12):2341–2353, 2010.
  23. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NIPS, 30, 2017.
  24. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2016.
  25. C2n: Practical generative noise modeling for real-world denoising. In ICCV, pages 2350–2359, 2021.
  26. Real-world super-resolution via kernel estimation and noise injection. In CVPRW, pages 466–467, 2020.
  27. Lightweight semi-supervised network for single image rain removal. Pattern Recognit, page 109277, 2023.
  28. Denoising diffusion restoration models. NIPS, 35:23593–23606, 2022.
  29. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.
  30. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
  31. Modeling srgb camera noise with normalizing flows. In CVPR, pages 17463–17471, 2022.
  32. Noise2void-learning denoising from single noisy images. In CVPR, pages 2129–2137, 2019.
  33. Autoencoding beyond pixels using a learned similarity metric. In ICML, pages 1558–1566. PMLR, 2016.
  34. Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network. In CVPR, pages 17725–17734, 2022.
  35. Semi-supervised image dehazing. IEEE Trans Image Process, 29:2766–2779, 2019.
  36. Unsupervised image denoising in real-world scenarios via self-collaboration parallel generative adversarial branches. In ICCV, pages 12642–12652, 2023.
  37. Unsupervised learning for real-world super-resolution. In ICCVW, pages 3408–3416. IEEE, 2019a.
  38. Aim 2019 challenge on real-world image super-resolution: Methods and results. In ICCVW, pages 3575–3583. IEEE, 2019b.
  39. Ntire 2020 challenge on real-world image super-resolution: Methods and results. In CVPRW, pages 494–495, 2020a.
  40. Srflow: Learning the super-resolution space with normalizing flow. In ECCV, pages 715–732. Springer, 2020b.
  41. Semi-supervised learning for low-light image restoration through quality assisted pseudo-labeling. In WACV, pages 4105–4114, 2023.
  42. Dancing under the stars: video denoising in starlight. In CVPR, pages 16241–16251, 2022.
  43. Cvf-sid: Cyclic multi-variate function for self-supervised image denoising by disentangling noise from image. In CVPR, pages 17583–17591, 2022.
  44. Random sub-samples generation for self-supervised real image denoising. In ICCV, pages 12150–12159, 2023.
  45. Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell., 12(7):629–639, 1990.
  46. Benchmarking denoising algorithms with real photographs. In CVPR, pages 1586–1595, 2017.
  47. Adaptive consistency prior based deep network for image denoising. In CVPR, pages 8596–8606, 2021.
  48. Nonlinear total variation based noise removal algorithms. Physica D, 60(1-4):259–268, 1992.
  49. Variational mixture-of-experts autoencoders for multi-modal deep generative models. NIPS, 32, 2019.
  50. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
  51. Solving inverse problems in medical imaging with score-based generative models. arXiv:2111.08005, 2021.
  52. Maxim: Multi-axis mlp for image processing. In CVPR, pages 5769–5780, 2022.
  53. Nvae: A deep hierarchical variational autoencoder. NIPS, 33:19667–19679, 2020.
  54. Esrgan: Enhanced super-resolution generative adversarial networks. In ECCVW, pages 0–0, 2018.
  55. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process, 13(4):600–612, 2004.
  56. Uformer: A general u-shaped transformer for image restoration. In CVPR, pages 17683–17693, 2022.
  57. Single image reflection removal exploiting misaligned training data and network enhancements. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8178–8187, 2019a.
  58. Semi-supervised transfer learning for image rain removal. In CVPR, pages 3877–3886, 2019b.
  59. Unsupervised real-world image super resolution via domain-distance aware training. In CVPR, pages 13385–13394, 2021a.
  60. Semi-deraingan: A new semi-supervised single image deraining. In ICME, pages 1–6. IEEE, 2021b.
  61. Deflow: Learning complex image degradations from unpaired data with conditional flows. In CVPR, pages 94–103, 2021.
  62. Variational denoising network: Toward blind noise modeling and removal. NIPS, 32, 2019.
  63. Dual adversarial network: Toward real-world noise removal and noise generation. In ECCV, pages 41–58. Springer, 2020.
  64. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, pages 5728–5739, 2022.
  65. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Trans Image Process, 26(7):3142–3155, 2017.
  66. Ffdnet: Toward a fast and flexible solution for cnn-based image denoising. IEEE Trans Image Process, 27(9):4608–4622, 2018a.
  67. Plug-and-play image restoration with deep denoiser prior. IEEE Trans. Pattern Anal. Mach. Intell., 44(10):6360–6376, 2021.
  68. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586–595, 2018b.
  69. Residual dense network for image super-resolution. In CVPR, pages 2472–2481, 2018c.
  70. An unsupervised deep learning approach for real-world image denoising. In ICLR, 2020.
  71. Learn from unpaired data for image restoration: A variational bayes approach. IEEE Trans. Pattern Anal. Mach. Intell., 2022.
  72. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, pages 2223–2232, 2017.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now