Papers
Topics
Authors
Recent
2000 character limit reached

Investigating Self-Supervised Image Denoising with Denaturation (2405.01124v5)

Published 2 May 2024 in stat.ML, cs.CV, cs.LG, eess.IV, math.ST, and stat.TH

Abstract: Self-supervised learning for image denoising problems in the presence of denaturation for noisy data is a crucial approach in machine learning. However, theoretical understanding of the performance of the approach that uses denatured data is lacking. To provide better understanding of the approach, in this paper, we analyze a self-supervised denoising algorithm that uses denatured data in depth through theoretical analysis and numerical experiments. Through the theoretical analysis, we discuss that the algorithm finds desired solutions to the optimization problem with the population risk, while the guarantee for the empirical risk depends on the hardness of the denoising task in terms of denaturation levels. We also conduct several experiments to investigate the performance of an extended algorithm in practice. The results indicate that the algorithm training with denatured images works, and the empirical performance aligns with the theoretical results. These results suggest several insights for further improvement of self-supervised image denoising that uses denatured data in future directions.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8):2080–2095, 2007. doi: 10.1109/tip.2007.901238.
  2. Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 26(7):3142–3155, 2017. doi: 10.1109/tip.2017.2662206.
  3. Noise2Noise: Learning image restoration without clean data. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2965–2974. PMLR, 10–15 Jul 2018.
  4. Noise2Void - learning denoising from single noisy images. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2124–2132, 2019. doi: 10.1109/CVPR.2019.00223.
  5. Neighbor2Neighbor: Self-supervised denoising from single noisy images. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14776–14785, 2021. doi: 10.1109/CVPR46437.2021.01454.
  6. Self2Self with dropout: Learning self-supervised denoising from single image. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1887–1895, 2020. doi: 10.1109/CVPR42600.2020.00196.
  7. Zero-shot Noise2Noise: Efficient image denoising without any data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14018–14027, June 2023.
  8. Extending Stein's unbiased risk estimator to train deep denoisers with correlated pairs of noisy images. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  9. Cryo-EM: beyond the microscope. Current Opinion in Structural Biology, 46:71–78, October 2017.
  10. Dmitry Lyumkis. Challenges and opportunities in cryo-EM single-particle analysis. Journal of Biological Chemistry, 294(13):5181–5197, March 2019.
  11. Sjors H.W. Scheres. Relion: Implementation of a bayesian approach to cryo-em structure determination. Journal of Structural Biology, 180(3):519–530, 2012.
  12. cryoSPARC: Algorithms for rapid unsupervised cryo-EM structure determination. Nature Methods, 14(3):290–296, 2017.
  13. Structure of human chromatin-remodelling PBAF complex bound to a nucleosome. Nature, 605(7908):166–171, 2022.
  14. Deformed2Self: Self-supervised denoising for dynamic medical imaging. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2021, pages 25–35, Cham, 2021. Springer International Publishing. doi: 10.1007/978-3-030-87196-3˙3.
  15. Deformation-compensated learning for image reconstruction without ground truth. IEEE Transactions on Medical Imaging, 41(9):2371–2384, 2022. doi: 10.1109/TMI.2022.3163018.
  16. Joint demosaicking and denoising by fine-tuning of bursts of raw images. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2019. doi: 10.1109/iccv.2019.00896.
  17. Noise2Self: Blind denoising by self-supervision. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 524–533. PMLR, 09–15 Jun 2019.
  18. Noisier2Noise: Learning to denoise from unpaired noisy data. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12061–12069, 2020. doi: 10.1109/CVPR42600.2020.01208.
  19. Recorrupted-to-Recorrupted: Unsupervised deep learning for image denoising. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2043–2052, 2021. doi: 10.1109/CVPR46437.2021.00208.
  20. Blind2unblind: Self-supervised image denoising with visible blind spots. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2017–2026, 2022. doi: 10.1109/CVPR52688.2022.00207.
  21. AP-BSN: Self-supervised denoising for real-world images via asymmetric PD and blind-spot network. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17704–17713, 2022. doi: 10.1109/CVPR52688.2022.01720.
  22. Patch-craft self-supervised training for correlated image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5795–5804, June 2023.
  23. Noise2Info: Noisy image to information of noise for self-supervised image denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 16034–16043, October 2023.
  24. Random sub-samples generation for self-supervised real image denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12150–12159, October 2023.
  25. Topaz-Denoise: general deep denoising models for cryoEM and cryoET. Nature Communications, 11(1), 2020. doi: 10.1038/s41467-020-18952-1.
  26. Multi-view self-supervised disentanglement for general image denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12281–12291, October 2023.
  27. Deep image prior. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9446–9454, 2018. doi: 10.1109/CVPR.2018.00984.
  28. A fast blind zero-shot denoiser. Nature Machine Intelligence, 4:953–963, 2022. doi: 10.1038/s42256-022-00547-8.
  29. Poisson2Sparse: Self-supervised Poisson denoising from a single image. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2022, pages 557–567, Cham, 2022. Springer Nature Switzerland. doi: 10.1007/978-3-031-16452-1˙53.
  30. Burst denoising with kernel prediction networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2502–2510, 2018. doi: 10.1109/CVPR.2018.00265.
  31. Deep burst denoising. In Computer Vision – ECCV 2018, pages 560–577. Springer International Publishing, 2018. doi: 10.1007/978-3-030-01267-0˙33.
  32. Burst image restoration and enhancement. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5749–5758, 2022. doi: 10.1109/CVPR52688.2022.00567.
  33. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc., 2020.
  34. DDM2: Self-supervised diffusion MRI denoising with generative diffusion models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=0vqjc50HfcC.
  35. The elements of statistical learning: data mining, inference, and prediction. Springer, second edition, 2009.
  36. Ulrike von Luxburg and Olivier Bousquet. Distance-based classification with Lipschitz functions. Journal of Machine Learning Research, 5(Jun):669–695, 2004.
  37. Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463–482, 2002.
  38. Foundations of machine learning. MIT press, second edition, 2018.
  39. Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018. doi: 10.1017/9781108231596.
  40. Noisy-as-Clean: Learning self-supervised denoising from corrupted image. IEEE Transactions on Image Processing, 29:9316–9329, 2020. doi: 10.1109/TIP.2020.3026622.
  41. Movies of ice-embedded particles enhance resolution in electron cryo-microscopy. Structure, 20(11):1823–1828, November 2012.
  42. MotionCor2: anisotropic correction of beam-induced motion for improved cryo-electron microscopy. Nature Methods, 14(4):331–332, April 2017.
  43. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958, 2014.
  44. Image quality metrics: Psnr vs. ssim. In 2010 20th International Conference on Pattern Recognition, pages 2366–2369, 2010. doi: 10.1109/ICPR.2010.579.
  45. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004. doi: 10.1109/TIP.2003.819861.
  46. scikit-image: image processing in python. PeerJ, 2:e453, 2014.
  47. Self-supervised Poisson-Gaussian denoising. In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2021. doi: 10.1109/wacv48630.2021.00218.
  48. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: Is the problem solved? IEEE Transactions on Medical Imaging, 37(11):2514–2525, 2018. doi: 10.1109/TMI.2018.2837502.
  49. “bm3d”. Python Package Index, 2022. URL https://pypi.org/project/bm3d/ [Last accessed on 5th February, 2024].
  50. Collaborative filtering of correlated noise: Exact transform-domain variance for improved shrinkage and patch matching. IEEE Transactions on Image Processing, 29:8339–8354, 2020. doi: 10.1109/TIP.2020.3014721.
  51. “jason-lequyer/Noise2Fast”. GitHub, 2023. URL: https://github.com/jason-lequyer/Noise2Fast [Last accessed on 5th February, 2024].
  52. “daviddmc/Deform2Self”. GitHub, 2022. URL: https://github.com/daviddmc/Deform2Self [Last accessed on 5th February, 2024].
  53. EMPIAR: the electron microscopy public image archive. Nucleic Acids Research, 51(D1):D1503–D1511, January 2023.
  54. Fiji: an open-source platform for biological-image analysis. Nature Methods, 9(7):676–682, June 2012.
  55. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  56. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pages 234–241, Cham, 2015. Springer International Publishing. doi: 10.1007/978-3-319-24574-4˙28.
  57. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Slide Deck Streamline Icon: https://streamlinehq.com

Whiteboard

Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Sign up for free to view the 7 tweets with 5 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free