Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Early Stopping for Deep Image Prior (2112.06074v4)

Published 11 Dec 2021 in cs.CV, cs.LG, eess.IV, and eess.SP

Abstract: Deep image prior (DIP) and its variants have showed remarkable potential for solving inverse problems in computer vision, without any extra training data. Practical DIP models are often substantially overparameterized. During the fitting process, these models learn mostly the desired visual content first, and then pick up the potential modeling and observational noise, i.e., overfitting. Thus, the practicality of DIP often depends critically on good early stopping (ES) that captures the transition period. In this regard, the majority of DIP works for vision tasks only demonstrates the potential of the models -- reporting the peak performance against the ground truth, but provides no clue about how to operationally obtain near-peak performance without access to the groundtruth. In this paper, we set to break this practicality barrier of DIP, and propose an efficient ES strategy, which consistently detects near-peak performance across several vision tasks and DIP variants. Based on a simple measure of dispersion of consecutive DIP reconstructions, our ES method not only outpaces the existing ones -- which only work in very narrow domains, but also remains effective when combined with a number of methods that try to mitigate the overfitting. The code is available at https://github.com/sun-umn/Early_Stopping_for_DIP.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. NTIRE 2020 challenge on real image denoising: Dataset, methods and results. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pp.  2077–2088. Computer Vision Foundation / IEEE, 2020. doi: 10.1109/CVPRW50498.2020.00256.
  2. A survey of methods for time series change point detection. Knowl. Inf. Syst., 51(2):339–367, 2017. doi: 10.1007/s10115-016-0987-z.
  3. Blind image deconvolution using deep generative priors. IEEE Trans. Computational Imaging, 6:1493–1506, 2020. doi: 10.1109/TCI.2020.3032671.
  4. Computed tomography reconstruction using deep image prior and learned reconstruction methods. CoRR, abs/2003.04989, 2020.
  5. A fast approach for no-reference image sharpness assessment based on maximum local variation. IEEE Signal Process. Lett., 21(6):751–755, 2014. doi: 10.1109/LSP.2014.2314487.
  6. Combining weighted total variation and deep image prior for natural and medical image restoration via ADMM. In 2021 21st International Conference on Computational Science and Its Applications (ICCSA), Cagliari, Italy, September 13-16, 2021 - Workshops, pp.  39–46. IEEE, 2021. doi: 10.1109/ICCSA54496.2021.00016.
  7. A bayesian perspective on the deep image prior. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp.  5443–5451. Computer Vision Foundation / IEEE, 2019. doi: 10.1109/CVPR.2019.00559.
  8. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=OnD9zGAGT0k.
  9. The blur effect: perception and estimation with a new no-reference perceptual blur metric. In Bernice E. Rogowitz, Thrasyvoulos N. Pappas, and Scott J. Daly (eds.), Human Vision and Electronic Imaging XII, San Jose, CA, USA, January 29 - February 1, 2007, volume 6492 of SPIE Proceedings, pp.  64920I. SPIE, 2007. doi: 10.1117/12.702790.
  10. Image restoration by sparse 3d transform-domain collaborative filtering. In Jaakko Astola, Karen O. Egiazarian, and Edward R. Dougherty (eds.), Image Processing: Algorithms and Systems VI, San Jose, California, USA, January 28-29, 2008, volume 6812 of SPIE Proceedings, pp.  681207. SPIE, 2008. doi: 10.1117/12.766355.
  11. Accelerated MRI with un-trained neural networks. IEEE Trans. Computational Imaging, 7:724–733, 2021. doi: 10.1109/TCI.2021.3097596.
  12. Rank overspecified robust matrix recovery: Subgradient method and exact recovery. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp.  26767–26778, 2021.
  13. A validation approach to over-parameterized matrix and image recovery. CoRR, abs/2209.10675, 2022. doi: 10.48550/arXiv.2209.10675.
  14. NIMA: neural image assessment. IEEE Trans. Image Process., 27(8):3998–4011, 2018. doi: 10.1109/TIP.2018.2831899.
  15. "double-dip": Unsupervised image decomposition via coupled deep-image-priors. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp.  11026–11035. Computer Vision Foundation / IEEE, 2019. doi: 10.1109/CVPR.2019.01128.
  16. Neural networks and the bias/variance dilemma. Neural Comput., 4(1):1–58, 1992. doi: 10.1162/neco.1992.4.1.1.
  17. Direct reconstruction of linear parametric images from dynamic PET using nonlocal deep image prior. IEEE Trans. Medical Imaging, 41(3):680–689, 2022. doi: 10.1109/TMI.2021.3120913.
  18. Phase retrieval under a generative prior. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp.  9154–9164, 2018.
  19. Direct PET image reconstruction incorporating deep image prior and a forward projection model. CoRR, abs/2109.00768, 2021.
  20. Deep decoder: Concise image representations from untrained non-convolutional networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  21. Compressive sensing with un-trained neural networks: Gradient descent finds a smooth approximation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp.  4149–4158. PMLR, 2020a.
  22. Denoising and regularization via exploiting the structural bias of convolutional generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020b.
  23. Benchmarking neural network robustness to common corruptions and perturbations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  24. Neural tangent kernel: Convergence and generalization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp.  8580–8589, 2018.
  25. Computer vision for autonomous vehicles: Problems, datasets and state of the art. Found. Trends Comput. Graph. Vis., 12(1-3):1–308, 2020. doi: 10.1561/0600000079.
  26. Rethinking deep image prior for denoising. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp.  5067–5076. IEEE, 2021. doi: 10.1109/ICCV48922.2021.00504.
  27. Blind deconvolution using a normalized sparsity measure. In The 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011, pp.  233–240. IEEE Computer Society, 2011. doi: 10.1109/CVPR.2011.5995521.
  28. Understanding blind deconvolution algorithms. IEEE Trans. Pattern Anal. Mach. Intell., 33(12):2354–2367, 2011. doi: 10.1109/TPAMI.2011.148.
  29. Self-validation: Early stopping for single-instance deep generative priors. In 32nd British Machine Vision Conference 2021, BMVC 2021, Online, November 22-25, 2021, pp.  108. BMVA Press, 2021.
  30. Joint demosaicing and denoising with double deep image priors. CoRR, abs/2309.09426, 2023a. doi: 10.48550/arXiv.2309.09426. URL https://doi.org/10.48550/arXiv.2309.09426.
  31. Robust Autoencoders for Collective Corruption Removal. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.  1–5, June 2023b. doi: 10.1109/ICASSP49357.2023.10095099. URL https://ieeexplore.ieee.org/abstract/document/10095099. ISSN: 2379-190X.
  32. Deep Random Projector: Accelerated Deep Image Prior. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  18176–18185, Vancouver, BC, Canada, June 2023c. IEEE. ISBN 9798350301298. doi: 10.1109/CVPR52729.2023.01743. URL https://ieeexplore.ieee.org/document/10205276/.
  33. Random Projector: Efficient Deep Image Prior. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.  1–5, Rhodes Island, Greece, June 2023d. IEEE. ISBN 978-1-72816-327-7. doi: 10.1109/ICASSP49357.2023.10097088. URL https://ieeexplore.ieee.org/document/10097088/.
  34. Image restoration using total variation regularized deep image prior. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019, pp.  7715–7719. IEEE, 2019. doi: 10.1109/ICASSP.2019.8682856.
  35. Unsupervised image fusion using deep image priors. CoRR, abs/2110.09490, 2021.
  36. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the Eighth International Conference On Computer Vision (ICCV-01), Vancouver, British Columbia, Canada, July 7-14, 2001 - Volume 2, pp.  416–425. IEEE Computer Society, 2001. doi: 10.1109/ICCV.2001.937655.
  37. Deepred: Deep image prior powered by red. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp.  0–0, 2019.
  38. Unsupervised learning with stein’s unbiased risk estimator. CoRR, abs/1805.10531, 2018.
  39. No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process., 21(12):4695–4708, 2012. doi: 10.1109/TIP.2012.2214050.
  40. Making a "completely blind" image quality analyzer. IEEE Signal Process. Lett., 20(3):209–212, 2013. doi: 10.1109/LSP.2012.2227726.
  41. Deep learning techniques for inverse problems in imaging. IEEE J. Sel. Areas Inf. Theory, 1(1):39–56, 2020. doi: 10.1109/jsait.2020.2991563.
  42. Untrained neural network priors for inverse imaging problems: A survey. TechRxiv, mar 2021. doi: 10.36227/techrxiv.14208215.v1.
  43. Neural blind deconvolution using deep priors. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp.  3338–3347. Computer Vision Foundation / IEEE, 2020. doi: 10.1109/CVPR42600.2020.00340.
  44. On measuring and controlling the spectral bias of the deep image prior. Int. J. Comput. Vis., 130(4):885–908, 2022. doi: 10.1007/s11263-021-01572-7.
  45. Implicit neural representations with periodic activation functions. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  46. Zhaodong Sun. Solving inverse problems with hybrid deep image priors: the challenge of preventing overfitting. CoRR, abs/2011.01748, 2020.
  47. Richard Szeliski. Computer Vision - Algorithms and Applications, Second Edition. Texts in Computer Science. Springer, 2022. ISBN 978-3-030-34371-2. doi: 10.1007/978-3-030-34372-9.
  48. Fourier features let networks learn high frequency functions in low dimensional domains. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  49. Phase retrieval using single-instance deep generative prior. CoRR, abs/2106.04812, 2021.
  50. Explore image deblurring via encoded blur kernel space. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp.  11956–11965. Computer Vision Foundation / IEEE, 2021. doi: 10.1109/CVPR46437.2021.01178.
  51. Deep image prior. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp.  9446–9454. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVPR.2018.00984.
  52. Implicit regularization for optimal sparse recovery. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp.  2968–2979, 2019.
  53. Compressed sensing with deep image prior and learned regularization. CoRR, abs/1806.06438, 2018.
  54. Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model, December 2022. URL http://arxiv.org/abs/2212.00490. arXiv:2212.00490 [cs].
  55. Image deconvolution with deep image and kernel priors. In 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019, pp.  980–989. IEEE, 2019.
  56. Deep geometric prior for surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp.  10130–10139. Computer Vision Foundation / IEEE, 2019. doi: 10.1109/CVPR.2019.01037.
  57. Real-world noisy image denoising: A new benchmark. CoRR, abs/1804.02603, 2018.
  58. Zero-shot physics-guided deep learning for subject-specific MRI reconstruction. In NeurIPS 2021 Workshop on Deep Learning and Inverse Problems, 2021.
  59. Rethinking bias-variance trade-off for generalization of neural networks. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp.  10767–10777. PMLR, 2020.
  60. Time-dependent deep image prior for dynamic MRI. IEEE Trans. Medical Imaging, 40(12):3337–3348, 2021. doi: 10.1109/TMI.2021.3084288.
  61. Robust recovery via implicit bias of discrepant learning rates for double over-parameterization. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  62. fastmri: An open dataset and benchmarks for accelerated MRI. CoRR, abs/1811.08839, 2018.
  63. On Single Image Scale-Up Using Sparse-Representations. In Jean-Daniel Boissonnat, Patrick Chenin, Albert Cohen, Christian Gout, Tom Lyche, Marie-Laurence Mazure, and Larry Schumaker (eds.), Curves and Surfaces, volume 6920, pp.  711–730. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-642-27412-1 978-3-642-27413-8. doi: 10.1007/978-3-642-27413-8_47. URL http://link.springer.com/10.1007/978-3-642-27413-8_47. Series Title: Lecture Notes in Computer Science.
  64. Denoising Diffusion Models for Plug-and-Play Image Restoration, May 2023. URL http://arxiv.org/abs/2305.08995. arXiv:2305.08995 [cs, eess].
  65. Blind image deblurring with unknown kernel size and substantial noise. CoRR, abs/2208.09483, 2022a. doi: 10.48550/arXiv.2208.09483.
  66. Practical phase retrieval using double deep image priors. arXiv preprint arXiv:2211.00799, 2022b.
Citations (52)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - sun-umn/Early_Stopping_for_DIP: Official Implementation of Early Stopping for Deep Image Prior (TMLR) (24 stars)