Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

PIP: Positional-encoding Image Prior (2211.14298v3)

Published 25 Nov 2022 in cs.CV and cs.AI

Abstract: In Deep Image Prior (DIP), a Convolutional Neural Network (CNN) is fitted to map a latent space to a degraded (e.g. noisy) image but in the process learns to reconstruct the clean image. This phenomenon is attributed to CNN's internal image-prior. We revisit the DIP framework, examining it from the perspective of a neural implicit representation. Motivated by this perspective, we replace the random or learned latent with Fourier-Features (Positional Encoding). We show that thanks to the Fourier features properties, we can replace the convolution layers with simple pixel-level MLPs. We name this scheme ``Positional Encoding Image Prior" (PIP) and exhibit that it performs very similarly to DIP on various image-reconstruction tasks with much less parameters required. Additionally, we demonstrate that PIP can be easily extended to videos, where 3D-DIP struggles and suffers from instability. Code and additional examples for all tasks, including videos, are available on the project page https://nimrodshabtay.github.io/PIP/

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. 2012.
  2. Ad-dmkde: Anomaly detection through density matrices and fourier features, 2022.
  3. The spectral bias of the deep image prior. CoRR, abs/1912.08905, 2019.
  4. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8628–8638, 2021.
  5. Nas-dip: Learning deep image prior with neural architecture search. In European Conference on Computer Vision, pages 442–459. Springer, 2020.
  6. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  7. Regularizing the Deep Image Prior with a Learned Denoiser for Linear Inverse Problems. In MMSP 2021 - IEEE 23rd International Workshop on Multimedia Siganl Processing, pages 1–6, Tampere, Finland, Oct. 2021. IEEE.
  8. Paul Fishwick. Clip inversion with dip. https://github.com/metaphorz/deep-image-prior-hqskipnet, 2021.
  9. ” double-dip”: Unsupervised image decomposition via coupled deep-image-priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11026–11035, 2019.
  10. Pet image reconstruction using deep image prior. IEEE Transactions on Medical Imaging, 38(7):1655–1665, 2019.
  11. Direct reconstruction of linear parametric images from dynamic pet using nonlocal deep image prior. IEEE Transactions on Medical Imaging, 41(3):680–689, 2022.
  12. Point2mesh: A self-prior for deformable meshes. ACM Trans. Graph., 39(4), 2020.
  13. PET image reconstruction incorporating deep image prior and a forward projection model. IEEE Transactions on Radiation and Plasma Medical Sciences, 6(8):841–846, nov 2022.
  14. Sape: Spatially-adaptive progressive encoding for neural optimization. Advances in Neural Information Processing Systems, 34:8820–8832, 2021.
  15. Dual prior learning for blind and blended image restoration. IEEE Transactions on Image Processing, 31:1042–1056, 2022.
  16. Zero-shot blind image denoising via implicit neural representations. arXiv preprint arXiv:2204.02405, 2022.
  17. Noise-resistant demosaicing with deep image prior network and random rgbw color filter array. Sensors, 22(5):1767, 2022.
  18. Blind video temporal consistency via deep video prior. Advances in Neural Information Processing Systems, 33:1083–1093, 2020.
  19. Titan: Bringing the deep image prior to implicit representations. arXiv preprint arXiv:2211.00219, 2022.
  20. Functional regularization for reinforcement learning via learned fourier features. Advances in Neural Information Processing Systems, 34:19046–19055, 2021.
  21. Learnable fourier features for multi-dimensional spatial positional encoding. Advances in Neural Information Processing Systems, 34:15816–15829, 2021.
  22. Bacon: Band-limited coordinate networks for multiscale scene representation. In CVPR, 2022.
  23. Omnimatte: Associating objects and their effects in video. In CVPR, 2021.
  24. Deepred: Deep image prior powered by red. ArXiv, abs/1903.10176, 2019.
  25. Fast training of convolutional networks through ffts. arXiv, 2013.
  26. B Mildenhall. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, 2020.
  27. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16190–16199, 2022.
  28. A robust volumetric transformer for accurate 3d tumor segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 162–172. Springer, 2022.
  29. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675, 2017.
  30. On the spectral bias of neural networks. In International Conference on Machine Learning, volume 97, pages 5301–5310, 2019.
  31. Beyond periodicity: towards a unifying framework for activations in coordinate-mlps. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, pages 142–158. Springer, 2022.
  32. Neural blind deconvolution using deep priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3341–3350, 2020.
  33. On measuring and controlling the spectral bias of the deep image prior. International Journal of Computer Vision, 2022.
  34. Implicit neural representations with periodic activation functions. In arXiv, 2020.
  35. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 2020.
  36. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9446–9454, 2018.
  37. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  38. Early stopping for deep image prior. Transactions on Machine Learning Research, 2023.
  39. Positional encoding as spatial inductive bias in gans, 2020.
  40. Dynamic pet image reconstruction using nonnegative matrix factorization incorporated with deep image prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  41. Time-dependent deep image prior for dynamic mri. IEEE Transactions on Medical Imaging, 2021.
  42. 3d-ssim for video quality assessment. In 2012 19th IEEE International Conference on Image Processing, pages 621–624, 2012.
  43. On single image scale-up using sparse-representations. In International conference on curves and surfaces, pages 711–730. Springer, 2010.
  44. An internal learning approach to video inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  45. Bp-dip: A backprojection based deep image prior. In 2020 28th European Signal Processing Conference (EUSIPCO), pages 675–679. IEEE, 2021.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now