Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Toward Accurate and Temporally Consistent Video Restoration from Raw Data (2312.16247v1)

Published 25 Dec 2023 in cs.CV and eess.IV

Abstract: Denoising and demosaicking are two fundamental steps in reconstructing a clean full-color video from raw data, while performing video denoising and demosaicking jointly, namely VJDD, could lead to better video restoration performance than performing them separately. In addition to restoration accuracy, another key challenge to VJDD lies in the temporal consistency of consecutive frames. This issue exacerbates when perceptual regularization terms are introduced to enhance video perceptual quality. To address these challenges, we present a new VJDD framework by consistent and accurate latent space propagation, which leverages the estimation of previous frames as prior knowledge to ensure consistent recovery of the current frame. A data temporal consistency (DTC) loss and a relational perception consistency (RPC) loss are accordingly designed. Compared with the commonly used flow-based losses, the proposed losses can circumvent the error accumulation problem caused by inaccurate flow estimation and effectively handle intensity changes in videos, improving much the temporal consistency of output videos while preserving texture details. Extensive experiments demonstrate the leading VJDD performance of our method in term of restoration accuracy, perceptual quality and temporal consistency. Codes and dataset are available at \url{https://github.com/GuoShi28/VJDD}.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. Unprocessing images for learned raw denoising. In IEEE Conf. Comput. Vis. Pattern Recog., pages 11036–11045, 2019.
  2. Basicvsr: The search for essential components in video super-resolution and beyond. arXiv preprint arXiv:2012.02181, 2020.
  3. Understanding deformable alignment in video super-resolution. In AAAI, pages 973–981, 2021.
  4. Basicvsr++: Improving video super-resolution with enhanced propagation and alignment. In IEEE Conf. Comput. Vis. Pattern Recog., pages 5972–5981, 2022a.
  5. Investigating tradeoffs in real-world video super-resolution. In IEEE Conf. Comput. Vis. Pattern Recog., pages 5962–5971, 2022b.
  6. Free-form video inpainting with 3d gated convolution and temporal patchgan. In Int. Conf. Comput. Vis., pages 9066–9075, 2019.
  7. Learning temporal coherence via self-supervision for gan-based video generation. ACM Trans. Graph., 39(4):75–1, 2020.
  8. David R Cok. Signal processing method and apparatus for producing interpolated chrominance values in a sampled color image signal, 1987. US Patent 4,642,678.
  9. Joint demosaicking and denoising by total variation minimization. In IEEE Int. Conf. Image Process., pages 2781–2784. IEEE, 2012.
  10. Video demoireing with relation-based temporal consistency. In IEEE Conf. Comput. Vis. Pattern Recog., pages 17622–17631, 2022.
  11. Image quality assessment: Unifying structure and texture similarity. IEEE Trans. Pattern Anal. Mach. Intell., 44(5):2567–2581, 2020.
  12. A study of two cnn demosaicking algorithms. Image Processing On Line, 9:220–230, 2019.
  13. Joint demosaicking and denoising by fine-tuning of bursts of raw images. In Int. Conf. Comput. Vis., pages 8868–8877, 2019.
  14. Single-frame regularization for temporally stable cnns. In IEEE Conf. Comput. Vis. Pattern Recog., pages 11176–11185, 2019.
  15. Efficient video super-resolution through recurrent latent space propagation. In Int. Conf. Comput. Vis. Worksh., pages 3476–3485. IEEE, 2019.
  16. Deep joint demosaicking and denoising. ACM Trans. Graph., 35(6):1–12, 2016.
  17. Generative adversarial nets. Adv. Neural Inform. Process. Syst., 27, 2014.
  18. Toward convolutional blind denoising of real photographs. In IEEE Conf. Comput. Vis. Pattern Recog., pages 1712–1722, 2019.
  19. Joint denoising and demosaicking with green channel prior for real-world burst images. IEEE Trans. Image Process., 30:6930–6942, 2021.
  20. A differentiable two-stage alignment scheme for burst image reconstruction with large shift. In IEEE Conf. Comput. Vis. Pattern Recog., pages 17472–17481, 2022.
  21. Flexisp: A flexible camera image processing framework. ACM Trans. Graph., 33(6):1–13, 2014.
  22. Deep joint design of color filter arrays and demosaicing. In Computer Graphics Forum, pages 389–399. Wiley Online Library, 2018.
  23. Bidirectional recurrent convolutional networks for multi-frame super-resolution. In Adv. Neural Inform. Process. Syst., pages 235–243, 2015.
  24. Video super-resolution via bidirectional recurrent convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell., 40(4):1015–1028, 2017.
  25. Video super-resolution with recurrent structure-detail network. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, pages 645–660. Springer, 2020a.
  26. Video super-resolution with temporal group attention. In IEEE Conf. Comput. Vis. Pattern Recog., pages 8008–8017, 2020b.
  27. Expanding synthetic real-world degradations for blind video super resolution. In IEEE Conf. Comput. Vis. Pattern Recog., pages 1199–1208, 2023.
  28. A review of an old dilemma: Demosaicking first, or denoising first? In IEEE Conf. Comput. Vis. Pattern Recog. Worksh., pages 514–515, 2020.
  29. Perceptual losses for real-time style transfer and super-resolution. In Eur. Conf. Comput. Vis., pages 694–711. Springer, 2016.
  30. Self-supervised poisson-gaussian denoising. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2131–2139, 2021.
  31. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  32. Iterative joint image demosaicking and denoising using a residual denoising network. IEEE Trans. Image Process., 28(8):4177–4188, 2019.
  33. Learning blind video temporal consistency. In Eur. Conf. Comput. Vis., pages 170–185, 2018.
  34. Fully automatic video colorization with self-regularization and diversity. In IEEE Conf. Comput. Vis. Pattern Recog., pages 3753–3761, 2019.
  35. Non-local recurrent network for image restoration. Adv. Neural Inform. Process. Syst., 31, 2018.
  36. Joint demosaicing and denoising with self guidance. In IEEE Conf. Comput. Vis. Pattern Recog., pages 2240–2249, 2020a.
  37. A new polarization image demosaicking algorithm by exploiting inter-channel correlations with guided filtering. IEEE Trans. Image Process., 29:7076–7089, 2020b.
  38. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.
  39. Image denoising in mixed poisson–gaussian noise. IEEE Trans. Image Process., 20(3):696–708, 2010.
  40. Burst denoising with kernel prediction networks. In IEEE Conf. Comput. Vis. Pattern Recog., pages 2502–2510, 2018.
  41. Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study. In IEEE Conf. Comput. Vis. Pattern Recog. Worksh., pages 0–0, 2019.
  42. Preserving semantic and temporal consistency for unpaired video-to-video translation. In ACM Int. Conf. Multimedia, pages 1248–1257, 2019.
  43. pixabey. pixabey website. https://www.pixabay.com/, 2020.
  44. Neural nearest neighbors networks. Adv. Neural Inform. Process. Syst., 31, 2018.
  45. Trinity of pixel enhancement: a joint solution for demosaicking, denoising and super-resolution. arXiv preprint arXiv:1905.02538, 2019.
  46. Optical flow estimation using a spatial pyramid network. In IEEE Conf. Comput. Vis. Pattern Recog., pages 4161–4170, 2017.
  47. Frame-recurrent video super-resolution. In IEEE Conf. Comput. Vis. Pattern Recog., pages 6626–6634, 2018.
  48. Tempformer: Temporally consistent transformer for video denoising. In Eur. Conf. Comput. Vis., pages 481–496. Springer, 2022.
  49. Video-to-video synthesis. arXiv preprint arXiv:1808.06601, 2018.
  50. Consistent video style transfer via compound regularization. In AAAI, pages 12233–12240, 2020.
  51. Edvr: Video restoration with enhanced deformable convolutional networks. In IEEE Conf. Comput. Vis. Pattern Recog. Worksh., pages 0–0, 2019.
  52. Animesr: learning real-world super-resolution models for animation videos. Adv. Neural Inform. Process. Syst., 35:11241–11252, 2022.
  53. Mitigating artifacts in real-world video super-resolution models. In AAAI, pages 2956–2964, 2023.
  54. Learning deformable kernels for image and video denoising. arXiv preprint arXiv:1904.06903, 2019.
  55. Cross-channel correlation preserved three-stream lightweight cnns for demosaicking. arXiv preprint arXiv:1906.09884, 2019.
  56. An efficient adaptive interpolation for bayer cfa demosaicking. Sensing and Imaging, 20(1):37, 2019.
  57. Real-world video super-resolution: A benchmark dataset and a decomposition based learning scheme. In Int. Conf. Comput. Vis., pages 4781–4790, 2021.
  58. Supervised raw video denoising with a benchmark dataset on dynamic scenes. In IEEE Conf. Comput. Vis. Pattern Recog., pages 2301–2310, 2020.
  59. Learning temporal consistency for low light video enhancement from single images. In IEEE Conf. Comput. Vis. Pattern Recog., pages 4967–4976, 2021.
  60. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Trans. Image Process., 26(7):3142–3155, 2017.
  61. Ffdnet: Toward a fast and flexible solution for cnn-based image denoising. IEEE Trans. Image Process., 27(9):4608–4622, 2018a.
  62. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conf. Comput. Vis. Pattern Recog., pages 586–595, 2018b.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets