Papers
Topics
Authors
Recent
Search
2000 character limit reached

Binarized Low-light Raw Video Enhancement

Published 29 Mar 2024 in cs.CV and eess.IV | (2403.19944v1)

Abstract: Recently, deep neural networks have achieved excellent performance on low-light raw video enhancement. However, they often come with high computational complexity and large memory costs, which hinder their applications on resource-limited devices. In this paper, we explore the feasibility of applying the extremely compact binary neural network (BNN) to low-light raw video enhancement. Nevertheless, there are two main issues with binarizing video enhancement models. One is how to fuse the temporal information to improve low-light denoising without complex modules. The other is how to narrow the performance gap between binary convolutions with the full precision ones. To address the first issue, we introduce a spatial-temporal shift operation, which is easy-to-binarize and effective. The temporal shift efficiently aggregates the features of neighbor frames and the spatial shift handles the misalignment caused by the large motion in videos. For the second issue, we present a distribution-aware binary convolution, which captures the distribution characteristics of real-valued input and incorporates them into plain binary convolutions to alleviate the degradation in performance. Extensive quantitative and qualitative experiments have shown our high-efficiency binarized low-light raw video enhancement method can attain a promising performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Video denoising via empirical bayesian estimation of space-time patches. J. Math. Imaging Vis., 60(1):70–93, 2018.
  2. Binarized spectral compressive imaging. arXiv preprint arXiv:2305.10299, 2023.
  3. Two deterministic half-quadratic regularization algorithms for computed imaging. In ICIP, pages 168–172, 1994.
  4. Learning to see in the dark. In CVPR, pages 3291–3300, 2018.
  5. Seeing motion in the dark. In ICCV, pages 3184–3193, 2019.
  6. Instance segmentation in the dark. IJCV, 131(8):2198–2218, 2023.
  7. Binaryconnect: Training deep neural networks with binary weights during propagations. In NeurIPS, pages 3123–3131, 2015.
  8. Dancing in the dark: A benchmark towards general low-light video enhancement. In ICCV, pages 12877–12886, 2023.
  9. LE-GAN: Unsupervised low-light image enhancement network using attention module and identity invariant loss. Knowledge-Based Systems, 240:108010, 2022a.
  10. Low-light raw video denoising with a high-quality realistic motion dataset. IEEE TMM, pages 1–13, 2022b.
  11. Crafting object detection in very low light. In BMVC, 2021.
  12. Towards low light enhancement with RAW images. IEEE TIP, 31:1391–1405, 2022.
  13. Binarized neural networks. In NeurIPS, pages 4107–4115, 2016.
  14. Learning to see moving objects in the dark. In ICCV, pages 7323–7332, 2019.
  15. Training binary neural network without batch normalization for image super-resolution. In AAAI, pages 1700–1707, 2021.
  16. Adam: A method for stochastic optimization. In ICLR, 2015.
  17. INSTA-BNN: Binary neural network with instance-aware threshold. In ICCV, pages 17325–17334, 2023.
  18. A simple baseline for video restoration with grouped spatial-temporal shift. In CVPR, pages 9822–9832, 2023.
  19. Unidirectional video denoising by mimicking backward recurrent modules with look-ahead forward ones. In ECCV, pages 592–609, 2022.
  20. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In ECCV, pages 747–763, 2018.
  21. ReActNet: Towards precise binary neural network with generalized activation functions. In ECCV, pages 143–159, 2020.
  22. Video denoising using separable 4d nonlocal spatiotemporal transforms. In Image Processing: Algorithms and Systems IX, pages 9–19, 2011.
  23. Efficient multi-stage video denoising with recurrent spatio-temporal fusion. In CVPR, pages 3466–3475, 2021.
  24. Binary neural networks: A survey. PR, 105:107281, 2020a.
  25. Forward and backward information retention for accurate binary neural networks. In CVPR, pages 2247–2256, 2020b.
  26. XNOR-Net: Imagenet classification using binary convolutional neural networks. In ECCV, pages 525–542, 2016.
  27. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234–241, 2015.
  28. Video quality assessment by reduced reference spatio-temporal entropic differencing. IEEE TCSVT, 23(4):684–694, 2013.
  29. DVDNET: A fast network for deep video denoising. In ICIP, pages 1805–1809, 2019.
  30. FastDVDnet: Towards real-time deep video denoising without flow estimation. In CVPR, pages 1351–1360, 2020.
  31. Low light video enhancement using synthetic data produced with an intermediate domain mapping. In ECCV, pages 103–119, 2020.
  32. Trainability preserving neural structured pruning. In ICLR, 2023.
  33. What makes a ”good” data augmentation in knowledge distillation - A statistical perspective. In NeurIPS, 2022.
  34. ECA-Net: Efficient channel attention for deep convolutional neural networks. In CVPR, pages 11531–11539, 2020.
  35. Seeing dynamic scene in the dark: A high-quality video dataset with mechatronic alignment. In ICCV, pages 9680–9689, 2019a.
  36. Enhancing low light videos by exploring high sensitivity camera noise. In ICCV, pages 4110–4118, 2019b.
  37. Recurrent self-supervised video denoising with denser receptive field. In ACM MM, pages 7363–7372, 2023.
  38. A physics-based noise formation model for extreme low-light raw denoising. In CVPR, pages 2755–2764, 2020.
  39. Physics-based noise modeling for extreme low-light photography. IEEE TPAMI, 44(11):8520–8537, 2022.
  40. Estimator meets equilibrium perspective: A rectified straight through estimator for binary neural networks training. In ICCV, pages 17055–17064, 2023.
  41. Basic binary convolution unit for binarized image restoration network. In ICLR, 2023.
  42. Binarized neural network for single image super resolution. In ECCV, pages 91–107, 2020.
  43. Learning frequency domain approximation for binary neural networks. In NeurIPS, pages 25553–25565, 2021.
  44. Zhe Xu and Ray C. C. Cheung. Accurate and compact convolutional neural networks with trained binarization. In BMVC, page 19, 2019.
  45. Video enhancement with task-oriented flow. IJCV, 127(8):1106–1125, 2019.
  46. Supervised raw video denoising with a benchmark dataset on dynamic scenes. In CVPR, pages 2298–2307, 2020.
  47. 3⁢D2⁢U⁢n⁢e⁢t3superscript𝐷2𝑈𝑛𝑒𝑡3D^{2}Unet3 italic_D start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_U italic_n italic_e italic_t: 3d deformable unet for low-light video enhancement. In PRCV, pages 66–77, 2021.
  48. Learning temporal consistency for low light video enhancement from single images. In CVPR, pages 4967–4976, 2021a.
  49. Learning rain location prior for nighttime deraining. In ICCV, pages 13148–13157, 2023.
  50. Hyperspectral image denoising with realistic data. In ICCV, pages 2228–2237, 2021b.
  51. Guided hyperspectral image denoising with realistic data. IJCV, 130(11):2885–2901, 2022a.
  52. ShuffleNet: An extremely efficient convolutional neural network for mobile devices. In CVPR, pages 6848–6856, 2018.
  53. Learning efficient image super-resolution networks via structure-regularized pruning. In ICLR, 2022b.
  54. Estimating fine-grained noise model via contrastive learning. In CVPR, pages 12682–12691, 2022.
  55. RawHDR: High dynamic range image reconstruction from a single raw image. In ICCV, pages 12334–12344, 2023.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up