Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Decoupling Degradations with Recurrent Network for Video Restoration in Under-Display Camera (2403.05660v1)

Published 8 Mar 2024 in cs.CV

Abstract: Under-display camera (UDC) systems are the foundation of full-screen display devices in which the lens mounts under the display. The pixel array of light-emitting diodes used for display diffracts and attenuates incident light, causing various degradations as the light intensity changes. Unlike general video restoration which recovers video by treating different degradation factors equally, video restoration for UDC systems is more challenging that concerns removing diverse degradation over time while preserving temporal consistency. In this paper, we introduce a novel video restoration network, called D$2$RNet, specifically designed for UDC systems. It employs a set of Decoupling Attention Modules (DAM) that effectively separate the various video degradation factors. More specifically, a soft mask generation function is proposed to formulate each frame into flare and haze based on the diffraction arising from incident light of different intensities, followed by the proposed flare and haze removal components that leverage long- and short-term feature learning to handle the respective degradations. Such a design offers an targeted and effective solution to eliminating various types of degradation in UDC systems. We further extend our design into multi-scale to overcome the scale-changing of degradation that often occur in long-range videos. To demonstrate the superiority of D$2$RNet, we propose a large-scale UDC video benchmark by gathering HDR videos and generating realistically degraded videos using the point spread function measured by a commercial UDC system. Extensive quantitative and qualitative evaluations demonstrate the superiority of D$2$RNet compared to other state-of-the-art video restoration and UDC image restoration methods. Code is available at https://github.com/ChengxuLiu/DDRNet.git

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Homography Theories Used for Image Mapping: A Review. In 2022 10th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions)(ICRITO), 1–5. IEEE.
  2. A decoupled kernel prediction network guided by soft mask for single image HDR reconstruction. ACM Transactions on Multimedia Computing, Communications and Applications, 19(2s): 1–23.
  3. BasicVSR++: Improving video super-resolution with enhanced propagation and alignment. In CVPR, 5972–5981.
  4. Rethinking coarse-to-fine approach in single image deblurring. In ICCV, 4641–4650.
  5. Flare7k: A phenomenological nighttime flare removal dataset. NeurIPS, 35: 3926–3937.
  6. HDR image reconstruction from a single exposure using deep CNNs. ACM TOG, 36(6): 1–15.
  7. Generating Aligned Pseudo-Supervision from Non-Aligned Data for Image Restoration in Under-Display Camera. In CVPR, 5013–5022.
  8. Removing diffraction image artifacts in under-display camera via dynamic skip connection network. In CVPR, 662–671.
  9. Mipi 2022 challenge on under-display camera image restoration: Methods and results. arXiv preprint arXiv:2209.07052.
  10. Deep residual learning for image recognition. In CVPR, 770–778.
  11. Bidirectional recurrent convolutional networks for multi-frame super-resolution. NeurIPS, 28.
  12. Video super-resolution via bidirectional recurrent convolutional networks. IEEE TPAMI, 40(4): 1015–1028.
  13. Video super-resolution with recurrent structure-detail network. In ECCV, 645–660. Springer.
  14. Spatio-temporal transformer network for video restoration. In ECCV, 106–122.
  15. Bnudc: A two-branched deep neural network for restoring images from under-display cameras. In CVPR, 1950–1959.
  16. Controllable image restoration for under-display camera in smartphones. In CVPR, 2073–2082.
  17. A Simple Baseline for Video Restoration With Grouped Spatial-Temporal Shift. In CVPR, 9822–9832.
  18. Arvo: Learning all-range volumetric correspondence for video deblurring. In CVPR, 7721–7731.
  19. VRT: A video restoration transformer. arXiv preprint arXiv:2201.12288.
  20. Recurrent video restoration transformer with guided deformable attention. NeurIPS, 35: 378–393.
  21. Flow-guided sparse transformer for video deblurring. arXiv preprint arXiv:2201.01893.
  22. FSI: Frequency and Spatial Interactive Learning for Image Restoration in Under-Display Cameras. In ICCV, 12537–12546.
  23. Learning trajectory-aware transformer for video super-resolution. In CVPR, 5687–5696.
  24. 4D LUT: learnable context-aware 4d lookup table for image enhancement. IEEE TIP, 32: 4742–4756.
  25. TTVFI: Learning trajectory-aware transformer for video frame interpolation. IEEE TIP.
  26. Unsupervised Global and Local Homography Estimation With Motion Basis Learning. IEEE TPAMI.
  27. UDC-UNet: Under-Display Camera Image Restoration via U-shape Dynamic Network. In ECCV, 113–129. Springer.
  28. Nighthazeformer: Single nighttime haze removal using prior query transformer. In ACM MM, 4119–4128.
  29. Transform domain pyramidal dilated convolution networks for restoration of under display camera images. In ECCVW, 364–378. Springer.
  30. P-78: Simulator-Based Efficient Panel Design and Image Retrieval for Under-Display Cameras. In SID Symposium Digest of Technical Papers, volume 52, 1372–1375. Wiley Online Library.
  31. Optical flow estimation using a spatial pyramid network. In CVPR, 4161–4170.
  32. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 234–241. Springer.
  33. Frame-recurrent video super-resolution. In CVPR, 6626–6634.
  34. Spatially-attentive patch-hierarchical network for adaptive motion deblurring. In CVPR, 3606–3615.
  35. Scale-recurrent network for deep image deblurring. In CVPR, 8174–8182.
  36. FastDVDnet: Towards real-time deep video denoising without flow estimation. In CVPR, 1354–1363.
  37. EDVR: Video restoration with enhanced deformable convolutional networks. In CVPRW, 0–0.
  38. Efficient video deblurring guided by motion magnitude. In ECCV, 413–429. Springer.
  39. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4): 600–612.
  40. Motion basis learning for unsupervised deep homography estimation with subspace projection. In ICCV, 13117–13125.
  41. Multi-stage progressive image restoration. In CVPR, 14821–14831.
  42. Deep stacked hierarchical multi-patch network for image deblurring. In CVPR, 5978–5986.
  43. Spatio-temporal deformable attention network for video deblurring. In ECCV, 581–596. Springer.
  44. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 586–595.
  45. Real-world video deblurring: A benchmark dataset and an efficient recurrent neural network. IJCV, 131(1): 284–301.
  46. UDC 2020 challenge on image restoration of under-display camera: Methods and results. In ECCVW, 337–351. Springer.
  47. Image restoration for under-display camera. In CVPR, 9179–9188.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now