Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Rolling Shutter Correction with Intermediate Distortion Flow Estimation (2404.06350v1)

Published 9 Apr 2024 in cs.CV

Abstract: This paper proposes to correct the rolling shutter (RS) distorted images by estimating the distortion flow from the global shutter (GS) to RS directly. Existing methods usually perform correction using the undistortion flow from the RS to GS. They initially predict the flow from consecutive RS frames, subsequently rescaling it as the displacement fields from the RS frame to the underlying GS image using time-dependent scaling factors. Following this, RS-aware forward warping is employed to convert the RS image into its GS counterpart. Nevertheless, this strategy is prone to two shortcomings. First, the undistortion flow estimation is rendered inaccurate by merely linear scaling the flow, due to the complex non-linear motion nature. Second, RS-aware forward warping often results in unavoidable artifacts. To address these limitations, we introduce a new framework that directly estimates the distortion flow and rectifies the RS image with the backward warping operation. More specifically, we first propose a global correlation-based flow attention mechanism to estimate the initial distortion flow and GS feature jointly, which are then refined by the following coarse-to-fine decoder layers. Additionally, a multi-distortion flow prediction strategy is integrated to mitigate the issue of inaccurate flow estimation further. Experimental results validate the effectiveness of the proposed method, which outperforms state-of-the-art approaches on various benchmarks while maintaining high efficiency. The project is available at \url{https://github.com/ljzycmd/DFRSC}.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. From two rolling shutters to one global shutter. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2505–2513, 2020.
  2. Learning adaptive warping for real-world rolling shutter correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17785–17793, 2022.
  3. Rolling shutter camera relative pose: Generalized epipolar geometry. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4132–4140, 2016.
  4. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 2758–2766, 2015.
  5. Direct sparse odometry. IEEE transactions on pattern analysis and machine intelligence, 40(3):611–625, 2017.
  6. Inverting a rolling shutter camera: bring rolling shutter images to high framerate global shutter video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4228–4237, 2021.
  7. Sunet: symmetric undistortion network for rolling shutter correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4541–4550, 2021a.
  8. Rolling-shutter-stereo-aware motion estimation and image correction. Computer Vision and Image Understanding, 213:103296, 2021b.
  9. Context-aware video reconstruction for rolling shutter cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17572–17582, 2022.
  10. Joint appearance and motion learning for efficient rolling shutter correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5671–5681, 2023.
  11. Calibration-free rolling shutter removal. In IEEE International Conference on Computational Photography, pages 1–8, 2012.
  12. Real-time intermediate flow estimation for video frame interpolation. In European Conference on Computer Vision, pages 624–642. Springer, 2022.
  13. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2462–2470, 2017.
  14. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9000–9008, 2018.
  15. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 694–711. Springer, 2016.
  16. Direct semi-dense slam for rolling shutter cameras. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 1308–1315. IEEE, 2016.
  17. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  18. Ifrnet: Intermediate feature refine network for efficient frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1969–1978, 2022.
  19. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 624–632, 2017.
  20. A robust method for strong rolling shutter effects correction using lines with automatic feature selection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018a.
  21. A robust method for strong rolling shutter effects correction using lines with automatic feature selection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4795–4803, 2018b.
  22. Amt: All-pairs multi-field transforms for efficient frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9801–9810, 2023.
  23. Revisiting rolling shutter bundle adjustment: Toward accurate and fast solution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4863–4871, 2023.
  24. Deep shutter unrolling network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5941–5949, 2020.
  25. Combining internal and external constraints for unrolling shutter in videos. In European Conference on Computer Vision, pages 119–134. Springer, 2022.
  26. Rolling shutter correction in manhattan world. In Proceedings of the IEEE International Conference on Computer Vision, pages 882–890, 2017.
  27. Towards nonlinear-motion-aware and occlusion-robust rolling shutter correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10680–10688, 2023.
  28. From bows to arrows: Rolling shutter rectification of urban scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016a.
  29. From bows to arrows: Rolling shutter rectification of urban scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2773–2781, 2016b.
  30. Unrolling the shutter: Cnn to correct motion distortions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
  31. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  32. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934–8943, 2018.
  33. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020.
  34. Occlusion-aware rolling shutter rectification of 3d scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 636–645, 2018.
  35. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  36. Learning feature descriptors using camera pose supervision. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 757–774. Springer, 2020.
  37. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  38. Gmflow: Learning optical flow via global matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8121–8130, 2022.
  39. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  40. Towards rolling shutter correction and deblurring in dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9219–9228, 2021.
  41. Image stitching and rectification for hand-held cameras. In European Conference on Computer Vision, pages 243–260. Springer, 2020.
  42. Rolling-shutter-aware differential sfm and image rectification. In Proceedings of the IEEE International Conference on Computer Vision, pages 948–956, 2017.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now