Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

HSTR-Net: Reference Based Video Super-resolution with Dual Cameras (2310.12092v2)

Published 18 Oct 2023 in cs.CV

Abstract: High-spatio-temporal resolution (HSTR) video recording plays a crucial role in enhancing various imagery tasks that require fine-detailed information. State-of-the-art cameras provide this required high frame-rate and high spatial resolution together, albeit at a high cost. To alleviate this issue, this paper proposes a dual camera system for the generation of HSTR video using reference-based super-resolution (RefSR). One camera captures high spatial resolution low frame rate (HSLF) video while the other captures low spatial resolution high frame rate (LSHF) video simultaneously for the same scene. A novel deep learning architecture is proposed to fuse HSLF and LSHF video feeds and synthesize HSTR video frames. The proposed model combines optical flow estimation and (channel-wise and spatial) attention mechanisms to capture the fine motion and complex dependencies between frames of the two video feeds. Simulations show that the proposed model provides significant improvement over existing reference-based SR techniques in terms of PSNR and SSIM metrics. The method also exhibits sufficient frames per second (FPS) for aerial monitoring when deployed on a power-constrained drone equipped with dual cameras.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Video enhancement with task-oriented flow, International Journal of Computer Vision (IJCV) 127 (2019) 1106–1125.
  2. A dual camera system for high spatiotemporal resolution video acquisition, IEEE transactions on pattern analysis and machine intelligence 43 (2020) 3275–3291.
  3. Image super-resolution using deep convolutional networks, IEEE transactions on pattern analysis and machine intelligence 38 (2015) 295–307.
  4. Accurate image super-resolution using very deep convolutional networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1646–1654.
  5. Enhanced deep residual networks for single image super-resolution, in: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2017, pp. 136–144.
  6. Photo-realistic single image super-resolution using a generative adversarial network, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4681–4690.
  7. Deep generative image models using a laplacian pyramid of adversarial networks, Advances in neural information processing systems 28 (2015).
  8. Esrgan: Enhanced super-resolution generative adversarial networks, in: Proceedings of the European conference on computer vision (ECCV) workshops, 2018, pp. 0–0.
  9. Single image super-resolution with non-local means and steering kernel regression, IEEE Transactions on Image Processing 21 (2012) 4544–4556. doi:10.1109/TIP.2012.2208977.
  10. Example-based super-resolution, IEEE Computer graphics and Applications 22 (2002) 56–65.
  11. Image super-resolution by neural texture transfer, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 7982–7991.
  12. Robust reference-based super-resolution with similarity-aware deformable convolution, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 8425–8434.
  13. Learning texture transformer network for image super-resolution, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 5791–5800.
  14. Masa-sr: Matching acceleration and spatial adaptation for reference-based image super-resolution, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 6368–6377.
  15. R. Lin, N. Xiao, Residual channel attention connection network for reference-based image super-resolution, in: 2021 8th International Conference on Information, Cybernetics, and Computational Social Systems (ICCSS), IEEE, 2021, pp. 307–313.
  16. Task decoupled framework for reference-based super-resolution, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5931–5940.
  17. Dsma: Reference-based image super-resolution method based on dual-view supervised learning and multi-attention mechanism, IEEE Access 10 (2022) 54649–54659.
  18. Crossnet: An end-to-end reference-based super resolution network using cross-scale warping, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 88–104.
  19. Robust reference-based super-resolution via c2-matching, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2103–2112.
  20. Dual-camera super-resolution with aligned attention modules, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2001–2010.
  21. Coarse-to-fine embedded patchmatch and multi-scale dynamic aggregation for reference-based super-resolution, arXiv preprint arXiv:2201.04358 (2022).
  22. Reference-based video super-resolution using multi-camera video triplets, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 17824–17833.
  23. Real-time intermediate flow estimation for video frame interpolation, in: European Conference on Computer Vision, Springer, 2022, pp. 624–642.
  24. A unified pyramid recurrent network for video frame interpolation, arXiv preprint arXiv:2211.03456 (2022).
  25. Flow guidance deformable compensation network for video frame interpolation, arXiv preprint arXiv:2211.12117 (2022).
  26. Ttvfi: Learning trajectory-aware transformer for video frame interpolation, arXiv preprint arXiv:2207.09048 (2022).
  27. Bi-directional pseudo-three-dimensional network for video frame interpolation, IEEE Transactions on Image Processing 31 (2022) 6773–6788.
  28. Enhanced spatial-temporal freedom for video frame interpolation, Applied Intelligence (2022) 1–13.
  29. Amt: All-pairs multi-field transforms for efficient frame interpolation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 9801–9810.
  30. H-vfi: Hierarchical frame interpolation for videos with large motions, arXiv preprint arXiv:2211.11309 (2022).
  31. Progressive spatial-temporal collaborative network for video frame interpolation, in: Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 2145–2153.
  32. Frame interpolation transformer and uncertainty guidance, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 9811–9821.
  33. Swin transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10012–10022.
  34. Dual camera based high spatio-temporal resolution video generation for wide area surveillance, in: 2022 18th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), IEEE, 2022, pp. 1–8.
  35. Vision meets drones: Past, present and future, arXiv preprint arXiv:2001.06303 1 (2020) 8.
  36. Double-layer search and adaptive pooling fusion for reference-based image super-resolution, ACM Trans. Multimedia Comput. Commun. Appl. 20 (2023). URL: https://doi.org/10.1145/3604937. doi:10.1145/3604937.
  37. A feature reuse framework with texture-adaptive aggregation for reference-based super-resolution, arXiv preprint arXiv:2306.01500 (2023).
  38. Geometry enhanced reference-based image super-resolution, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 6123–6132.
  39. Mvfi-net: Motion-aware video frame interpolation network, in: Proceedings of the Asian Conference on Computer Vision, 2022, pp. 3690–3706.
  40. Hybrid warping fusion for video frame interpolation, International Journal of Computer Vision 130 (2022) 2980–2993.
  41. Video frame interpolation with densely queried bilateral correlation, arXiv preprint arXiv:2304.13596 (2023).
  42. Optical flow reusing for high-efficiency space-time video super resolution, IEEE Transactions on Circuits and Systems for Video Technology (2022).
  43. U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical image computing and computer-assisted intervention, Springer, 2015, pp. 234–241.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now