Papers
Topics
Authors
Recent
Search
2000 character limit reached

Bitstream-Corrupted Video Recovery: A Novel Benchmark Dataset and Method

Published 25 Sep 2023 in cs.CV and eess.IV | (2309.13890v2)

Abstract: The past decade has witnessed great strides in video recovery by specialist technologies, like video inpainting, completion, and error concealment. However, they typically simulate the missing content by manual-designed error masks, thus failing to fill in the realistic video loss in video communication (e.g., telepresence, live streaming, and internet video) and multimedia forensics. To address this, we introduce the bitstream-corrupted video (BSCV) benchmark, the first benchmark dataset with more than 28,000 video clips, which can be used for bitstream-corrupted video recovery in the real world. The BSCV is a collection of 1) a proposed three-parameter corruption model for video bitstream, 2) a large-scale dataset containing rich error patterns, multiple corruption levels, and flexible dataset branches, and 3) a plug-and-play module in video recovery framework that serves as a benchmark. We evaluate state-of-the-art video inpainting methods on the BSCV dataset, demonstrating existing approaches' limitations and our framework's advantages in solving the bitstream-corrupted video recovery problem. The benchmark and dataset are released at https://github.com/LIUTIGHE/BSCV-Dataset.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. Yuv sequence. http://trace.eas.asu.edu/yuv/. Accessed: 2023-06-06.
  2. The 6th annual bitmovin video developer report. https://bitmovin.com/video-developer-report, 2022.
  3. Impact of packet loss on 4k uhd video for portable devices. Multimedia tools and applications, 78:31733–31755, 2019.
  4. Depth-aware video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3703–3712, 2019.
  5. Robust h. 264 video decoding using crc-based single error correction and non-desynchronizing bits validation. In 2020 IEEE International Conference on Image Processing (ICIP), pages 1098–1102. IEEE, 2020.
  6. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  7. Free-form video inpainting with 3d gated convolution and temporal patchgan. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9066–9075, 2019.
  8. Motion compensated error concealment for hevc based on block-merging and residual energy. In 2013 20th International Packet Video Workshop, pages 1–6. IEEE, 2013.
  9. Bi-sequential video error concealment method using adaptive homography-based registration. IEEE Transactions on Circuits and Systems for Video Technology, 30(6):1535–1549, 2020.
  10. Cisco. Cisco visual networking index: Forecast and trends, 2017–2022. https://twiki.cern.ch/twiki/pub/HEPIX/TechwatchNetwork/HtwNetworkDocuments/white-paper-c11-741490.pdf.
  11. Dictionary-based multiple frame video super-resolution. In 2015 IEEE International Conference on Image Processing (ICIP), pages 83–87. IEEE, 2015.
  12. Video inpainting with short-term windows: application to object removal and error concealment. IEEE Transactions on Image Processing, 24(10):3034–3047, 2015.
  13. Flow-edge guided video completion. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, pages 713–729. Springer, 2020.
  14. Checksum-filtered list decoding applied to h. 264 and h. 265 video error correction. IEEE Transactions on Circuits and Systems for Video Technology, 28(8):1993–2006, 2017.
  15. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  16. How not to be seen—object removal from videos of crowded scenes. In Computer Graphics Forum, volume 31, pages 219–228. Wiley Online Library, 2012.
  17. A differentiable two-stage alignment scheme for burst image reconstruction with large shift. 2022.
  18. A multistage motion vector processing method for motion-compensated frame interpolation. IEEE transactions on image processing, 17(5):694–708, 2008.
  19. Temporally coherent completion of dynamic video. ACM Transactions on Graphics (TOG), 35(6):1–11, 2016.
  20. Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3224–3232, 2018.
  21. Video Team JVT Joint et al. Draft itu-t recommendation and final draft international standard of joint video specification. ITU-T Rec. H. 264/ISO/IEC 14496-10 AVC, 2003.
  22. Error compensation framework for flow-guided video inpainting. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV, pages 375–390. Springer, 2022.
  23. A review of temporal video error concealment techniques and their suitability for hevc and vvc. Multimedia Tools and Applications, 80:12685–12730, 2021.
  24. Deep video inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5792–5801, 2019.
  25. Sequential error concealment for video/images by sparse linear prediction. IEEE Transactions on Multimedia, 15(4):957–969, 2013.
  26. Kernel-based mmse multimedia signal reconstruction and its application to spatial error concealment. IEEE Transactions on Multimedia, 16(6):1729–1738, 2014.
  27. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84–90, 2017.
  28. Flow-guided video inpainting with scene templates. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14599–14608, 2021.
  29. Towards an end-to-end framework for flow-guided video inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17562–17571, 2022.
  30. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1833–1844, 2021.
  31. Pea265: Perceptual assessment of video compression artifacts. IEEE Transactions on Circuits and Systems for Video Technology, 30(11):3898–3910, 2020.
  32. Error concealment algorithm for hevc coded video using block partition decisions. In 2013 IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC 2013), pages 1–5. IEEE, 2013.
  33. Fuseformer: Fusing fine-grained information in transformers for video inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14040–14049, 2021.
  34. Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  35. Video inpainting of complex scenes. Siam journal on imaging sciences, 7(4):1993–2019, 2014.
  36. Video frame interpolation via adaptive convolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 670–679, 2017.
  37. Video frame interpolation via adaptive separable convolution. In Proceedings of the IEEE international conference on computer vision, pages 261–270, 2017.
  38. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  39. The 2017 davis challenge on video object segmentation. arXiv:1704.00675, 2017.
  40. Film: Frame interpolation for large motion. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII, pages 250–266. Springer, 2022.
  41. Dlformer: Discrete latent transformer for video inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3511–3520, 2022.
  42. Video error concealment using deep neural networks. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 380–384. IEEE, 2018.
  43. Super-resolution without explicit subpixel motion estimation. IEEE Transactions on Image Processing, 18(9):1958–1975, 2009.
  44. Tdan: Temporally-deformable alignment network for video super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3360–3369, 2020.
  45. The most dangerous codec in the world: Finding and exploiting vulnerabilities in h.264 decoders. In USENIX Security Symposium, 2023.
  46. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  47. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  48. Youtube ugc dataset for video compression research. In 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP), pages 1–5. IEEE, 2019.
  49. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004.
  50. Reed-Solomon codes and their applications. John Wiley & Sons, 1999.
  51. A spatial-focal error concealment scheme for corrupted focal stack video. In 2023 Data Compression Conference (DCC), pages 91–100, 2023.
  52. Generative adversarial networks based error concealment for low resolution video. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1827–1831. IEEE, 2019.
  53. Deep flow-guided video inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3723–3732, 2019.
  54. Video enhancement with task-oriented flow. International Journal of Computer Vision, 127:1106–1125, 2019.
  55. The 2nd large-scale video object segmentation challenge - video object segmentation track, October 2019.
  56. Hybrid spatial and temporal error concealment for distributed video coding. In 2008 IEEE International Conference on Multimedia and Expo, pages 633–636. IEEE, 2008.
  57. Progressive fusion video super-resolution network via exploiting non-local spatio-temporal correlations. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3106–3115, 2019.
  58. Learning joint spatial-temporal transformations for video inpainting. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pages 528–543. Springer, 2020.
  59. A low complexity motion compensated frame interpolation method. In 2005 IEEE International Symposium on Circuits and Systems (ISCAS), pages 4927–4930. IEEE, 2005.
  60. Flow-guided transformer for video inpainting. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVIII, pages 74–90. Springer, 2022.
  61. Inertia-guided flow completion and style fusion for video inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5982–5991, 2022.
  62. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up