Papers
Topics
Authors
Recent
Search
2000 character limit reached

CPGA: Coding Priors-Guided Aggregation Network for Compressed Video Quality Enhancement

Published 15 Mar 2024 in eess.IV and cs.CV | (2403.10362v2)

Abstract: Recently, numerous approaches have achieved notable success in compressed video quality enhancement (VQE). However, these methods usually ignore the utilization of valuable coding priors inherently embedded in compressed videos, such as motion vectors and residual frames, which carry abundant temporal and spatial information. To remedy this problem, we propose the Coding Priors-Guided Aggregation (CPGA) network to utilize temporal and spatial information from coding priors. The CPGA mainly consists of an inter-frame temporal aggregation (ITA) module and a multi-scale non-local aggregation (MNA) module. Specifically, the ITA module aggregates temporal information from consecutive frames and coding priors, while the MNA module globally captures spatial information guided by residual frames. In addition, to facilitate research in VQE task, we newly construct the Video Coding Priors (VCP) dataset, comprising 300 videos with various coding priors extracted from corresponding bitstreams. It remedies the shortage of previous datasets on the lack of coding information. Experimental results demonstrate the superiority of our method compared to existing state-of-the-art methods. The code and dataset will be released at https://github.com/VQE-CPGA/CPGA.git .

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Study of temporal effects on subjective video quality of experience. IEEE Transactions on Image Processing, 26(11):5217–5231, 2017.
  2. Overview of the versatile video coding (vvc) standard and its applications. IEEE Transactions on Circuits and Systems for Video Technology, 31(10):3736–3764, 2021.
  3. Compressed domain deep video super-resolution. IEEE Transactions on Image Processing, 30:7156–7169, 2021.
  4. An overview of core coding tools in the av1 video codec. In 2018 picture coding symposium (PCS), pages 41–45. IEEE, 2018.
  5. A convolutional neural network approach for post-processing in hevc intra coding. In MultiMedia Modeling: 23rd International Conference, MMM 2017, Reykjavik, Iceland, January 4-6, 2017, Proceedings, Part I 23, pages 28–39. Springer, 2017.
  6. Spatio-temporal deformable convolution for compressed video quality enhancement. In Proceedings of the AAAI conference on artificial intelligence, pages 10696–10703, 2020.
  7. Compression artifacts reduction by a deep convolutional network. In Proceedings of the IEEE international conference on computer vision, pages 576–584, 2015.
  8. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 2758–2766, 2015.
  9. Rpan: An end-to-end recurrent pose-attention network for action recognition in videos. In Proceedings of the IEEE international conference on computer vision, pages 3725–3734, 2017.
  10. Mfqe 2.0: A new approach for multi-frame quality enhancement on compressed video. IEEE transactions on pattern analysis and machine intelligence, 43(3):949–963, 2019.
  11. Hybrid video coding scheme based on vvc and spatio-temporal attention convolution neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 1791–1794, 2022.
  12. Enhancing hevc compressed videos with a partition-masked convolutional neural network. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 216–220. IEEE, 2018.
  13. Adam: A method for stochastic optimization. 2014.
  14. Partition-aware adaptive switching neural networks for post-processing in hevc. IEEE Transactions on Multimedia, 22(11):2749–2763, 2019.
  15. Ada-dqa: Adaptive diverse quality-aware feature acquisition for video quality assessment. In Proceedings of the 31th ACM International Conference on Multimedia, pages 6695–6704, 2023.
  16. Coarse-to-fine spatio-temporal information fusion for compressed video quality enhancement. IEEE Signal Processing Letters, 29:543–547, 2022a.
  17. Spatio-temporal detail information retrieval for compressed video quality enhancement. IEEE Transactions on Multimedia, 2022b.
  18. Bvi-dvc: A training database for deep video compression. IEEE Transactions on Multimedia, 24:3847–3858, 2021.
  19. Comparison of the coding efficiency of video coding standards—including high efficiency video coding (hevc). IEEE Transactions on circuits and systems for video technology, 22(12):1669–1684, 2012.
  20. Deepid-net: Deformable deep convolutional neural networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2403–2412, 2015.
  21. Cnn-based in-loop filtering for coding efficiency improvement. In 2016 IEEE 12th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), pages 1–5. IEEE, 2016.
  22. M. H. Pinson. The consumer digital video library [best of the web]. pages 5646–5654, 2021.
  23. Blind image super-resolution with rich texture-aware codebook. In Proceedings of the 31st ACM International Conference on Multimedia, pages 676–687, 2023.
  24. Dual circle contrastive learning-based blind image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology, 34(3):1757–1771, 2024.
  25. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  26. Study of subjective and objective quality assessment of video. IEEE transactions on Image Processing, 19(6):1427–1441, 2010.
  27. Overview of the high efficiency video coding (hevc) standard. IEEE Transactions on circuits and systems for video technology, 22(12):1649–1668, 2012.
  28. Video quality evaluation methodology and verification testing of hevc compression performance. IEEE Transactions on Circuits and Systems for Video Technology, 26(1):76–90, 2015.
  29. Partition tree guided progressive rethinking network for in-loop filtering of hevc. In 2019 IEEE International Conference on Image Processing (ICIP), pages 2671–2675. IEEE, 2019.
  30. A novel deep learning-based method of improving coding efficiency from the decoder-end for hevc. In 2017 data compression conference (DCC), pages 410–419. IEEE, 2017.
  31. Compression-aware video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2012–2021, 2023.
  32. Overview of the h. 264/avc video coding standard. IEEE Transactions on circuits and systems for video technology, 13(7):560–576, 2003.
  33. Mathias Wien. High efficiency video coding. Coding Tools and specification, 24, 2015.
  34. Transcoded video restoration by temporal spatial auxiliary network. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2875–2883, 2022.
  35. Tencent video dataset (tvd): A video dataset for learning-based visual data compression and analysis. arXiv preprint arXiv:2105.05961, 2021.
  36. Video enhancement with task-oriented flow. International Journal of Computer Vision, 127:1106–1125, 2019.
  37. Ren Yang. Ntire 2021 challenge on quality enhancement of compressed video: Dataset and study. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 667–676, 2021.
  38. Enhancing quality for hevc compressed videos. IEEE Transactions on Circuits and Systems for Video Technology, 29(7):2039–2054, 2018a.
  39. Enhancing quality for hevc compressed videos. IEEE Transactions on Circuits and Systems for Video Technology, 29(7):2039–2054, 2018b.
  40. Multi-frame quality enhancement for compressed video. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6664–6673, 2018c.
  41. Ntire 2022 challenge on super-resolution and quality enhancement of compressed video: Dataset, methods and results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1221–1238, 2022.
  42. Capturing co-existing distortions in user-generated content for no-reference video quality assessment. In Proceedings of the 31th ACM International Conference on Multimedia, pages 1098–1107, 2023.
  43. Real-time action recognition with deeply transferred motion vector cnns. IEEE Transactions on Image Processing, 27(5):2326–2339, 2018a.
  44. A codec information assisted framework for efficient compressed video super-resolution. In European Conference on Computer Vision, pages 220–235. Springer, 2022.
  45. Boosting single image super-resolution via partial channel shifting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13223–13232, 2023.
  46. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European conference on computer vision (ECCV), pages 286–301, 2018b.
  47. Residual non-local attention networks for image restoration. 2019.
  48. Zoom-vqa: Patches, frames and clips integration for video quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 1302–1310, 2023a.
  49. Quality-aware pretrained models for blind image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22302–22313, 2023b.
  50. Real-time moving object segmentation and classification from hevc compressed surveillance video. IEEE Transactions on Circuits and Systems for Video Technology, 28(6):1346–1357, 2016.
  51. Recursive fusion and deformable spatiotemporal attention for video compression artifact reduction. In Proceedings of the 29th ACM international conference on multimedia, pages 5646–5654, 2021.
  52. Dynamic spatial focus for efficient compressed video action recognition. IEEE Transactions on Circuits and Systems for Video Technology, 34(2):695–708, 2024.
  53. Infrared small target detection using local feature-based density peaks searching. IEEE Geoscience and Remote Sensing Letters, 19:1–5, 2022.
  54. Attention retractable frequency fusion transformer for image super resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1756–1763, 2023.
  55. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9308–9316, 2019.
Citations (4)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up