Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Video Super-Resolution for Optimized Bitrate and Green Online Streaming (2402.03513v1)

Published 5 Feb 2024 in cs.MM

Abstract: Conventional per-title encoding schemes strive to optimize encoding resolutions to deliver the utmost perceptual quality for each bitrate ladder representation. Nevertheless, maintaining encoding time within an acceptable threshold is equally imperative in online streaming applications. Furthermore, modern client devices are equipped with the capability for fast deep-learning-based video super-resolution (VSR) techniques, enhancing the perceptual quality of the decoded bitstream. This suggests that opting for lower resolutions in representations during the encoding process can curtail the overall energy consumption without substantially compromising perceptual quality. In this context, this paper introduces a video super-resolution-based latency-aware optimized bitrate encoding scheme (ViSOR) designed for online adaptive streaming applications. ViSOR determines the encoding resolution for each target bitrate, ensuring the highest achievable perceptual quality after VSR within the bound of a maximum acceptable latency. Random forest-based prediction models are trained to predict the perceptual quality after VSR and the encoding time for each resolution using the spatiotemporal features extracted for each video segment. Experimental results show that ViSOR targeting fast super-resolution convolutional neural network (FSRCNN) achieves an overall average bitrate reduction of 24.65 % and 32.70 % to maintain the same PSNR and VMAF, compared to the HTTP Live Streaming (HLS) bitrate ladder encoding of 4 s segments using the x265 encoder, when the maximum acceptable latency for each representation is set as two seconds. Considering a just noticeable difference (JND) of six VMAF points, the average cumulative storage consumption and encoding energy for each segment is reduced by 79.32 % and 68.21 %, respectively, contributing towards greener streaming.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. A. Bentaleb et al., “A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP,” in IEEE Communications Surveys Tutorials, vol. 21, no. 1, 2019, pp. 562–585.
  2. Apple Inc., “HLS Authoring Specification for Apple Devices.” [Online]. Available: https://developer.apple.com/documentation/http-live-streaming/hls-authoring-specification-for-apple-devices
  3. J. De Cock et al., “Complexity-based consistent-quality encoding in the cloud,” in 2016 IEEE International Conference on Image Processing (ICIP), 2016, pp. 1484–1488.
  4. A. V. Katsenou, J. Sole, and D. R. Bull, “Content-gnostic Bitrate Ladder Prediction for Adaptive Video Streaming,” in 2019 Picture Coding Symposium (PCS), 2019, pp. 1–5.
  5. M. Bhat, J.-M. Thiesse, and P. L. Callet, “Combining Video Quality Metrics To Select Perceptually Accurate Resolution In A Wide Quality Range: A Case Study,” in 2021 IEEE International Conference on Image Processing (ICIP), 2021, pp. 2164–2168.
  6. A. Zabrovskiy et al., “FAUST: Fast Per-Scene Encoding Using Entropy-Based Scene Detection and Machine Learning,” in 2021 30th Conference of Open Innovations Association FRUCT, 2021, pp. 292–302.
  7. V. V. Menon et al., “All-Intra Rate Control Using Low Complexity Video Features for Versatile Video Coding,” in 2023 IEEE International Conference on Image Processing (ICIP), 2023, pp. 2760–2764.
  8. V. V. Menon et al., “Content-Adaptive Variable Framerate Encoding Scheme for Green Live Streaming,” 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2311.08074
  9. F. Nasiri et al., “Multi-Preset Video Encoder Bitrate Ladder Prediction,” in Proceedings of the 2nd International Workshop on Design, Deployment, and Evaluation of Network-Assisted Video Streaming, 2022, p. 8–13.
  10. V. V. Menon et al., “Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low latency Encoding,” in Proceedings of the 3rd Mile-High Video Conference, 2024.
  11. P. Ramachandran et al., “Content Adaptive Live Encoding with Open Source Codecs,” in Proceedings of the 11th ACM Multimedia Systems Conference, 2020, p. 345–348.
  12. H. Liu et al., “Video Super-Resolution Based on Deep Learning: A Comprehensive Survey,” in Artif. Intell. Rev., vol. 55, no. 8, Dec. 2022, p. 5981–6035.
  13. C. Dong, C. C. Loy, and X. Tang, “Accelerating the Super-Resolution Convolutional Neural Network,” in Computer Vision – ECCV 2016.   Cham: Springer International Publishing, 2016, pp. 391–407.
  14. W. Shi et al., “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2016, pp. 1874–1883.
  15. B. Lim et al., “Enhanced deep residual networks for single image super-resolution,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Jul. 2017.
  16. S. Liu et al., “EVSRNet: Efficient Video Super-Resolution with Neural Architecture Search,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021, pp. 2480–2485.
  17. C. Ledig et al., “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 105–114.
  18. N. Ahn, B. Kang, and K.-A. Sohn, “Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network,” in Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part X, 2018, p. 256–272.
  19. M. Haris, G. Shakhnarovich, and N. Ukita, “Recurrent Back-Projection Network for Video Super-Resolution,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2019, pp. 3892–3901.
  20. Z. Li et al., “VMAF: The journey continues,” in Netflix Technology Blog, vol. 25, 2018.
  21. H. Amirpour et al., “VCD: Video Complexity Dataset,” in Proceedings of the 13th ACM Multimedia Systems Conference, 2022, p. 234–239.
  22. V. V. Menon et al., “JND-aware Two-pass Per-title Encoding Scheme for Adaptive Live Streaming,” in IEEE Transactions on Circuits and Systems for Video Technology, 2023, pp. 1–1.
  23. V. V. Menon et al., “Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming,” in First International ACM Green Multimedia Systems Workshop (GMSys ’23), 2023.
  24. Q. Cai et al., “Real-Time Constant Objective Quality Video Coding Strategy in High Efficiency Video Coding,” in IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 7, 2020, pp. 2215–2228.
  25. J. Y. Lin et al., “Experimental design and analysis of JND test on coded image/video,” in Applications of Digital Image Processing XXXVIII, vol. 9599, 2015, pp. 324–334.
  26. H. Wang et al., “VideoSet: A large-scale compressed video quality dataset based on JND measurement,” in Journal of Visual Communication and Image Representation, vol. 46, 2017, pp. 292–302.
  27. J. Zhu et al., “A Framework to Map VMAF with the Probability of Just Noticeable Difference between Video Encoding Recipes,” in 2022 IEEE 14th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), 2022, pp. 1–5.
  28. V. V. Menon et al., “Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming,” in 2023 IEEE International Conference on Visual Communications and Image Processing (VCIP), 2023.
  29. VideoLAN, “x265.” [Online]. Available: https://www.videolan.org/developers/x265.html
  30. P. K. Tiwari et al., “Accelerating x265 with Intel® Advanced Vector Extensions 512,” in White Paper on the Intel Developers Page, 2018.
  31. HSTP-VID-WPOM, “Working practices using objective metrics for evaluation of video coding efficiency experiments,” in International Telecommunication Union, 2020.
  32. V. V. Menon et al., “EMES: Efficient Multi-Encoding Schemes for HEVC-Based Adaptive Bitrate Streaming,” in ACM Trans. Multimedia Comput. Commun. Appl., vol. 19, no. 3s, Mar. 2023.
  33. BCG-GAMMA and MILA, “CodeCarbon.” [Online]. Available: https://codecarbon.io/
  34. R. Farahani et al., “Hybrid P2P-CDN architecture for live video streaming: An online learning approach,” in GLOBECOM 2022-2022 IEEE Global Communications Conference, 2022, pp. 1911–1917.
  35. B. Bross et al., “Overview of the Versatile Video Coding (VVC) Standard and its Applications,” vol. 31, no. 10, 2021, pp. 3736–3764.
  36. A. Bentaleb et al., “Common Media Client Data (CMCD): Initial Findings,” in Proceedings of the 31st ACM Workshop on Network and Operating Systems Support for Digital Audio and Video, Jul. 2021, p. 25–33.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now