Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Progressive Frame Patching for FoV-based Point Cloud Video Streaming (2303.08336v3)

Published 15 Mar 2023 in cs.MM and eess.IV

Abstract: Many XR applications require the delivery of volumetric video to users with six degrees of freedom (6-DoF) movements. Point Cloud has become a popular volumetric video format. A dense point cloud consumes much higher bandwidth than a 2D/360 degree video frame. User Field of View (FoV) is more dynamic with 6-DoF movement than 3-DoF movement. To save bandwidth, FoV-adaptive streaming predicts a user's FoV and only downloads point cloud data falling in the predicted FoV. However, it is vulnerable to FoV prediction errors, which can be significant when a long buffer is utilized for smoothed streaming. In this work, we propose a multi-round progressive refinement framework for point cloud video streaming. Instead of sequentially downloading point cloud frames, our solution simultaneously downloads/patches multiple frames falling into a sliding time-window, leveraging the inherent scalability of octree-based point-cloud coding. The optimal rate allocation among all tiles of active frames are solved analytically using the heterogeneous tile rate-quality functions calibrated by the predicted user FoV. Multi-frame downloading/patching simultaneously takes advantage of the streaming smoothness resulting from long buffer and the FoV prediction accuracy at short buffer length. We evaluate our streaming solution using simulations driven by real point cloud videos, real bandwidth traces, and 6-DoF FoV traces of real users. Our solution is robust against the bandwidth/FoV prediction errors, and can deliver high and smooth view quality in the face of bandwidth variations and dynamic user and point cloud movements.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. E. Cuervo, K. Chintalapudi, and M. Kotaru, “Creating the perfect illusion: What will it take to create life-like virtual reality headsets?” in Proceedings of the 19th International Workshop on Mobile Computing Systems & Applications, 2018, pp. 7–12.
  2. Z. Tu, T. Zong, X. Xi, L. Ai, Y. Jin, X. Zeng, and Y. Fan, “Content adaptive tiling method based on user access preference for streaming panoramic video,” in 2018 IEEE International Conference on Consumer Electronics (ICCE).   IEEE, 2018, pp. 1–4.
  3. Y. Zhou, L. Tian, C. Zhu, X. Jin, and Y. Sun, “Video coding optimization for virtual reality 360-degree source,” IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 1, pp. 118–129, 2020.
  4. M. Budagavi, J. Furton, G. Jin, A. Saxena, J. Wilkinson, and A. Dickerson, “360 degrees video coding using region adaptive smoothing,” in 2015 IEEE International Conference on Image Processing (ICIP), 2015, pp. 750–754.
  5. J. Zhou, N. Li, Y. Liu, S. Yao, and S. Chen, “Exploring spherical autoencoder for spherical video content processing,” in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 3115–3123.
  6. L. Hsiao, B. Krajancich, P. Levis, G. Wetzstein, and K. Winstein, “Towards retina-quality vr video streaming: 15ms could save you 80% of your bandwidth,” ACM SIGCOMM Computer Communication Review, vol. 52, no. 1, pp. 10–19, 2022.
  7. T. Zong, Z. Tu, L. Ai, X. Xi, Y. Jin, X. Zeng, and Y. Fan, “Panoramic video delivery based on laplace compensation and sphere-markov probability model,” in 2018 IEEE International Conference on Consumer Electronics (ICCE).   IEEE, 2018, pp. 1–4.
  8. S. Afzal, J. Chen, and K. Ramakrishnan, “Characterization of 360-degree videos,” in Proceedings of the Workshop on Virtual Reality and Augmented Reality Network, 2017, pp. 1–6.
  9. M. Zink, R. Sitaraman, and K. Nahrstedt, “Scalable 360 video stream delivery: Challenges, solutions, and opportunities,” Proceedings of the IEEE, vol. 107, no. 4, pp. 639–650, 2019.
  10. Y. Guan, C. Zheng, X. Zhang, Z. Guo, and J. Jiang, “Pano: Optimizing 360 video streaming with a better understanding of quality perception,” in Proceedings of the ACM Special Interest Group on Data Communication, 2019, pp. 394–407.
  11. J. He, M. A. Qureshi, L. Qiu, J. Li, F. Li, and L. Han, “Rubiks: Practical 360-degree streaming for smartphones,” in Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services, 2018, pp. 482–494.
  12. M. Palash, V. Popescu, A. Sheoran, and S. Fahmy, “Robust 360° video streaming via non-linear sampling,” in IEEE INFOCOM 2021 - IEEE Conference on Computer Communications, 2021.
  13. F. Duanmu, E. Kurdoglu, Y. Liu, and Y. Wang, “View direction and bandwidth adaptive 360 degree video streaming using a two-tier system,” in 2017 IEEE International Symposium on Circuits and Systems (ISCAS), May 2017, pp. 1–4.
  14. F. Duanmu, E. Kurdoglu, S. A. Hosseini, Y. Liu, and Y. Wang, “Prioritized buffer control in two-tier 360 video streaming,” in Proceedings of the Workshop on Virtual Reality and Augmented Reality Network, ser. VR/AR Network ’17.   New York, NY, USA: ACM, 2017, pp. 13–18. [Online]. Available: http://doi.acm.org/10.1145/3097895.3097898
  15. L. Sun, F. Duanmu, Y. Liu, Y. Wang, Y. Ye, H. Shi, and D. Dai, “Multi-path multi-tier 360-degree video streaming in 5g networks,” in Proceedings of the ACM Multimedia Systems Conference, 2018.
  16. ——, “A two-tier system for on-demand streaming of 360 degree video over dynamic networks,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 9, no. 1, pp. 43–57, 2019.
  17. Y. Mao, L. Sun, Y. Liu, and Y. Wang, “Low-latency fov-adaptive coding and streaming for interactive 360° video streaming,” in Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 3696–3704.
  18. L. Sun, Y. Mao, T. Zong, Y. Liu, and Y. Wang, “Flocking-based live streaming of 360-degree video,” in Proceedings of the 11th ACM Multimedia Systems Conference, 2020, pp. 26–37.
  19. ——, “Live 360 Degree Video Delivery based on User Collaboration in a Streaming Flock,” IEEE Transactions on Multimedia, February 2022.
  20. N. Jiang, Y. Liu, T. Guo, W. Xu, V. Swaminathan, L. Xu, and S. Wei, “Qurate: Power-efficient mobile immersive video streaming,” in Proceedings of the 11th ACM Multimedia Systems Conference, 2020, pp. 99–111.
  21. J. Chakareski, X. Corbillon, G. Simon, and V. Swaminathan, “User navigation modeling, rate-distortion analysis, and end-to-end optimization for viewport-driven 360 degree video streaming,” IEEE Transactions on Multimedia, pp. 1–16, 2022.
  22. S. Wang, S. Yang, H. Li, X. Zhang, C. Zhou, C. Xu, F. Qian, N. Wang, and Z. Xu, “Salientvr: saliency-driven mobile 360-degree video streaming with gaze information,” in Proceedings of the 28th Annual International Conference on Mobile Computing And Networking, 2022, pp. 542–555.
  23. Y. Bao, H. Wu, T. Zhang, A. A. Ramli, and X. Liu, “Shooting a moving target: Motion-prediction-based transmission for 360-degree videos,” in Big Data (Big Data), 2016 IEEE International Conference on.   IEEE, 2016, pp. 1161–1170.
  24. A. Nguyen, Z. Yan, and K. Nahrstedt, “Your attention is unique: Detecting 360-degree video saliency in head-mounted display for head movement prediction,” in MM 2018 - Proceedings of the 2018 ACM Multimedia Conference, Oct. 2018, pp. 1190–1198.
  25. J. Yu and Y. Liu, “Field-of-view prediction in 360-degree videos with attention-based neural encoder-decoder networks,” in Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, 2019, pp. 37–42.
  26. J. Park, M. Wu, K.-Y. Lee, B. Chen, K. Nahrstedt, M. Zink, and R. Sitaraman, “Seaware: Semantic aware view prediction system for 360-degree video streaming,” in 2020 IEEE International Symposium on Multimedia (ISM).   IEEE, 2020, pp. 57–64.
  27. C. Li, W. Zhang, Y. Liu, and Y. Wang, “Very long term field of view prediction for 360-degree video streaming,” in 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR).   IEEE, 2019, pp. 297–302.
  28. J. Dai, Z. Zhang, S. Mao, and D. Liu, “A view synthesis-based 360° vr caching system over mec-enabled c-ran,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 10, pp. 3843–3855, 2019.
  29. J. Liu, G. Simon, X. Corbillon, J. Chakareski, and Q. Yang, “Delivering viewport-adaptive 360-degree videos in cache-aided mec networks,” in 2020 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB).   IEEE, 2020, pp. 1–6.
  30. A. Sarkar, J. Murray, M. Dasari, M. Zink, and K. Nahrstedt, “L3bou: Low latency, low bandwidth, optimized super-resolution backhaul for 360-degree video streaming,” in 2021 IEEE International Symposium on Multimedia (ISM).   IEEE, 2021, pp. 138–147.
  31. C. Li, T. Ye, T. Zong, L. Sun, H. Cao, and Y. Liu, “Coffee: Cost-effective edge caching for 360 degree live video streaming,” arXiv preprint arXiv:2312.13470, 2023.
  32. R. Chen, M. Xiao, D. Yu, G. Zhang, and Y. Liu, “patchvvc: A real-time compression framework for streaming volumetric videos,” in Proceedings of the 14th Conference on ACM Multimedia Systems, 2023, pp. 119–129.
  33. S. Wang, M. Zhu, N. Li, M. Xiao, and Y. Liu, “Vqba: Visual-quality-driven bit allocation for low-latency point cloud streaming,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 9143–9151.
  34. J. Zhang, T. Chen, D. Ding, and Z. Ma, “G-pcc++: Enhanced geometry-based point cloud compression,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 1352–1363.
  35. Y. Wang, D. Zhao, H. Zhang, C. Huang, T. Gao, Z. Guo, L. Pang, and H. Ma, “Hermes: Leveraging implicit inter-frame correlation for bandwidth-efficient mobile volumetric video streaming,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 9185–9193.
  36. J. Park, P. A. Chou, and J.-N. Hwang, “Rate-utility optimized streaming of volumetric media for augmented reality,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 9, no. 1, pp. 149–162, 2019.
  37. B. Han, Y. Liu, and F. Qian, “Vivo: Visibility-aware mobile volumetric video streaming,” in Proceedings of the 26th annual international conference on mobile computing and networking, 2020, pp. 1–13.
  38. K. Lee, J. Yi, Y. Lee, S. Choi, and Y. M. Kim, “Groot: A real-time streaming system of high-fidelity volumetric videos,” in Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, ser. MobiCom ’20.   New York, NY, USA: Association for Computing Machinery, 2020. [Online]. Available: https://doi.org/10.1145/3372224.3419214
  39. A. Zhang, C. Wang, B. Han, and F. Qian, “YuZu: Neural-Enhanced Volumetric Video Streaming,” in 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), 2022, pp. 137–154.
  40. Y. Liu, B. Han, F. Qian, A. Narayanan, and Z.-L. Zhang, “Vues: practical mobile volumetric video streaming through multiview transcoding,” in Proceedings of the 28th Annual International Conference on Mobile Computing And Networking, 2022, pp. 514–527.
  41. L. Wang, C. Li, W. Dai, J. Zou, and H. Xiong, “Qoe-driven and tile-based adaptive streaming for point clouds,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2021, pp. 1930–1934.
  42. L. Wang, C. Li, W. Dai, S. Li, J. Zou, and H. Xiong, “Qoe-driven adaptive streaming for point clouds,” IEEE Transactions on Multimedia, 2022.
  43. S. Rossi, I. Viola, L. Toni, and P. Cesar, “Extending 3-dof metrics to model user behaviour similarity in 6-dof immersive applications,” in Proceedings of the 14th Conference on ACM Multimedia Systems, 2023, pp. 39–50.
  44. S. Subramanyam, I. Viola, A. Hanjalic, and P. Cesar, “User centered adaptive streaming of dynamic point clouds with low complexity tiling,” in Proceedings of the 28th ACM international conference on multimedia, 2020, pp. 3669–3677.
  45. J. Kammerl, N. Blodow, R. B. Rusu, S. Gedikli, M. Beetz, and E. Steinbach, “Real-time compression of point cloud streams,” in 2012 IEEE international conference on robotics and automation.   IEEE, 2012, pp. 778–785.
  46. Y. Shi, P. Venkatram, Y. Ding, and W. T. Ooi, “Enabling low bit-rate mpeg v-pcc-encoded volumetric video streaming with 3d sub-sampling,” in Proceedings of the 14th Conference on ACM Multimedia Systems, 2023, pp. 108–118.
  47. R. Mekuria, K. Blom, and P. Cesar, “Design, implementation, and evaluation of a point cloud codec for tele-immersive video,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 27, no. 4, pp. 828–842, 2016.
  48. A. Zhang, C. Wang, B. Han, and F. Qian, “Efficient volumetric video streaming through super resolution,” in Proceedings of the 22nd International Workshop on Mobile Computing Systems and Applications, 2021, pp. 106–111.
  49. X. Sheng, L. Li, D. Liu, Z. Xiong, Z. Li, and F. Wu, “Deep-pcac: An end-to-end deep lossy compression framework for point cloud attributes,” IEEE Transactions on Multimedia, vol. 24, pp. 2617–2632, 2021.
  50. J. Wang, H. Zhu, H. Liu, and Z. Ma, “Lossy point cloud geometry compression via end-to-end learning,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 12, pp. 4909–4923, 2021.
  51. Y. Mao, Y. Hu, and Y. Wang, “Learning to predict on octree for scalable point cloud geometry coding,” in 2022 IEEE 5th International Conference on Multimedia Information Processing and Retrieval (MIPR).   IEEE, 2022, pp. 96–102.
  52. Z. Liu, J. Li, X. Chen, C. Wu, S. Ishihara, and Y. Ji, “Fuzzy logic-based adaptive point cloud video streaming,” IEEE Open Journal of the Computer Society, vol. 1, pp. 121–130, 2020.
  53. J. Van Der Hooft, T. Wauters, F. De Turck, C. Timmerer, and H. Hellwagner, “Towards 6dof http adaptive streaming through point cloud compression,” in Proceedings of the 27th ACM International Conference on Multimedia, 2019, pp. 2405–2413.
  54. Y. Gao, P. Zhou, Z. Liu, B. Han, and P. Hui, “Fras: Federated reinforcement learning empowered adaptive point cloud video streaming,” arXiv preprint arXiv:2207.07394, 2022.
  55. J. Li, C. Zhang, Z. Liu, R. Hong, and H. Hu, “Optimal volumetric video streaming with hybrid saliency based tiling,” IEEE Transactions on Multimedia, 2022.
  56. J. Li, H. Wang, Z. Liu, P. Zhou, X. Chen, Q. Li, and R. Hong, “Towards optimal real-time volumetric video streaming: A rolling optimization and deep reinforcement learning based approach,” IEEE Transactions on Circuits and Systems for Video Technology, 2023.
  57. M. Rudolph, S. Schneegass, and A. Rizk, “Rabbit: Live transcoding of v-pcc point cloud streams,” in Proceedings of the 14th Conference on ACM Multimedia Systems, 2023, pp. 97–107.
  58. J. Zhang, T. Chen, D. Ding, and Z. Ma, “Yoga: Yet another geometry-based point cloud compressor,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 9070–9081.
  59. D. Graziosi, O. Nakagami, S. Kuma, A. Zaghetto, T. Suzuki, and A. Tabatabai, “An overview of ongoing point cloud compression standardization activities: Video-based (v-pcc) and geometry-based (g-pcc),” APSIPA Transactions on Signal and Information Processing, vol. 9, 2020.
  60. Google. (2017) Draco 3d data compression. [Online]. Available: https://google.github.io/draco/
  61. M. Hosseini and C. Timmerer, “Dynamic adaptive point cloud streaming,” in Proceedings of the 23rd Packet Video Workshop, 2018, pp. 25–30.
  62. J. H. Westerink and J. A. Roufs, “Subjective image quality as a function of viewing distance, resolution, and picture size,” SMPTE journal, vol. 98, no. 2, pp. 113–119, 1989.
  63. wikipedia, “visual acuity,” 2023. [Online]. Available: https://en.wikipedia.org/wiki/Visual_acuity
  64. E. d’Eon, B. Harrison, T. Myers, and P. A. Chou, “8i voxelized full bodies-a voxelized point cloud dataset,” ISO/IEC JTC1/SC29 Joint WG11/WG1 (MPEG/JPEG) input document WG11M40059/WG1M74006, vol. 7, no. 8, p. 11, 2017.
  65. L. Mei, R. Hu, H. Cao, Y. Liu, Z. Han, F. Li, and J. Li, “Realtime mobile bandwidth prediction using lstm neural network,” in Passive and Active Measurement: 20th International Conference, PAM 2019, Puerto Varas, Chile, March 27–29, 2019, Proceedings 20.   Springer, 2019, pp. 34–47.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now