Papers
Topics
Authors
Recent
Search
2000 character limit reached

Viewport Prediction, Bitrate Selection, and Beamforming Design for THz-Enabled 360° Video Streaming

Published 23 Jan 2024 in eess.IV and eess.SP | (2401.13114v2)

Abstract: 360{\deg} videos require significant bandwidth to provide an immersive viewing experience. Wireless systems using terahertz (THz) frequency band can meet this high data rate demand. However, self-blockage is a challenge in such systems. To ensure reliable transmission, this paper explores THz-enabled 360{\deg} video streaming through multiple multi-antenna access points (APs). Guaranteeing users' quality of experience (QoE) requires accurate viewport prediction to determine which video tiles to send, followed by asynchronous bitrate selection for those tiles and beamforming design at the APs. To address users' privacy and data heterogeneity, we propose a content-based viewport prediction framework, wherein users' head movement prediction models are trained using a personalized federated learning (PFL) algorithm. To address asynchronous decision-making for tile bitrates and dynamic THz link connections, we formulate the optimization of bitrate selection and beamforming as a macro-action decentralized partially observable Markov decision process (MacDec-POMDP) problem. To efficiently tackle this problem for multiple users, we develop two deep reinforcement learning (DRL) algorithms based on multi-agent actor-critic methods and propose a hierarchical learning framework to train the actor and critic networks. Experimental results show that our proposed approach provides a higher QoE when compared with three benchmark algorithms.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. M. Setayesh and V. W.S. Wong, “A content-based viewport prediction framework for 360∘{}^{\circ}start_FLOATSUPERSCRIPT ∘ end_FLOATSUPERSCRIPT video using personalized federated learning and fusion techniques,” in Proc. of Int’l Conf. on Multimedia and Expo (ICME), Brisbane, Australia, Jul. 2023.
  2. ——, “Asynchronous DRL-based bitrate selection for 360-degree video streaming over THz wireless systems,” accepted for publication in IEEE Int’l Conf. Commun. (ICC), Denver, CO, Jun. 2024.
  3. R. Zhang, J. Liu, F. Liu, T. Huang, Q. Tang, S. Wang, and F. R. Yu, “Buffer-aware virtual reality video streaming with personalized and private viewport prediction,” IEEE J. Sel. Areas in Commun., vol. 40, no. 2, pp. 694–709, Feb. 2022.
  4. C. Chaccour, M. N. Soorki, W. Saad, M. Bennis, and P. Popovski, “Can terahertz provide high-rate reliable low-latency communications for wireless VR?” IEEE Internet Things J., vol. 9, no. 12, pp. 9712–9729, Jun. 2022.
  5. A. Shafie, N. Yang, S. Durrani, X. Zhou, C. Han, and M. Juntti, “Coverage analysis for 3D terahertz communication systems,” IEEE J. Sel. Areas Commun., vol. 39, no. 6, pp. 1817–1832, Jun. 2021.
  6. A. Shafie, N. Yang, S. A. Alvi, C. Han, S. Durrani, and J. M. Jornet, “Spectrum allocation with adaptive sub-band bandwidth for terahertz communication systems,” IEEE Trans. Commun., vol. 70, no. 2, pp. 1407–1422, Feb. 2022.
  7. M. S. Elbamby, C. Perfecto, M. Bennis, and K. Doppler, “Toward low-latency and ultra-reliable virtual reality,” IEEE Netw., vol. 32, no. 2, pp. 78–84, Mar.-Apr. 2018.
  8. N. Kan, J. Zou, C. Li, W. Dai, and H. Xiong, “RAPT360: Reinforcement learning-based rate adaptation for 360-degree video streaming with adaptive prediction and tiling,” IEEE Trans. Circuits Syst. Video Technol., vol. 32, no. 3, pp. 1607–1623, Mar. 2022.
  9. A. Yaqoob and G.-M. Muntean, “A combined field-of-view prediction-assisted viewport adaptive delivery scheme for 360∘{}^{\circ}start_FLOATSUPERSCRIPT ∘ end_FLOATSUPERSCRIPT videos,” IEEE Trans. Broadcast., vol. 67, no. 3, pp. 746–760, Sept. 2021.
  10. F. Qian, B. Han, Q. Xiao, and V. Gopalakrishnan, “Flare: Practical viewport-adaptive 360-degree video streaming for mobile devices,” in Proc. of ACM Int’l Conf. Mobile Comput. Netw. (MobiCom), New Delhi, India, Oct. 2018.
  11. P. Maniotis, E. Bourtsoulatze, and N. Thomos, “Tile-based joint caching and delivery of 360 videos in heterogeneous networks,” IEEE Trans. Multimedia, vol. 22, no. 9, pp. 2382–2395, Sept. 2020.
  12. P. Yang, T. Q. Quek, J. Chen, C. You, and X. Cao, “Feeling of presence maximization: mmWave-enabled virtual reality meets deep reinforcement learning,” IEEE Trans. Wireless Commun., vol. 21, no. 11, pp. 10 005–10 019, Nov. 2022.
  13. L. Zhao, Y. Cui, S. Yang, and S. S. Shitz, “An optimization framework for general rate splitting for general multicast,” IEEE Trans. Wireless Commun., vol. 22, no. 3, pp. 1573–1587, Mar. 2022.
  14. R. Huang, V. W.S. Wong, and R. Schober, “Rate-splitting for intelligent reflecting surface-aided multiuser VR streaming,” IEEE J. Sel. Areas Commun., vol. 41, no. 5, pp. 1516–1535, May 2023.
  15. F.-Y. Chao, C. Ozcinar, and A. Smolic, “Transformer-based long-term viewport prediction in 360∘{}^{\circ}start_FLOATSUPERSCRIPT ∘ end_FLOATSUPERSCRIPT video: Scanpath is all you need.” in Proc. of Int’l Workshop Multimedia Signal Process. (MMSP), Tampere, Finland, Oct. 2021.
  16. C. Perfecto, M. S. Elbamby, J. Del Ser, and M. Bennis, “Taming the latency in multi-user VR 360∘{}^{\circ}start_FLOATSUPERSCRIPT ∘ end_FLOATSUPERSCRIPT: A QoE-aware deep learning-aided multicast framework,” IEEE Trans. Commun., vol. 68, no. 4, pp. 2491–2508, Apr. 2020.
  17. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Proc. of Int’l Conf. Artificial Intelligence and Statistics (AISTATS), Ft. Lauderdale, FL, Apr. 2017.
  18. X. Liu, Y. Deng, C. Han, and M. Di Renzo, “Learning-based prediction, rendering and transmission for interactive virtual reality in RIS-assisted terahertz networks,” IEEE J. Sel. Areas in Commun., vol. 40, no. 2, pp. 710–724, Feb. 2022.
  19. J. Oh, S. Kim, and S.-Y. Yun, “FedBABU: Towards enhanced representation for federated image classification,” in Proc. of Int’l Conf. on Learning Representations (ICLR), Apr. 2022.
  20. A. Nguyen, Z. Yan, and K. Nahrstedt, “Your attention is unique: Detecting 360-degree video saliency in head-mounted display for head movement prediction,” in Proc. of ACM Int’l Conf. on Multimedia, Seoul, Republic of Korea, Oct. 2018.
  21. Y. Xiao, W. Tan, and C. Amato, “Asynchronous actor-critic for multi-agent reinforcement learning,” in Proc. of Conf. Neural Inf. Process. Syst. (NIPS), New Orleans, LA, Nov. 2022.
  22. X. Lyu, A. Banitalebi-Dehkordi, M. Chen, and Y. Zhang, “Asynchronous, option-based multi-agent policy gradient: A conditional reasoning approach,” in Proc. of IEEE/RSJ Int’l Conf. on Intell. Robots Syst. (IROS), Detroit, MI, Oct. 2023.
  23. Z. Zhang, Y. Xu, J. Yu, and S. Gao, “Saliency detection in 360∘{}^{\circ}start_FLOATSUPERSCRIPT ∘ end_FLOATSUPERSCRIPT videos,” in Proc. of Eur. Conf. on Comput. Vis., Munich, Germany, Sept. 2018.
  24. J. M. Jornet and I. F. Akyildiz, “Channel modeling and capacity analysis for electromagnetic wireless nanonetworks in the terahertz band,” IEEE Trans. Wireless Commun., vol. 10, no. 10, pp. 3211–3221, Oct. 2011.
  25. H. Zhang, Y. Duan, K. Long, and V. C. M. Leung, “Energy efficient resource allocation in terahertz downlink NOMA systems,” IEEE Trans. Commun., vol. 69, no. 2, pp. 1375–1384, Feb. 2021.
  26. M. Hu, X. Luo, J. Chen, Y. C. Lee, Y. Zhou, and D. Wu, “Virtual reality: A survey of enabling technologies and its applications in IoT,” J. Netw. Comput. Appl., vol. 178, p. 102970, Mar. 2021.
  27. Q. Shi, M. Razaviyayn, Z.-Q. Luo, and C. He, “An iteratively weighted MMSE approach to distributed sum-utility maximization for a MIMO interfering broadcast channel,” IEEE Trans. Signal Process., vol. 59, no. 9, pp. 4331–4340, Sept. 2011.
  28. Z. Zhang, M. Zeng, M. Chen, D. Liu, W. Saad, S. Cui, and H. V. Poor, “Joint user grouping, version selection, and bandwidth allocation for live video multicasting,” IEEE Trans. Commun., vol. 70, no. 1, pp. 350–365, Jan. 2022.
  29. K. Long, Y. Cui, C. Ye, and Z. Liu, “Optimal wireless streaming of multi-quality 360 VR video by exploiting natural, relative smoothness-enabled, and transcoding-enabled multicast opportunities,” IEEE Trans. Multimedia, vol. 23, pp. 3670–3683, Oct. 2021.
  30. B. Han, Z. Ren, Z. Wu, Y. Zhou, and J. Peng, “Off-policy reinforcement learning with delayed rewards,” in Proc. of Int’l Conf. on Machine Learning (ICML), Baltimore, MD, Jul. 2022.
  31. C. Yu, X. Yang, J. Gao, J. Chen, Y. Li, J. Liu, Y. Xiang, R. Huang, H. Yang, Y. Wu, and Y. Wang, “Asynchronous multi-agent reinforcement learning for efficient real-time multi-robot cooperative exploration,” in Proc. of Int’l Conf. on Auton. Agents and Multiagent Syst. (AAMAS), London, United Kingdom, May 2023.
  32. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in Proc. of Int’l Conf. on Learning Representations (ICLR), San Juan, Puerto Rico, May 2016.
  33. T. D. Kulkarni, K. Narasimhan, A. Saeedi, and J. Tenenbaum, “Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation,” in Proc. of Conf. Neural Inf. Process. Syst. (NIPS), Barcelona, Spain, Dec. 2016.
  34. H. Yun, S. Lee, and G. Kim, “Panoramic vision transformer for saliency detection in 360∘{}^{\circ}start_FLOATSUPERSCRIPT ∘ end_FLOATSUPERSCRIPT videos,” in Proc. of Eur. Conf. on Comput. Vis. (ECCV), Tel Aviv, Israel, Oct. 2022.
  35. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. of Int’l Conf. Learning Representations (ICLR), San Diego, CA, May 2015.
  36. M. Setayesh, X. Li, and V. W.S. Wong, “PerFedMask: Personalized federated learning with optimized masking vectors,” in Proc. of Int’l Conf. on Learning Representations (ICLR), Kigali, Rwanda, May 2023.
  37. L. Teng, G. Zhai, Y. Wu, X. Min, W. Zhang, Z. Ding, and C. Xiao, “QoE driven VR 360∘{}^{\circ}start_FLOATSUPERSCRIPT ∘ end_FLOATSUPERSCRIPT video massive MIMO transmission,” IEEE Trans. Wireless Commun., vol. 21, no. 1, pp. 18–33, Jan. 2022.
  38. R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” in Proc. of Conf. Neural Inf. Process. Syst. (NIPS), Long Beach, CA, Dec. 2017.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up