Wireless Multi-User Interactive Virtual Reality in Metaverse with Edge-Device Collaborative Computing (2407.20523v1)
Abstract: The immersive nature of the metaverse presents significant challenges for wireless multi-user interactive virtual reality (VR), such as ultra-low latency, high throughput and intensive computing, which place substantial demands on the wireless bandwidth and rendering resources of mobile edge computing (MEC). In this paper, we propose a wireless multi-user interactive VR with edge-device collaborative computing framework to overcome the motion-to-photon (MTP) threshold bottleneck. Specifically, we model the serial-parallel task execution in queues within a foreground and background separation architecture. The rendering indices of background tiles within the prediction window are determined, and both the foreground and selected background tiles are loaded into respective processing queues based on the rendering locations. To minimize the age of sensor information and the power consumption of mobile devices, we optimize rendering decisions and MEC resource allocation subject to the MTP constraint. To address this optimization problem, we design a safe reinforcement learning (RL) algorithm, active queue management-constrained updated projection (AQM-CUP). AQM-CUP constructs an environment suitable for queues, incorporating expired tiles actively discarded in processing buffers into its state and reward system. Experimental results demonstrate that the proposed framework significantly enhances user immersion while reducing device power consumption, and the superiority of the proposed AQM-CUP algorithm over conventional methods in terms of the training convergence and performance metrics.
- C. Xu, Z. Chen, M. Tao, and W. Zhang, “Edge-device collaborative rendering for wireless multi-user interactive virtual reality in metaverse,” in Proc. IEEE Global Commun. Conf. (GLOBECOM), Dec. 2023, pp. 3542–3547.
- “Draft New Recommendation ITU-R M.[IMT.FRAMEWORK FOR 2030 AND BEYOND] - Framework and overall objectives of the future development of IMT for 2030 and beyond”, Document 5/131, June 2023. https://www.itu.int/md/R19-SG05-C-0131.
- W. Y. B. Lim, Z. Xiong, D. Niyato, X. Cao, C. Miao, S. Sun, and Q. Yang, “Realizing the metaverse with edge intelligence: A match made in heaven,” IEEE Wireless Commun., pp. 1–9, 2022.
- S. M. LaValle, A. Yershova, M. Katsev, and M. Antonov, “Head tracking for the oculus rift,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2014, pp. 187–194.
- “Meta Quest 3”, https://www.meta.com/quest/quest-3/.
- Huawei iLab, “Cloud VR Network Solution White Paper”, 2018. https://www.huawei.com/minisite/pdf/ilab/cloud_vr_network_solution_ white_paper_en.pdf.
- “Apple Vision Pro”, https://www.apple.com/apple-vision-pro/.
- Z. Gu, H. Lu, P. Hong, and Y. Zhang, “Reliability enhancement for vr delivery in mobile-edge empowered dual-connectivity sub-6 ghz and mmwave hetnets,” IEEE Trans. Wireless Commun., vol. 21, no. 4, pp. 2210–2226, 2022.
- S. Gupta, J. Chakareski, and P. Popovski, “mmwave networking and edge computing for scalable 360° video multi-user virtual reality,” IEEE Trans. Image Process., vol. 32, pp. 377–391, 2023.
- H. C. Ates, P. Q. Nguyen, L. Gonzalez-Macia et al., “End-to-end design of wearable sensors,” Nat. Rev. Mater., vol. 7, pp. 887–907, Nov. 2022.
- Z. Pan, Y. Zhang, T. Lin, and J. Yan, “Liveae: Attention-based and edge-assisted viewport prediction for live 360° video streaming,” in Proc. Workshop Emerg. Multimed. Syst. (EMS), 2023, pp. 28–33.
- X. Peng, S. Mao, and Z. Wu, “Trajectory-aware body interaction transformer for multi-person pose forecasting,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023, pp. 17 121–17 130.
- Y. Sun, Z. Chen, M. Tao, and H. Liu, “Communications, caching, and computing for mobile virtual reality: Modeling and tradeoff,” IEEE Trans. Commun., vol. 67, no. 11, pp. 7573–7586, 2019.
- L. Wang, L. Jiao, T. He, J. Li, and M. Mühlhäuser, “Service entity placement for social virtual reality applications in edge computing,” in Proc. IEEE Int. Conf. Comput. Commun., 2018, pp. 468–476.
- Y. Zhang, L. Jiao, J. Yan, and X. Lin, “Dynamic service placement for virtual reality group gaming on mobile edge cloudlets,” IEEE J. Sel. Areas Commun., vol. 37, no. 8, pp. 1881–1897, 2019.
- Z. Chen, H. Zhu, L. Song, D. He, and B. Xia, “Wireless multiplayer interactive virtual reality game systems with edge computing: Modeling and optimization,” IEEE Trans. Wireless Commun., vol. 21, no. 11, pp. 9684–9699, 2022.
- X. Wei, C. Yang, and S. Han, “Prediction, communication, and computing duration optimization for vr video streaming,” IEEE Trans. Commun., vol. 69, no. 3, pp. 1947–1959, 2021.
- L. Zhao, Y. Cui, Z. Liu, Y. Zhang, and S. Yang, “Adaptive streaming of 360 videos with perfect, imperfect, and unknown fov viewing probabilities in wireless networks,” IEEE Trans. Image Process., vol. 30, pp. 7744–7759, 2021.
- X. Liu, Y. Deng, C. Han, and M. D. Renzo, “Learning-based prediction, rendering and transmission for interactive virtual reality in ris-assisted terahertz networks,” IEEE J. Sel. Areas Commun., vol. 40, no. 2, pp. 710–724, 2022.
- C. Perfecto, M. S. Elbamby, J. D. Ser, and M. Bennis, “Taming the latency in multi-user vr 360°: A qoe-aware deep learning-aided multicast framework,” IEEE Trans. Commun., vol. 68, no. 4, pp. 2491–2508, 2020.
- C. Y. Chen and H. Y. Hsieh, “Cross-frame resource allocation with context-aware qoe estimation for 360° video streaming in wireless virtual reality,” IEEE Trans. Wireless Commun., vol. 22, no. 11, pp. 7887–7901, 2023.
- Z. Meng, C. She, G. Zhao, and D. De Martini, “Sampling, communication, and prediction co-design for synchronizing the real-world device and digital model in metaverse,” IEEE J. Sel. Areas Commun., vol. 41, no. 1, pp. 288–300, 2023.
- Z. Meng, K. Chen, Y. Diao, C. She, G. Zhao, M. A. Imran, and B. Vucetic, “Task-oriented cross-system design for timely and accurate modeling in the metaverse,” IEEE J. Sel. Areas Commun., vol. 42, no. 3, pp. 752–766, 2024.
- L. Huang and E. Modiano, “Optimizing age-of-information in a multi-class queueing system,” in Proc. IEEE Int. Symp. Inf. Theor. (ISIT), 2015, pp. 1681–1685.
- Z. Lai, Y. C. Hu, Y. Cui, L. Sun, N. Dai, and H.-S. Lee, “Furion: Engineering high-quality immersive virtual reality on today’s mobile devices,” IEEE Trans. Mobile Comput., vol. 19, no. 7, pp. 1586–1602, 2020.
- C. Xie, X. Li, Y. Hu, H. Peng, M. Taylor, and S. L. Song, “Q-vr: system-level design for future mobile collaborative virtual reality,” in Proc. ACM Int. Conf. Archit. Support Program. Lang. Oper. Syst. (ASPLOS), 2021, pp. 587–599.
- M. Segal and K. Akeley, “The opengl® graphics system: A specification (version 4.6),” 2022.
- V. Mnih, K. Kavukcuoglu, D. Silver et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, pp. 529–533, Feb. 2015.
- J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv:1707.06347, 2017.
- A. Ray et al., “Benchmarking Safe Exploration in Deep Reinforcement Learning,” OpenAI, 2019, https://openai.com/research/benchmarking-safe-exploration-in-deep-reinforcement-learning.
- J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” in Proc. Int. Conf. Mach. Learn., 2017, pp. 22–31.
- L. Yang, J. Ji, J. Dai, L. Zhang, B. Zhou, P. Li, Y. Yang, and G. Pan, “Constrained update projection approach to safe policy optimization,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 35. Curran Associates, Inc., 2022, pp. 9111–9124.
- J. M. P. van Waveren, “The asynchronous time warp for virtual reality on consumer hardware,” in Proc. ACM Symp. VR Softw. Technol. New York, USA: Association for Computing Machinery, 2016, pp. 37–46.
- 3GPP, “ETSI TR 138 901 V16.1.0: 5G; Study on channel model for frequencies from 0.5 to 100 GHz (3GPP TR 38.901 version 16.1.0 release 16)”, European Telecomm. Std. Institute (ETSI), 2020. https://www.etsi.org/deliver/etsi_tr/138900_138999/138901/16.01.00_60/ tr_138901v160100p.pdf.
- M. Di Renzo, “Stochastic geometry modeling and analysis of multi-tier millimeter wave cellular networks,” IEEE Trans. Wireless Commun., vol. 14, no. 9, pp. 5038–5057, 2015.
- T. A. Funkhouser and C. H. Sequin, “Adaptive display algorithm for interactive frame rates during visualization of complex virtual environments,” in Proc. ACM Spec. Int. Group Comp. Graph. Interact. Tech. Conf. (SIGGRAPH), Anaheim, CA, USA, 1993, pp. 247–254.
- M. Wimmer and P. Wonka, “Rendering time estimation for real-time rendering,” in Proc. Eurographics Rendering Workshop (EGRW), Goslar, DEU, 2003, pp. 118–129.
- E. Altman, “Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program,” Math. Method Oper. Res., vol. 48, no. 3, pp. 387–417, 1998.
- J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-dimensional continuous control using generalized advantage estimation,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2016.
- “Sketchfab”, https://sketchfab.com/feed.
- Caolu Xu (1 paper)
- Zhiyong Chen (101 papers)
- Meixia Tao (155 papers)
- Wenjun Zhang (160 papers)