Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Embracing Radiance Field Rendering in 6G: Over-the-Air Training and Inference with 3D Contents (2405.12155v2)

Published 20 May 2024 in cs.IT and math.IT

Abstract: The efficient representation, transmission, and reconstruction of three-dimensional (3D) contents are becoming increasingly important for sixth-generation (6G) networks that aim to merge virtual and physical worlds for offering immersive communication experiences. Neural radiance field (NeRF) and 3D Gaussian splatting (3D-GS) have recently emerged as two promising 3D representation techniques based on radiance field rendering, which are able to provide photorealistic rendering results for complex scenes. Therefore, embracing NeRF and 3D-GS in 6G networks is envisioned to be a prominent solution to support emerging 3D applications with enhanced quality of experience. This paper provides a comprehensive overview on the integration of NeRF and 3D-GS in 6G. First, we review the basics of the radiance field rendering techniques, and highlight their applications and implementation challenges over wireless networks. Next, we consider the over-the-air training of NeRF and 3D-GS models over wireless networks by presenting various learning techniques. We particularly focus on the federated learning design over a hierarchical device-edge-cloud architecture, which is suitable for exploiting distributed data and computing resources over 6G networks to train large models representing large-scale scenes. Then, we consider the over-the-air rendering of NeRF and 3D-GS models at wireless network edge. We present three practical rendering architectures, namely local, remote, and co-rendering, respectively, and provide model compression approaches to facilitate the transmission of radiance field models for rendering. We also present rendering acceleration approaches and joint computation and communication designs to enhance the rendering efficiency. In a case study, we propose a new semantic communication enabled 3D content transmission design.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. G. Wu, Z. Lyu, J. Zhang, and J. Xu, “Semantic communications for 3D human face transmission with neural radiance fields,” submitted to IEEE Proc. International Symposium on Wireless Communication Systems (ISWCS), 2024.
  2. ITU-R WP5D, “Future technology trends of terrestrial international mobile telecommunications systems towards 2030 and beyond,” Nov. 2022. [Online]. Available: https://www.itu.int/pub/R-REP-M.2516
  3. G. Zhu, Z. Lyu, X. Jiao, P. Liu, M. Chen, J. Xu, S. Cui, and P. Zhang, “Pushing AI to wireless network edge: an overview on integrated sensing, communication, and computation towards 6G,” Sci. China Inf. Sci., vol. 66, no. 3, p. 130301, Feb. 2023.
  4. F. Liu, Y. Cui, C. Masouros, J. Xu, T. X. Han, Y. C. Eldar, and S. Buzzi, “Integrated sensing and communications: Toward dual-functional wireless networks for 6G and beyond,” IEEE J. Sel. Areas Commun., vol. 40, no. 6, pp. 1728–1767, Jun. 2022.
  5. X. S. Shen, J. Gao, M. Li, C. Zhou, S. Hu, M. He, and W. Zhuang, “Toward immersive communications in 6G,” Front. Comput. Sci., vol. 4, Jan. 2023.
  6. E. Ahmed, A. Saint, A. E. R. Shabayek, K. Cherenkova, R. Das, G. Gusev, D. Aouada, and B. Ottersten, “A survey on deep learning advances on different 3D data representations,” 2019. [Online]. Available: https://arxiv.org/pdf/1808.01462
  7. X. Li, Q. Zhang, D. Kang, W. Cheng, Y. Gao, J. Zhang, Z. Liang, J. Liao, Y.-P. Cao, and Y. Shan, “Advances in 3D generation: A survey,” 2024. [Online]. Available: https://arxiv.org/pdf/2401.17807
  8. D. Bommes, B. Lévy, N. Pietroni, E. Puppo, C. Silva, M. Tarini, and D. Zorin, “Quad-Mesh generation and processing: A survey,” Comput. Graph. Forum, vol. 32, no. 6, pp. 51–76, Mar. 2013.
  9. M. Aleksandrov, S. Zlatanova, and D. J. Heslop, “Voxelisation algorithms and data structures: A review,” Sensors, vol. 21, no. 24, Dec. 2021.
  10. R. B. Rusu and S. Cousins, “3D is here: Point cloud library (PCL),” in 2011 IEEE International Conference on Robotics and Automation, May 2011, pp. 1–4.
  11. B. R. de Araújo, D. S. Lopes, P. Jepp, J. A. Jorge, and B. Wyvill, “A survey on implicit surface polygonization,” ACM Comput. Surv., vol. 47, no. 4, May 2015.
  12. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing scenes as neural radiance fields for view synthesis,” Commun. ACM, vol. 65, no. 1, p. 99–106, Dec. 2021.
  13. B. Kerbl, G. Kopanas, T. Leimkuehler, and G. Drettakis, “3D Gaussian splatting for real-time radiance field rendering,” ACM Trans. Graph., vol. 42, no. 4, July 2023.
  14. K. Gao, Y. Gao, H. He, D. Lu, L. Xu, and J. Li, “NeRF: Neural radiance field in 3D vision, a comprehensive review,” 2023. [Online]. Available: https://arxiv.org/pdf/2210.00379.pdf
  15. A. Yaqoob, T. Bi, and G.-M. Muntean, “A survey on adaptive 360° video streaming: Solutions, challenges and opportunities,” IEEE Commun. Surv. Tutorials, vol. 22, no. 4, pp. 2801–2838, July, 2020.
  16. Y. Jin, K. Hu, J. Liu, F. Wang, and X. Liu, “From capture to display: A survey on volumetric video,” 2023. [Online]. Available: https://arxiv.org/pdf/2309.05658.pdf
  17. G. Chen and W. Wang, “A survey on 3D Gaussian splatting,” 2024. [Online]. Available: https://arxiv.org/pdf/2401.03890
  18. B. Jedari, G. Premsankar, G. Illahi, M. D. Francesco, A. Mehrabi, and A. Ylä-Jääski, “Video caching, analytics, and delivery at the wireless edge: A survey and future directions,” IEEE Commun. Surv. Tutorials, vol. 23, no. 1, pp. 431–471, 2021.
  19. R. Cheng, K. Liu, N. Wu, and B. Han, “Enriching telepresence with semantic-driven holographic communication,” in Proceedings of the 22nd ACM Workshop on Hot Topics in Networks, Nov. 2023, p. 147–156.
  20. C. Xu, B. Wu, J. Hou, S. Tsai, R. Li, J. Wang, W. Zhan, Z. He, P. Vajda, K. Keutzer, and M. Tomizuka, “NeRF-Det: Learning geometry-aware volumetric representation for multi-view 3D object detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023, pp. 23 320–23 330.
  21. F. Tosi, Y. Zhang, Z. Gong, E. Sandström, S. Mattoccia, M. R. Oswald, and M. Poggi, “How NeRFs and 3D Gaussian splatting are reshaping SLAM: A survey,” 2024. [Online]. Available: https://arxiv.org/pdf/2402.13255
  22. M. Adamkiewicz, T. Chen, A. Caccavale, R. Gardner, P. Culbertson, J. Bohg, and M. Schwager, “Vision-only robot navigation in a neural radiance world,” IEEE Rob. Autom. Lett., vol. 7, no. 2, pp. 4606–4613, Apr. 2022.
  23. Z. Lyu, G. Zhu, and J. Xu, “Joint maneuver and beamforming design for UAV-Enabled integrated sensing and communication,” IEEE Trans. Wireless Commun., vol. 22, no. 4, pp. 2424–2440, Apr. 2023.
  24. X. Zhao, Z. An, Q. Pan, and L. Yang, “NeRF2: Neural radio-frequency radiance fields,” Proceedings of the 29th Annual International Conference on Mobile Computing and Networking (MobiCom), 2023. [Online]. Available: https://doi.org/10.1145/3570361.3592527
  25. Z. Lei, F. Xu, J. Wei, F. Cai, F. Wang, and Y.-Q. Jin, “SAR-NeRF: Neural radiance fields for synthetic aperture radar multi-view representation,” 2023. [Online]. Available: https://arxiv.org/pdf/2307.05087
  26. Y. Zeng, J. Chen, J. Xu, D. Wu, X. Xu, S. Jin, X. Gao, D. Gesbert, S. Cui, and R. Zhang, “A tutorial on environment-aware communications via channel knowledge map for 6G,” IEEE Commun. Surv. Tutorials, early access.
  27. G. Charan, M. Alrabeiah, and A. Alkhateeb, “Vision-aided 6G wireless communications: Blockage prediction and proactive handoff,” IEEE Trans. Veh. Technol., vol. 70, no. 10, pp. 10 193–10 208, Oct. 2021.
  28. Y. Feng, F. Gao, X. Tao, S. Ma, and H. V. Poor, “Vision-aided ultra-reliable low-latency communications for smart factory,” IEEE Trans. Commun., early access.
  29. P. Hedman, P. P. Srinivasan, B. Mildenhall, J. T. Barron, and P. Debevec, “Baking neural radiance fields for real-time view synthesis,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 5875–5884.
  30. T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Trans. Graph., vol. 41, no. 4, July 2022.
  31. L. Wu, J. Y. Lee, A. Bhattad, Y.-X. Wang, and D. Forsyth, “Diver: Real-time and accurate neural radiance fields with deterministic integration for volume rendering,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 16 200–16 209.
  32. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, vol. 54, Apr. 2017, pp. 1273–1282.
  33. T. Suzuki, “Federated learning for large-scale scene modeling with neural radiance fields,” 2024. [Online]. Available: https://arxiv.org/pdf/2309.06030
  34. ——, “Fed3DGS: Scalable 3D Gaussian splatting with federated learning,” 2024. [Online]. Available: https://arxiv.org/pdf/2403.11460
  35. X. Cao, Z. Lyu, G. Zhu, J. Xu, L. Xu, and S. Cui, “An overview on over-the-air federated edge learning,” IEEE Wireless Commun., early access.
  36. L. Bottou, F. E. Curtis, and J. Nocedal, “Optimization methods for large-scale machine learning,” SIAM Rev., vol. 60, no. 2, pp. 223–311, 2018.
  37. E. Hoffer, I. Hubara, and D. Soudry, “Train longer, generalize better: closing the generalization gap in large batch training of neural networks,” in Advances in Neural Information Processing Systems, vol. 30, 2017.
  38. X. Pan, Z. Lai, S. Song, and G. Huang, “ActiveNeRF: Learning where to see with uncertainty estimation,” in Computer Vision – ECCV 2022, Nov. 2022, pp. 230–246.
  39. C. Xie, S. Koyejo, and I. Gupta, “Asynchronous federated optimization,” 2020. [Online]. Available: https://arxiv.org/pdf/1903.03934
  40. X. Yu, L. Cherkasova, H. Vardhan, Q. Zhao, E. Ekaireb, X. Zhang, A. Mazumdar, and T. Rosing, “Async-HFL: Efficient and robust asynchronous federated learning in hierarchical IoT networks,” in Proceedings of the 8th ACM/IEEE Conference on Internet of Things Design and Implementation, May 2023, p. 236–248.
  41. C. Xu, Y. Qu, Y. Xiang, and L. Gao, “Asynchronous federated learning on heterogeneous devices: A survey,” Comput. Sci. Rev., vol. 50, p. 100595, Nov. 2023.
  42. A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su, “MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 14 124–14 133.
  43. A. Yu, V. Ye, M. Tancik, and A. Kanazawa, “pixelNeRF: Neural radiance fields from one or few images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 4578–4587.
  44. Z. Lyu, Y. Li, G. Zhu, J. Xu, H. V. Poor, and S. Cui, “Rethinking resource management in edge learning: A joint pre-training and fine-tuning design paradigm,” 2024. [Online]. Available: https://arxiv.org/pdf/2404.00836
  45. Y. Liu, Y. Kang, T. Zou, Y. Pu, Y. He, X. Ye, Y. Ouyang, Y.-Q. Zhang, and Q. Yang, “Vertical federated learning: Concepts, advances, and challenges,” IEEE Trans. Knowl. Data Eng., early access.
  46. K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan, “Depth-supervised NeRF: Fewer views and faster training for free,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 12 882–12 891.
  47. G. Zhu, J. Xu, K. Huang, and S. Cui, “Over-the-Air computing for wireless data aggregation in massive IoT,” IEEE Wireless Commun., vol. 28, no. 4, pp. 57–65, Aug. 2021.
  48. J. Yan, S. Bi, and Y.-J. A. Zhang, “Optimal model placement and online model splitting for device-edge co-inference,” IEEE Trans. Wireless Commun., vol. 21, no. 10, pp. 8354–8367, Oct. 2022.
  49. Z. Fan, K. Wang, K. Wen, Z. Zhu, D. Xu, and Z. Wang, “LightGaussian: Unbounded 3D Gaussian compression with 15x reduction and 200+ fps,” 2024. [Online]. Available: https://arxiv.org/pdf/2311.17245
  50. K. Navaneet, K. P. Meibodi, S. A. Koohpayegani, and H. Pirsiavash, “Compact3D: Compressing Gaussian splat radiance field models with vector quantization,” 2023. [Online]. Available: https://arxiv.org/pdf/2311.18159
  51. J. Du, F. R. Yu, G. Lu, J. Wang, J. Jiang, and X. Chu, “MEC-assisted immersive VR video streaming over terahertz wireless networks: A deep reinforcement learning approach,” IEEE Internet Things J., vol. 7, no. 10, pp. 9517–9529, Oct. 2020.
  52. Y. Sun, Z. Chen, M. Tao, and H. Liu, “Communications, caching, and computing for mobile virtual reality: Modeling and tradeoff,” IEEE Trans. Commun., vol. 67, no. 11, pp. 7573–7586, Nov. 2019.
  53. C. Xu, Z. Chen, M. Tao, and W. Zhang, “Edge-device collaborative rendering for wireless multi-user interactive virtual reality in metaverse,” in GLOBECOM 2023 - 2023 IEEE Global Communications Conference, Dec. 2023, pp. 3542–3547.
  54. J. Ren, Z. Zhang, J. Xu, G. Chen, Y. Sun, P. Zhang, and S. Cui, “Knowledge base enabled semantic communication: A generative perspective,” 2023. [Online]. Available: https://arxiv.org/pdf/2311.12443.pdf
  55. Z. Lyu, G. Zhu, J. Xu, B. Ai, and S. Cui, “Semantic communications for image recovery and classification via deep joint source and channel coding,” IEEE Trans. Wireless Commun., early access.
  56. L. Yan, Z. Qin, R. Zhang, Y. Li, and G. Y. Li, “Resource allocation for text semantic communications,” IEEE Wireless Commun. Lett., vol. 11, no. 7, pp. 1394–1398, July 2022.
  57. Z. Wang, Y. Deng, and A. H. Aghvami, “Task-oriented and semantics-aware communication framework for avatar-centric augmented reality,” 2024. [Online]. Available: https://arxiv.org/pdf/2306.15470
  58. X. Gao, C. Zhong, J. Xiang, Y. Hong, Y. Guo, and J. Zhang, “Reconstructing personalized semantic facial NeRF models from monocular video,” ACM Trans. Graph., vol. 41, no. 6, Nov. 2022.
  59. C. Cao, Y. Weng, S. Zhou, Y. Tong, and K. Zhou, “FaceWarehouse: A 3D facial expression database for visual computing,” IEEE Trans. Visual Comput. Graphics, vol. 20, no. 3, pp. 413–425, Mar. 2014.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets