Transformer Masked Autoencoders for Next-Generation Wireless Communications: Architecture and Opportunities (2401.06274v1)
Abstract: Next-generation communication networks are expected to exploit recent advances in data science and cutting-edge communications technologies to improve the utilization of the available communications resources. In this article, we introduce an emerging deep learning (DL) architecture, the transformer-masked autoencoder (TMAE), and discuss its potential in next-generation wireless networks. We discuss the limitations of current DL techniques in meeting the requirements of 5G and beyond 5G networks, and how the TMAE differs from the classical DL techniques can potentially address several wireless communication problems. We highlight various areas in next-generation mobile networks which can be addressed using a TMAE, including source and channel coding, estimation, and security. Furthermore, we demonstrate a case study showing how a TMAE can improve data compression performance and complexity compared to existing schemes. Finally, we discuss key challenges and open future research directions for deploying the TMAE in intelligent next-generation mobile networks.
- Y. Xu, G. Gui, H. Gacanin, and F. Adachi, “A survey on resource allocation for 5G heterogeneous networks: Current research, future trends, and challenges,” IEEE Commun. Surveys Tuts, vol. 23, no. 2, pp. 668–695, 2021.
- Y. Sun, M. Peng, Y. Zhou, Y. Huang, and S. Mao, “Application of machine learning in wireless networks: Key techniques and open issues,” IEEE Commun. Surveys Tuts, vol. 21, no. 4, pp. 3072–3108, 2019.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc. 31st Int. Conf. Neural Info. Process. Sys., ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 6000–6010.
- X. Chen, Y. Hu, Z. Dong, P. Zheng, and J. Wei, “Transformer operating state monitoring system based on wireless sensor networks,” IEEE Sensors J., vol. 21, no. 22, pp. 25 098–25 105, 2021.
- Z. Qin, H. Ye, G. Y. Li, and B.-H. F. Juang, “Deep learning in physical layer communications,” IEEE Wireless Commun., vol. 26, no. 2, pp. 93–99, 2019.
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in Int. Conf. Learning Representations, virtual, 2021.
- K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in 2022 IEEE/CVF Conf. Computer Vision and Pattern Recognition, 2022, pp. 15 979–15 988.
- M. Li, P. Ling, S. Wen, X. Chen, and F. Wen, “Bubble-wave-mitigation algorithm and transformer-based neural network demodulator for water-air optical camera communications,” IEEE Photonics J., pp. 1–10, 2023.
- Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, 2004.
- Kodak, “Kodak lossless true color image suite,” http://r0k.us/graphics/kodak/, 1999.
- D. Minnen, J. Ballé, and G. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” in Annual Conf. Neural Inf. Processing Systems (NeurIPS), 3-8 December 2018, Montréal, Canada, 2018, pp. 10 794–10 803.
- Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” in Proceedings of the IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2020.
- G. Toderici, D. Vincent, N. Johnston, S. J. Hwang, D. Minnen, J. Shor, and M. Covell, “Full resolution image compression with recurrent neural networks,” CoRR, vol. abs/1608.05148, 2016. [Online]. Available: http://arxiv.org/abs/1608.05148
- Y. Jiang, H. Kim, H. Asnani, S. Kannan, S. Oh, and P. Viswanath, “Joint channel coding and modulation via deep learning,” in IEEE 21st Int. Workshop Signal Process. Advances Wireless Commun. (SPAWC), Aug. 2020, pp. 1–5.
- M. Soltani, V. Pourahmadi, A. Mirzaei, and H. Sheikhzadeh, “Deep learning-based channel estimation,” IEEE Commun. Lett., vol. 23, no. 4, pp. 652–655, 2019.