Codebook-enabled Generative End-to-end Semantic Communication Powered by Transformer (2402.16868v2)
Abstract: Codebook-based generative semantic communication attracts increasing attention, since only indices are required to be transmitted when the codebook is shared between transmitter and receiver. However, due to the fact that the semantic relations among code vectors are not necessarily related to the distance of the corresponding code indices, the performance of the codebook-enabled semantic communication system is susceptible to the channel noise. Thus, how to improve the system robustness against the noise requires careful design. This paper proposes a robust codebook-assisted image semantic communication system, where semantic codec and codebook are first jointly constructed, and then vector-to-index transformer is designed guided by the codebook to eliminate the effects of channel noise, and achieve image generation. Thanks to the assistance of the high-quality codebook to the Transformer, the generated images at the receiver outperform those of the compared methods in terms of visual perception. In the end, numerical results and generated images demonstrate the advantages of the generative semantic communication method over JPEG+LDPC and traditional joint source channel coding (JSCC) methods.
- Holistic network virtualization and pervasive network intelligence for 6g. IEEE Communications Surveys & Tutorials, 24(1):1–30, 2022.
- Split learning over wireless networks: Parallel design and resource management. IEEE Journal on Selected Areas in Communications, 41(4):1051–1066, 2023.
- Joint task and data oriented semantic communications: A deep separate source-channel coding scheme. IEEE Internet of Things Journal, pages 1–1, 2023.
- Knowledge base enabled semantic communication: A generative perspective. arXiv preprint arXiv:2311.12443, 2023.
- Semantic knowledge base-enabled zero-shot multi-level feature transmission optimization. IEEE Transactions on Wireless Communications, 2023.
- Domain knowledge driven semantic communication for image transmission over wireless channels. IEEE Wireless Communications Letters, 12(1):55–59, 2022.
- Robust semantic communications with masked vq-vae enabled codebook. IEEE Transactions on Wireless Communications, 2023.
- Federated codebook for multi - user deep source coding. In 2022 13th International Conference on Information and Communication Technology Convergence (ICTC), pages 994–996, 2022.
- User association and power allocation for user-centric smart-duplex networks via tree-structured deep reinforcement learning. IEEE Internet of Things Journal, 10(22):20216–20229, June 2023.
- Attention is all you need. Advances in neural information processing systems, 30, 2017.
- Towards robust blind face restoration with codebook lookup transformer. Advances in Neural Information Processing Systems, 35:30599–30611, 2022.
- Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021.
- Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
- A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
- Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
- Deep joint source-channel coding for wireless image transmission. IEEE Transactions on Cognitive Communications and Networking, 5(3):567–579, 2019.