Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Agent-driven Generative Semantic Communication with Cross-Modality and Prediction (2404.06997v3)

Published 10 Apr 2024 in cs.NI and cs.LG

Abstract: In the era of 6G, with compelling visions of intelligent transportation systems and digital twins, remote surveillance is poised to become a ubiquitous practice. Substantial data volume and frequent updates present challenges in wireless networks. To address these challenges, we propose a novel agent-driven generative semantic communication (A-GSC) framework based on reinforcement learning. In contrast to the existing research on semantic communication (SemCom), which mainly focuses on either semantic extraction or semantic sampling, we seamlessly integrate both by jointly considering the intrinsic attributes of source information and the contextual information regarding the task. Notably, the introduction of generative artificial intelligence (GAI) enables the independent design of semantic encoders and decoders. In this work, we develop an agent-assisted semantic encoder with cross-modality capability, which can track the semantic changes, channel condition, to perform adaptive semantic extraction and sampling. Accordingly, we design a semantic decoder with both predictive and generative capabilities, consisting of two tailored modules. Moreover, the effectiveness of the designed models has been verified using the UA-DETRAC dataset, demonstrating the performance gains of the overall A-GSC framework in both energy saving and reconstruction accuracy.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. S. Iyer, R. Khanai, D. Torse, R. J. Pandya, K. M. Rabie, K. Pai, W. U. Khan, and Z. Fadlullah, “A survey on semantic communications for intelligent wireless networks,” Wireless Personal Communications, vol. 129, no. 1, pp. 569–611, 2023.
  2. Y. Zhang, F. Wang, W. Xu, and C. Liu, “Semantic communications: A new paradigm for networked intelligence,” in 2022 IEEE 32nd International Workshop on Machine Learning for Signal Processing (MLSP).   IEEE, 2022, pp. 1–6.
  3. C. E. Shannon, “A mathematical theory of communication,” The Bell system technical journal, vol. 27, no. 3, pp. 379–423, 1948.
  4. Z. Qin, X. Tao, J. Lu, W. Tong, and G. Y. Li, “Semantic communications: Principles and challenges,” arXiv preprint arXiv:2201.01389, 2021.
  5. R. D. Yates, Y. Sun, D. R. Brown, S. K. Kaul, E. Modiano, and S. Ulukus, “Age of information: An introduction and survey,” IEEE Journal on Selected Areas in Communications, vol. 39, no. 5, pp. 1183–1210, 2021.
  6. N. Pappas and M. Kountouris, “Goal-oriented communication for real-time tracking in autonomous systems,” in 2021 IEEE International Conference on Autonomous Systems (ICAS).   IEEE, 2021, pp. 1–5.
  7. W. Yang, Z. Xiong, Y. Yuan, and T. Q. Quek, “Semantic change driven generative semantic communication framework,” 2024 IEEE Wireless Communications and Networking Conference (WCNC), Accepted.
  8. Y. Wang, P.-M. Jodoin, F. Porikli, J. Konrad, Y. Benezeth, and P. Ishwar, “CDnet 2014: An expanded change detection benchmark dataset,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2014, pp. 387–394.
  9. E. Uysal, O. Kaya, A. Ephremides, J. Gross, M. Codreanu, P. Popovski, M. Assaad, G. Liva, A. Munari, B. Soret, T. Soleymani, and K. H. Johansson, “Semantic Communications in Networked Systems: A Data Significance Perspective,” IEEE Network, vol. 36, no. 4, pp. 233–240, 4 2022.
  10. Q. Chen, S. Guo, W. Xu, J. Li, K. Wei, Z. Cai, and H. Gao, “Peak AoI minimization with directional charging for data collection at wireless-powered network edge,” IEEE Transactions on Services Computing, pp. 1–14, 2023.
  11. F. Zhao, N. Pappas, C. Ma, X. Sun, T. Q. Quek, and H. H. Yang, “Age-threshold slotted aloha for optimizing information freshness in mobile networks,” arXiv preprint arXiv:2312.10888, 2023.
  12. O. Ayan, M. Vilgelm, M. Klügel, S. Hirche, and W. Kellerer, “Age-of-information vs. value-of-information scheduling for cellular networked control systems,” in Proceedings of the 10th ACM/IEEE International Conference on Cyber-Physical Systems, 2019, pp. 109–117.
  13. A. Maatouk, M. Assaad, and A. Ephremides, “The age of incorrect information: An enabler of semantics-empowered communication,” IEEE Transactions on Wireless Communications, vol. 22, no. 4, pp. 2621–2635, 2023.
  14. J. Holm, A. E. Kalør, F. Chiariotti, B. Soret, S. K. Jensen, T. B. Pedersen, and P. Popovski, “Freshness on demand: Optimizing age of information for the query process,” in ICC 2021-IEEE International Conference on Communications.   IEEE, 2021, pp. 1–6.
  15. A. Nikkhah, A. Ephremides, and N. Pappas, “Age of actuation in a wireless power transfer system,” arXiv preprint arXiv:2303.00507, 2023.
  16. W. Zhang, H. Zhang, H. Ma, H. Shao, N. Wang, and V. C. Leung, “Predictive and adaptive deep coding for wireless image transmission in semantic communication,” IEEE Transactions on Wireless Communications, 2023.
  17. S. Wang, J. Dai, Z. Liang, K. Niu, Z. Si, C. Dong, X. Qin, and P. Zhang, “Wireless deep video semantic transmission,” IEEE Journal on Selected Areas in Communications, vol. 41, no. 1, pp. 214–229, 2022.
  18. H. Zhang, S. Shao, M. Tao, X. Bi, and K. B. Letaief, “Deep learning-enabled semantic communication systems with task-unaware transmitter and dynamic data,” IEEE Journal on Selected Areas in Communications, vol. 41, no. 1, pp. 170–185, 2022.
  19. W. Zhang, K. Bai, S. Zeadally, H. Zhang, H. Shao, H. Ma, and V. C. Leung, “Deepma: End-to-end deep multiple access for wireless image transmission in semantic communication,” IEEE Transactions on Cognitive Communications and Networking, 2023.
  20. W. Yang, H. Du, Z. Q. Liew, W. Y. B. Lim, Z. Xiong, D. Niyato, X. Chi, X. S. Shen, and C. Miao, “Semantic Communications for Future Internet: Fundamentals, Applications, and Challenges,” IEEE Communications Surveys & Tutorials, vol. 25, no. 1, pp. 213–250, 2023.
  21. W. Yang, X. Chi, L. Zhao, Z. Xiong, and W. Jiang, “Task-driven semantic-aware green cooperative transmission strategy for vehicular networks,” IEEE Transactions on Communications, 2023.
  22. M. K. Farshbafan, W. Saad, and M. Debbah, “Curriculum learning for goal-oriented semantic communications with a common language,” IEEE Transactions on Communications, vol. 71, no. 3, pp. 1430–1446, 2023.
  23. J. Ren, Z. Zhang, J. Xu, G. Chen, Y. Sun, P. Zhang, and S. Cui, “Knowledge base enabled semantic communication: A generative perspective,” arXiv preprint arXiv:2311.12443, 2023.
  24. H. Nam, J. Park, J. Choi, M. Bennis, and S.-L. Kim, “Language-oriented communication with semantic coding and knowledge distillation for text-to-image generation,” arXiv preprint arXiv:2309.11127, 2023.
  25. E. Grassucci, S. Barbarossa, and D. Comminiello, “Generative semantic communication: Diffusion models beyond bit recovery,” arXiv preprint arXiv:2306.04321, 2023.
  26. H. Feng, Y. Yang, and Z. Han, “Scalable ai generative content for vehicular network semantic communication,” arXiv preprint arXiv:2311.13782, 2023.
  27. A. D. Raha, M. S. Munir, A. Adhikary, Y. Qiao, and C. S. Hong, “Generative ai-driven semantic communication framework for nextg wireless network,” arXiv preprint arXiv:2310.09021, 2023.
  28. E. Grassucci, C. Marinoni, A. Rodriguez, and D. Comminiello, “Diffusion models for audio semantic communication,” arXiv preprint arXiv:2309.07195, 2023.
  29. S. K. Yoo, P. C. Sofotasios, S. L. Cotton, S. Muhaidat, F. J. Lopez-Martinez, J. M. Romero-Jerez, and G. K. Karagiannidis, “A comprehensive analysis of the achievable channel capacity in ℱℱ\mathcal{F}caligraphic_F composite fading channels,” IEEE Access, vol. 7, pp. 34 078–34 094, 2019.
  30. Y. Zhou, H. Dong, and A. El Saddik, “Deep learning in next-frame prediction: A benchmark review,” IEEE Access, vol. 8, pp. 69 273–69 283, 2020.
  31. P. Ramachandran, B. Zoph, and Q. V. Le, “Swish: a self-gated activation function,” arXiv preprint arXiv:1710.05941, vol. 7, no. 1, p. 5, 2017.
  32. X. Liu, X. Zhang, J. Ma, J. Peng, and Q. Liu, “Instaflow: One step is enough for high-quality diffusion-based text-to-image generation,” arXiv preprint arXiv:2309.06380, 2023.
  33. J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020.
  34. P. Dhariwal and A. Nichol, “Diffusion models beat GANs on image synthesis,” Advances in neural information processing systems, vol. 34, pp. 8780–8794, 2021.
  35. W. Wang, J. Bao, W. Zhou, D. Chen, D. Chen, L. Yuan, and H. Li, “Semantic image synthesis via diffusion models,” arXiv preprint arXiv:2207.00050, 2022.
  36. A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al., “Segment anything,” arXiv preprint arXiv:2304.02643, 2023.
  37. A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan et al., “Searching for mobilenetv3,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1314–1324.
  38. T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel et al., “Soft actor-critic algorithms and applications,” arXiv preprint arXiv:1812.05905, 2018.
  39. F. Xu, Y. Ruan, and Y. Li, “Soft actor-critic based 3d deployment and power allocation in cell-free unmanned aerial vehicle networks,” IEEE Wireless Communications Letters, 2023.
  40. C. Diehl, T. S. Sievernich, M. Krüger, F. Hoffmann, and T. Bertram, “Uncertainty-aware model-based offline reinforcement learning for automated driving,” IEEE Robotics and Automation Letters, vol. 8, no. 2, pp. 1167–1174, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Wanting Yang (15 papers)
  2. Zehui Xiong (177 papers)
  3. Yanli Yuan (4 papers)
  4. Wenchao Jiang (10 papers)
  5. Tony Q. S. Quek (237 papers)
  6. Merouane Debbah (269 papers)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets