Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Double Deep Q-Learning-based Path Selection and Service Placement for Latency-Sensitive Beyond 5G Applications (2309.10180v1)

Published 18 Sep 2023 in cs.NI, cs.AI, cs.ET, cs.LG, cs.NA, and math.NA

Abstract: Nowadays, as the need for capacity continues to grow, entirely novel services are emerging. A solid cloud-network integrated infrastructure is necessary to supply these services in a real-time responsive, and scalable way. Due to their diverse characteristics and limited capacity, communication and computing resources must be collaboratively managed to unleash their full potential. Although several innovative methods have been proposed to orchestrate the resources, most ignored network resources or relaxed the network as a simple graph, focusing only on cloud resources. This paper fills the gap by studying the joint problem of communication and computing resource allocation, dubbed CCRA, including function placement and assignment, traffic prioritization, and path selection considering capacity constraints and quality requirements, to minimize total cost. We formulate the problem as a non-linear programming model and propose two approaches, dubbed B&B-CCRA and WF-CCRA, based on the Branch & Bound and Water-Filling algorithms to solve it when the system is fully known. Then, for partially known systems, a Double Deep Q-Learning (DDQL) architecture is designed. Numerical simulations show that B&B-CCRA optimally solves the problem, whereas WF-CCRA delivers near-optimal solutions in a substantially shorter time. Furthermore, it is demonstrated that DDQL-CCRA obtains near-optimal solutions in the absence of request-specific information.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. T. Taleb, A. Boudi, L. Rosa, L. Cordeiro, T. Theodoropoulos, K. Tsepes, P. Dazzi, A. Protopsaltis, and R. Li, “Towards Supporting XR Services: Architecture and Enablers,” IEEE Internet of Things Journal.
  2. L. Corneo, M. Eder, Mohan, et al., “Surrounded by the Clouds: A Comprehensive Cloud Reachability Study,” in Proceedings of the Web Conference 2021, ser. WWW ’21.   New York, NY, USA: Association for Computing Machinery, Apr. 2021, pp. 295–304.
  3. X. Yang, Z. Zho, and B. Huang, “URLLC Key Technologies and Standardization for 6G Power Internet of Things,” IEEE Communications Standards Magazine, vol. 5, no. 2, pp. 52–59, June 2021.
  4. T. Taleb, I. Afolabi, K. Samdanis, and F. Z. Yousaf, “On Multi-Domain Network Slicing Orchestration Architecture and Federated Resource Control,” IEEE Network, vol. 33, no. 5, pp. 242–252, Sept. 2019.
  5. T. Taleb, P. A. Frangoudis, I. Benkacem, and A. Ksentini, “CDN Slicing over a Multi-Domain Edge Cloud,” IEEE Transactions on Mobile Computing, vol. 19, no. 9, pp. 2010–2027, Sept. 2020.
  6. Y. Li, J. Huang, Q. Sun, T. Sun, and S. Wang, “Cognitive Service Architecture for 6G Core Network,” IEEE Transactions on Industrial Informatics, vol. 17, no. 10, pp. 7193–7203, Oct. 2021.
  7. M. Emu, P. Yan, and S. Choudhury, “Latency Aware VNF Deployment at Edge Devices for IoT Services: An Artificial Neural Network Based Approach,” in 2020 IEEE International Conference on Communications Workshops (ICC Workshops), June 2020, pp. 1–6, iSSN: 2474-9133.
  8. X. Vasilakos, M. Bunyakitanon, R. Nejabati, and D. Simeonidou, “Towards Low-latent & Load-balanced VNF Placement with Hierarchical Reinforcement Learning,” in 2021 IEEE International Mediterranean Conference on Communications and Networking (MeditCom), Sept. 2021, pp. 162–167.
  9. H. Sami, A. Mourad, H. Otrok, and J. Bentahar, “Demand-Driven Deep Reinforcement Learning for Scalable Fog and Service Placement,” IEEE Transactions on Services Computing, pp. 1–1, 2021.
  10. M. Liu and S. B. Alias, “Cost-Efficient Virtual Network Function Placement in an Industrial Edge System: A Proposed Method,” IEEE Systems, Man, and Cybernetics Magazine, vol. 9, no. 1, pp. 10–17, Jan. 2023.
  11. N. He, S. Yang, F. Li, S. Trajanovski, L. Zhu, Y. Wang, and X. Fu, “Leveraging Deep Reinforcement Learning With Attention Mechanism for Virtual Network Function Placement and Routing,” IEEE Transactions on Parallel and Distributed Systems, vol. 34, no. 4, pp. 1186–1201, Apr. 2023.
  12. M. Iwamoto, A. Suzuki, and M. Kobayashi, “Optimal VNF Scheduling for Minimizing Duration of QoS Degradation,” in 2023 IEEE 20th Consumer Communications & Networking Conference (CCNC), Jan. 2023, pp. 855–858, iSSN: 2331-9860.
  13. D. H. P. Nguyen, Y.-H. Lien, B.-H. Liu, S.-I. Chu, and T. N. Nguyen, “Virtual Network Function Placement for Serving Weighted Services in NFV-Enabled Networks,” IEEE Systems Journal, pp. 1–12, 2023.
  14. H. Xuan, Y. Zhou, X. Zhao, and Z. Liu, “Multi-agent deep reinforcement learning algorithm with self-adaption division strategy for VNF-SC deployment in SDN/NFV-Enabled Networks,” Applied Soft Computing, vol. 138, p. 110189, May 2023.
  15. T. Miyamura and A. Misawa, “Joint optimization of optical path provisioning and VNF placement in vCDN,” Optical Switching and Networking, vol. 49, p. 100740, May 2023.
  16. C. Yang, B. Hu, Y. Feng, H. Huang, H. Lai, and J. Tan, “An online service function chain orchestration method for profit maximization in edge computing networks,” Engineering Reports, vol. n/a, no. n/a, p. e12653.
  17. T.-W. Kuo, B.-H. Liou, K. C.-J. Lin, and M.-J. Tsai, “Deploying Chains of Virtual Network Functions: On the Relation Between Link and Server Usage,” IEEE/ACM Transactions on Networking, vol. 26, no. 4, pp. 1562–1576, Aug. 2018.
  18. B. E. Mada, M. Bagaa, T. Tale, and H. Flinck, “Latency-aware Service Placement and Live Migrations in 5G and Beyond Mobile Systems,” in ICC 2020 - 2020 IEEE International Conference on Communications (ICC), June 2020, pp. 1–6, iSSN: 1938-1883.
  19. Q. Zhang, F. Liu, and C. Zeng, “Adaptive Interference-Aware VNF Placement for Service-Customized 5G Network Slices,” in IEEE INFOCOM 2019 - IEEE Conference on Computer Communications, Apr. 2019, pp. 2449–2457, iSSN: 2641-9874.
  20. Q. Yuan, X. Ji, H. Tang, and W. You, “Toward Latency-Optimal Placement and Autoscaling of Monitoring Functions in MEC,” IEEE Access, vol. 8, pp. 41 649–41 658, 2020.
  21. T. Gao, X. Li, Y. Wu, W. Zou, S. Huang, M. Tornatore, and B. Mukherjee, “Cost-Efficient VNF Placement and Scheduling in Public Cloud Networks,” IEEE Transactions on Communications, vol. 68, no. 8, pp. 4946–4959, Aug. 2020.
  22. C. D. Alwis, A. Kalla, Q.-V. Pham, P. Kumar, K. Dev, W.-J. Hwang, and M. Liyanage, “Survey on 6G Frontiers: Trends, Applications, Requirements, Technologies and Future Research,” IEEE Open Journal of the Communications Society, vol. 2, pp. 836–886, 2021.
  23. M. Shokrnezhad and T. Taleb, “Near-optimal Cloud-Network Integrated Resource Allocation for Latency-Sensitive B5G,” in 2022 IEEE Global Communications Conference (GLOBECOM), Rio De Janeiro, Brazil, Dec. 2022.
  24. J. Lei, S. Deng, Z. Lu, Y. He, and X. Gao, “Energy-saving traffic scheduling in backbone networks with software-defined networks,” Cluster Computing, vol. 24, no. 1, pp. 279–292, Mar. 2021.
  25. J. Specht and S. Samii, “Urgency-Based Scheduler for Time-Sensitive Switched Ethernet Networks,” in 2016 28th Euromicro Conference on Real-Time Systems (ECRTS), July 2016, pp. 75–85, iSSN: 2159-3833.
  26. U. Arshad, M. Aleem, G. Srivastava, and J. C.-W. Lin, “Utilizing power consumption and SLA violations using dynamic VM consolidation in cloud data centers,” Renewable and Sustainable Energy Reviews, vol. 167, p. 112782, Oct. 2022.
  27. S. Kianpisheh and T. Taleb, “A Survey on In-network Computing: Programmable Data Plane And Technology Specific Applications,” IEEE Communications Surveys and Tutorials (COMST).
  28. J. R. Bhat and S. A. Alqahtani, “6G Ecosystem: Current Status and Future Perspective,” IEEE Access, vol. 9, pp. 43 134–43 167, 2021.
  29. M. Giordani, M. Polese, M. Mezzavilla, S. Rangan, and M. Zorzi, “Toward 6G Networks: Use Cases and Technologies,” IEEE Communications Magazine, vol. 58, no. 3, pp. 55–61, Mar. 2020.
  30. U. Gustavsson, P. Frenger, C. Fager, T. Eriksson, H. Zirath, F. Dielacher, C. Studer, A. Pärssinen, R. Correia, J. N. Matos, D. Belo, and N. B. Carvalho, “Implementation Challenges and Opportunities in Beyond-5G and 6G Communication,” IEEE Journal of Microwaves, vol. 1, no. 1, pp. 86–100, Jan. 2021.
  31. N. H. Mahmood, S. Böcker, Moerman, et al., “Machine type communications: key drivers and enablers towards the 6G era,” EURASIP Journal on Wireless Communications and Networking, vol. 2021, no. 1, p. 134, June 2021.
  32. H. Yu, C. Wang, T. Taleb, and J. Zhang, “Deep Reinforcement Learning based Deterministic Routing and Scheduling for Mixed-Criticality Flows,” IEEE Transactions on Industrial Informatics.
  33. Q. Guo, R. Gu, H. Yu, T. Taleb, and Y. Ji, “Probabilistic-Assured Resource Provisioning with Customizable Hybrid Isolation for Vertical Industrial Slicing,” IEEE Transactions on Network and Service Management (TNSM).
  34. T. Taleb, I. Afolabi, and M. Bagaa, “Orchestrating 5G Network Slices to Support Industrial Internet and to Shape Next-Generation Smart Factories,” IEEE Network, vol. 33, no. 4, pp. 146–154, July 2019.
  35. H. Kellerer, U. Pferschy, and D. Pisinger, “Multidimensional Knapsack Problems,” in Knapsack Problems, H. Kellerer, U. Pferschy, and D. Pisinger, Eds.   Berlin, Heidelberg: Springer, 2004, pp. 235–283.
  36. G. Pataki, M. Tural, and E. B. Wong, “Basis Reduction and the Complexity of Branch-and-Bound,” in Proceedings of the 2010 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), ser. Proceedings.   Society for Industrial and Applied Mathematics, Jan. 2010, pp. 1254–1261.
  37. H. v. Hasselt, A. Guez, and D. Silver, “Deep Reinforcement Learning with Double Q-Learning,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30, no. 1, Mar. 2016, number: 1.
  38. C. J. C. H. Watkins and P. Dayan, “Technical Note,” in Reinforcement Learning, ser. The Springer International Series in Engineering and Computer Science, R. S. Sutton, Ed.   Boston, MA: Springer US, 1992, pp. 55–68.
  39. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, Feb. 2015.
  40. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  41. C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, pp. 279–292, 1992.
  42. H. X. Nguyen, R. Trestian, D. To, and M. Tatipamula, “Digital Twin for 5G and Beyond,” IEEE Communications Magazine, vol. 59, no. 2, pp. 10–15, Feb. 2021.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now