Optimized Computation Offloading Performance in Virtual Edge Computing Systems via Deep Reinforcement Learning (1805.06146v1)

Abstract: To improve the quality of computation experience for mobile devices, mobile-edge computing (MEC) is a promising paradigm by providing computing capabilities in close proximity within a sliced radio access network (RAN), which supports both traditional communication and MEC services. Nevertheless, the design of computation offloading policies for a virtual MEC system remains challenging. Specifically, whether to execute a computation task at the mobile device or to offload it for MEC server execution should adapt to the time-varying network dynamics. In this paper, we consider MEC for a representative mobile user in an ultra-dense sliced RAN, where multiple base stations (BSs) are available to be selected for computation offloading. The problem of solving an optimal computation offloading policy is modelled as a Markov decision process, where our objective is to maximize the long-term utility performance whereby an offloading decision is made based on the task queue state, the energy queue state as well as the channel qualities between MU and BSs. To break the curse of high dimensionality in state space, we first propose a double deep Q-network (DQN) based strategic computation offloading algorithm to learn the optimal policy without knowing a priori knowledge of network dynamics. Then motivated by the additive structure of the utility function, a Q-function decomposition technique is combined with the double DQN, which leads to novel learning algorithm for the solving of stochastic computation offloading. Numerical experiments show that our proposed learning algorithms achieve a significant improvement in computation offloading performance compared with the baseline policies.

• The paper formulates computation offloading as an MDP that maximizes long-term utility by balancing execution delay, task queues, energy, and service costs.
• It introduces a Double DQN combined with Q-function decomposition to overcome high-dimensional state challenges and enhance offloading policy learning.
• The implemented algorithms, DARLING and Deep-SARL, outperform baseline models by significantly improving energy efficiency and reducing computational delay.

Optimized Computation Offloading in MEC via Deep Reinforcement Learning

The paper, "Optimized Computation Offloading Performance in Virtual Edge Computing Systems via Deep Reinforcement Learning," explores the enhancement of computation capabilities for mobile devices through optimized offloading strategies in ultra-dense sliced Radio Access Networks (RAN). The paradigm of Mobile-Edge Computing (MEC) is leveraged due to its proximity and resource richness compared to traditional cloud environments. The authors address the core challenge of dynamic computation offloading policies which adapt to the variable network conditions, captured through a Markov decision process (MDP) framework.

Core Contributions

The primary contribution of the paper is the formulation of computation offloading as an MDP problem aimed at maximizing long-term utility. The utility function incorporates a balance between execution delay, task queue status, energy constraints, and MEC service payments. To overcome the traditional reinforcement learning limitations regarding state space complexity, the authors introduce advanced deep learning strategies.

1. Double Deep Q-Network (DQN): The paper proposes a DQN-based algorithm to learn optimal offloading policies effectively without prior network statistics. This improves upon baseline reinforcement learning approaches which suffer from scalability and adaptability issues due to high-dimensional state spaces.
2. Q-Function Decomposition: Leveraging the additive structure of the utility function, a novel technique of Q-function decomposition is combined with the double DQN. This innovative approach abstracts the problem's complexity by assigning different satisfaction categories to utility components, thereby simplifying the learning task into manageable segments.
3. Practical Implementation: The proposed algorithms, DARLING and Deep-SARL, are implemented in TensorFlow, demonstrating significant improvements in computation performance against baseline policies in simulated environments. Noteworthy numerical results showcase that Deep-SARL achieves superior performance due to the efficient breakdown of the utility function.

Results and Implications

Experiments highlight substantial improvements over three baseline models: Mobile Execution, Server Execution, and Greedy Execution. The algorithms achieve a balance between energy consumption and computational efficiency, optimally allocating tasks between local device processing and MEC servers. As task and energy arrival dynamics are inherently unpredictable in real-world scenarios, the adaptability of the proposed methods suggests significant potential for deployment in real MEC environments.

Future Directions

The research opens several avenues for further exploration:

• Enhanced Learning Architectures: Investigating broader and deeper network architectures might yield additional performance gains and cater to more complex task environments.
• Integration with Network Slicing: Real-world application can benefit from exploring integration with network slicing to manage different service requirements and resource allocation dynamically.
• Scalability and Robustness: Further enhancement in algorithmic robustness and scalability will be critical for deployment in real-time dynamic and resource-constrained environments.

Conclusion

In summary, the paper sets forth a detailed and technically sound exploration of computation offloading in MEC through the application of advanced deep reinforcement learning techniques. By addressing high-dimensional state space challenges and introducing innovations like Q-function decomposition, it provides a strong foundation for optimizing MEC systems' performance. The substantial improvements in experimental results underscore the potential for these methods to significantly impact the efficiency of mobile computing infrastructures.