Online Learning for Offloading and Autoscaling in Energy Harvesting Mobile Edge Computing (1703.06060v1)

Abstract: Mobile edge computing (a.k.a. fog computing) has recently emerged to enable in-situ processing of delay-sensitive applications at the edge of mobile networks. Providing grid power supply in support of mobile edge computing, however, is costly and even infeasible (in certain rugged or under-developed areas), thus mandating on-site renewable energy as a major or even sole power supply in increasingly many scenarios. Nonetheless, the high intermittency and unpredictability of renewable energy make it very challenging to deliver a high quality of service to users in energy harvesting mobile edge computing systems. In this paper, we address the challenge of incorporating renewables into mobile edge computing and propose an efficient reinforcement learning-based resource management algorithm, which learns on-the-fly the optimal policy of dynamic workload offloading (to the centralized cloud) and edge server provisioning to minimize the long-term system cost (including both service delay and operational cost). Our online learning algorithm uses a decomposition of the (offline) value iteration and (online) reinforcement learning, thus achieving a significant improvement of learning rate and run-time performance when compared to standard reinforcement learning algorithms such as Q-learning. We prove the convergence of the proposed algorithm and analytically show that the learned policy has a simple monotone structure amenable to practical implementation. Our simulation results validate the efficacy of our algorithm, which significantly improves the edge computing performance compared to fixed or myopic optimization schemes and conventional reinforcement learning algorithms.

• The paper proposes a novel online reinforcement learning algorithm using a post-decision state framework to optimize workload offloading and autoscaling in energy-harvesting mobile edge computing systems.
• The developed method efficiently decomposes learning and execution, addressing the curse of dimensionality and demonstrating faster convergence and runtime performance over standard approaches like Q-learning.
• Simulation results validate that the proposed optimal policy exhibits a simple monotone structure based on battery levels, leading to lower average costs and reduced reliance on backup power sources.

Online Learning for Offloading and Autoscaling in Energy Harvesting Mobile Edge Computing

The paper "Online Learning for Offloading and Autoscaling in Energy Harvesting Mobile Edge Computing" addresses the critical challenge of managing workloads in mobile edge computing (MEC) systems powered by intermittent renewable energy sources. The authors propose a reinforcement learning-based algorithm that dynamically optimizes workload offloading to centralized cloud infrastructures and the autoscaling of edge servers. The objective is to minimize the long-term system cost, which is a function of both service delay and operational costs.

Contributions and Methodology

The primary contribution of this paper lies in its development of a novel post-decision state (PDS) based learning algorithm. This proposed method enables the system to learn an optimal offloading and autoscaling policy effectively while accommodating the unpredictable nature of renewable energy. The algorithm provides an efficient way to decompose (offline) value iteration and (online) reinforcement learning, addressing the "curse of dimensionality" typical in Markov decision process (MDP) formulations.

Key contributions and insights include:

1. MDP Formulation: The authors model the joint offloading and autoscaling problem as an MDP and define the system state by workload arrival rate, network congestion state, environment state, and battery level. The power demand decisions are made based on a PDS that separates known dynamics from stochastic components, allowing for an efficient learning process.
2. Reinforcement Learning Mechanism: The proposed algorithm leverages the structure of state transitions to expedite learning and execution compared to standard approaches like Q-learning. The paper emphasizes fast convergence and improved run-time performance by employing a batch updating scheme that adjusts learning based on environment states.
3. Monotone Structure of the Policy: It is analytically demonstrated that the optimal policy has a simple monotone structure in terms of battery levels, which significantly eases the practical implementation.
4. Simulation and Performance Validation: Extensive simulations are conducted to validate the performance improvements over existing schemes such as Q-learning and myopic optimization methods. The results show that the proposed approach yields lower average costs and reduces reliance on backup power sources.

Implications and Future Directions

The implications of this research are significant for designing and deploying energy-efficient MEC systems, particularly in environments where grid power is unreliable or unavailable. It reinforces the potential of integrating renewable sources into edge computing infrastructures without compromising service quality or cost-effectiveness.

From a theoretical standpoint, the application's PDS framework and reinforcement learning embody advanced techniques that address the dimensionality and uncertainty issues inherent in complex decision-making environments.

Looking ahead, the paper suggests extending this work to broader scales and more complex systems, like exploring geographical load balancing while considering green energy access. The adaptation of these strategies to diverse and large-scale network environments will likely be a promising and deeply impactful research direction, especially as the demand for sustainable and scalable computing solutions accelerates.

In summary, this paper provides a robust framework for enhancing the operational efficiency of MEC systems powered by renewable energy and contributes broader insights into the intersection of machine learning and sustainable computing. It stands as a foundational piece for further explorations into adaptive resource management in green computing infrastructures.