Private Markovian Equilibrium in Stackelberg Markov Games for Smart Grid Demand Response

Abstract: The increasing integration of renewable energy introduces a great challenge to the supply and demand balance of the power grid. To address this challenge, this paper formulates a Stackelberg Markov game (SMG) between an aggregator and multiple users, where the aggregator sets electricity prices and users make demand and storage decisions. Considering that users' storage levels are private information, we introduce private states and propose the new concepts of private Markovian strategies (PMS) and private Markovian equilibrium (PME). We establish the existence of a pure PME in the lower-level Markov game and prove that it can be computed in polynomial time. Notably, computing equilibrium in general Markov games is hard, and polynomial-time algorithms are rarely available. Based on these theoretical results, we develop a scalable solution framework combining centralized and decentralized algorithms for the lower-level PME computation with upper-level pricing optimization. Numerical simulations with up to 50 users based on real data validate the effectiveness and scalability of the proposed methods, whereas prior studies typically consider no more than 5 users.