Multi-agent Reinforcement Learning-based Joint Design of Low-Carbon P2P Market and Bidding Strategy in Microgrids

Abstract: The challenges of the uncertainties in renewable energy generation and the instability of the real-time market limit the effective utilization of clean energy in microgrid communities. Existing peer-to-peer (P2P) and microgrid coordination approaches typically rely on certain centralized optimization or restrictive coordination rules which are difficult to be implemented in real-life applications. To address the challenge, we propose an intraday P2P trading framework that allows self-interested microgrids to pursue their economic benefits, while allowing the market operator to maximize the social welfare, namely the low carbon emission objective, of the entire community. Specifically, the decision-making processes of the microgrids are formulated as a Decentralized Partially Observable Markov Decision Process (DEC-POMDP) and solved using a Multi-Agent Reinforcement Learning (MARL) framework. Such an approach grants each microgrid a high degree of decision-making autonomy, while a novel market clearing mechanism is introduced to provide macro-regulation, incentivizing microgrids to prioritize local renewable energy consumption and hence reduce carbon emissions. Simulation results demonstrate that the combination of the self-interested bidding strategy and the P2P market design helps significantly improve renewable energy utilization and reduce reliance on external electricity with high carbon-emissions. The framework achieves a balanced integration of local autonomy, self-interest pursuit, and improved community-level economic and environmental benefits.

• The paper presents a novel MARL framework that integrates carbon-aware JPQ auction design with decentralized decision-making in microgrid P2P markets.
• It leverages LSTM-enhanced MAPPO with a CTDE scheme to achieve up to 36.7% increased economic rewards and a 31.3% reduction in emergency grid dependency.
• Experimental results show improved storage utilization and community profit, demonstrating the framework's scalability and practical potential for sustainable energy.

Multi-Agent Reinforcement Learning for Low-Carbon P2P Market and Bidding Strategy in Microgrids

Introduction

The paper "Multi-agent Reinforcement Learning-based Joint Design of Low-Carbon P2P Market and Bidding Strategy in Microgrids" (2604.02728) presents a framework that integrates market-based incentives and decentralized, self-interested microgrid behaviors in a peer-to-peer (P2P) electricity trading context. The authors approach key limitations in prior work, such as the reliance on centralized or restrictive optimization schemes and their poor practical scalability. The proposed system leverages a Decentralized Partially Observable Markov Decision Process (DEC-POMDP) formulation, solved via a multi-agent reinforcement learning (MARL) approach, specifically LSTM-enhanced MAPPO with centralized training and decentralized execution (CTDE).

Problem Formulation and System Model

The considered system is composed of several interconnected microgrids and a main grid, with heterogeneity in load and distributed generation (DG) resources. Each microgrid is endowed with non-trivial flexibility via local energy storage systems (ESS), which are governed by practical constraints on state-of-charge (SoC), charge/discharge rates, and reservation policy.

Figure 1: Structure of the power distribution network comprising four representative microgrids.

The market operates in two stages: a day-ahead (DA) phase, where forecasted demand and generation inform base-load procurement, and an intraday P2P market to resolve forecast errors and stochastic deviations. A critical innovation is the double auction market clearing mechanism, which enforces budget balance and incorporates carbon emission intensity signals into price formation, thus directly internalizing carbon objectives into local trading strategies.

Market Mechanism and Incentive Design

The core of the proposed market architecture is a joint price-quantity (JPQ) auction with explicit carbon-aware incentive signals. Microgrids submit bids or asks integrating role, price, and quantity flexibly within bounds defined by grid prices. The market clearing algorithm ensures:

• Individual rationality: Trades are only matched if mutually beneficial.
• System-level incentives: During surplus, buyers absorbing renewables are prioritized; during deficit, sellers with high marginal benefit are matched.
• Equity: The allocation is balanced to avoid liquidity monopolization by select players.

The market operator acts as a neutral, non-surplus entity, centralizing only the auction matching but not the operation.

MARL Formulation: DEC-POMDP and LSTM-MAPPO

The P2P market operation is formally cast as a DEC-POMDP. Each microgrid agent observes its own state (including SoC, windowed local observations, and market factor summaries), and selects bids/asks and storage actions to optimize its own utility—a profit function covering P2P trades and grid interactions, including emergency procurement at high-carbon price and feed-in operations.

The MARL solution leverages a CTDE scheme: policies are trained with centralized critics accessing global state for low-variance value estimation but are executed based only on local observation history. The agent policies are parameterized by LSTM-enhanced networks, which effectively capture multi-scale temporal dependencies, essential for sequential bidding and energy management under uncertainty.

Figure 2: The LSTM-MAPPO training architecture: centralized critic during training and decentralized actor execution.

Simulation Environment and Experimental Design

Experiments employ a four-microgrid test system with realistic demand and PV generation profiles synthesized from real-world datasets and injected with Gaussian perturbations and renewable disruptions. The market price design includes a time-invariant feed-in tariff (FiT) and a dynamically-evolving emergency procurement price, reflecting marginal grid carbon intensity variations.

Figure 3: Hourly dynamics of emergency price and FiT, reflecting real-time grid carbon intensity and regulatory signals.

Key baselines for evaluation include alternative auction mechanisms (Greedy, MRDA, Vickrey-variant) and state-of-the-art MARL algorithms (MAPPO, MADDPG, IPPO, MASAC), allowing isolation of gains attributable to both market design and agent policy optimization.

Results: Market and Algorithmic Performance

Quantitative results demonstrate that the proposed JPQ market clearing mechanism significantly outperforms Greedy, MRDA, and Vickrey-style designs in terms of social welfare, external grid dependency, and energy storage utilization.

Figure 4: Convergence and comparative trajectories of four clearing mechanisms (with LSTM-MAPPO) over 7000 episodes; JPQ achieves higher economic rewards, lower emergency grid reliance, and increased flexibility via ESS.

• JPQ boosts community profit by ~33%, reduces emergency power drawn by ~30%, and raises average storage utilization by more than 3x compared to baselines.
• On individual microgrid analysis, JPQ especially benefits storage-optimal and risk-averse policies, with deficit nodes seeing the lowest emergency purchases.

The learning algorithmic comparison further highlights the superiority of the LSTM-MAPPO approach in complex, non-stationary P2P environments.

Figure 5: Comparative training performance of six MARL algorithms, with LSTM-MAPPO achieving breakthroughs in reward, grid independence, and storage utilization.

• LSTM-MAPPO increases economic reward by 36.7% over classical MAPPO, drops emergency procurement by 31.3%, and more than triples storage deployment.
• Policy-gradient methods (LSTM-MAPPO, MAPPO, IPPO) consistently dominate value-based approaches (MADDPG, MASAC) in this strategic setting.

Implications and Theoretical Significance

This work rigorously evidences that an appropriately designed market clearing mechanism—one which embeds carbon awareness through the real-time auction process—can elicit efficient low-carbon behaviors from self-interested, decentralized actors without heavy-handed regulation or full cooperation assumptions. The empirical results affirm that temporal modeling in agent policies (via LSTM) is critical for robust adaptation and performance under partial observability and uncertainty, suggesting that future energy communities must leverage both institutional and algorithmic innovations.

The proposed framework is scalable, privacy-preserving (CTDE), and compatible with real-world operational constraints. Theoretically, it provides an actionable paradigm for aligning rational agent incentives with community-scale sustainability objectives via endogenous market signals.

Future Developments

Potential extensions include joint day-ahead and intraday market coupling, integration of richer physical power flow and network constraints, and formal security/privacy mechanisms in policy learning (possibly federated or split learning [e.g., (Lewis et al., 2024)]). As P2P electricity networks scale, demand aggregation, heterogeneity in risk/utility models, and the evolution toward transactive energy communities will motivate further advances in both economic mechanism design and distributed RL for sustainable power systems.

Conclusion

The study establishes a comprehensive framework coupling market-based incentive engineering and decentralized MARL for low-carbon P2P electricity trading in microgrids. The carbon-aware JPQ auction, combined with LSTM-enabled learning agents, yields superior community welfare, operational reliability, and environmental performance compared to existing approaches. This work substantiates both the necessity and effectiveness of integrating dynamic market incentives and temporal policy optimization in scaling future autonomous, low-carbon electricity markets.