Towards Affordable Energy: A Gymnasium Environment for Electric Utility Demand-Response Programs

Abstract: Extreme weather and volatile wholesale electricity markets expose residential consumers to catastrophic financial risks, yet demand response at the distribution level remains an underutilized tool for grid flexibility and energy affordability. While a demand-response program can shield consumers by issuing financial credits during high-price periods, optimizing this sequential decision-making process presents a unique challenge for reinforcement learning despite the plentiful offline historical smart meter and wholesale pricing data available publicly. Offline historical data fails to capture the dynamic, interactive feedback loop between an electric utility's pricing signals and customer acceptance and adaptation to a demand-response program. To address this, we introduce DR-Gym, an open-source, online Gymnasium-compatible environment designed to train and evaluate demand-response from the electric utility's perspective. Unlike existing device-level energy simulators, our environment focuses on the market-level electric utility setting and provides a rich observational space relevant to the electric utility. The simulator additionally features a regime-switching wholesale price model calibrated to real-world extreme events, alongside physics-based building demand profiles. For our learning signal, we use a configurable, multi-objective reward function for specifying diverse learning objectives. We demonstrate through baseline strategies and data snapshots the capability of our simulator to create realistic and learnable environments.

• The paper introduces DR-Gym, a novel RL environment that models market-level utility demand response with realistic price dynamics and consumer behavior.
• It employs a regime-switching wholesale price model and physics-based building demand simulations to accurately replicate market spikes and load patterns.
• Experiments using PPO demonstrate enhanced policy learning, efficient budget management, and a significant reduction in consumer risk compared to baseline approaches.

DR-Gym: A Market-Level RL Environment for Demand Response in Electricity Markets

Introduction

The proliferation of extreme weather events and corresponding volatility in wholesale electricity markets has exposed consumers to substantial financial risk, particularly during market spikes. Traditional demand response (DR) mechanisms—which incentivize customers to reduce or shift consumption during peak events—remain underutilized at the distribution market level, largely due to modeling and optimization challenges inherent to dynamic, sequential market interactions. In "Towards Affordable Energy: A Gymnasium Environment for Electric Utility Demand-Response Programs" (2605.12462), the authors address this research gap by introducing DR-Gym, a Gymnasium-compatible RL environment purpose-built for simulating and optimizing electric utility DR credit policies. The environment moves beyond device-level control prevalent in existing simulators, enabling research for market-level, risk-aware policy optimization under realistic market and consumer dynamics.

Figure 1: Overview of simulator architecture showing data ingestion, module composition, and RL interaction interface.

Environment Architecture and Modeling Framework

DR-Gym is architected as a modular, open-source simulator exposing an RL interface tailored for electric utilities. The primary agent's action is to set a per-kWh credit for residential DR participation, and the observation space encompasses detailed contextual signals: time, aggregate and per-building demand, wholesale and forecasted price, stress indicators, operational budget, history, and more (~32 dimensions). Importantly, the simulator directly integrates several key real-world complexities:

• Regime-switching wholesale price model: Prices are generated using a Markov regime-switching process, calibrated to capture realistic clustering of price spikes as observed in ERCOT and CAISO markets, with AR(1) autocorrelated noise and TOU tariff baselines.
• Physics-based building demand: Load profiles are sourced from CityLearn's EnergyPlus/ResStock simulations, delivering realistic aggregation and temporal dynamics unattainable with generic synthetic models.
• Heterogeneous household response: Participation likelihood and demand reduction are drawn from empirically grounded archetypes, featuring logistic acceptance conditioned on incentives, fatigue from repeated activations, and reduction magnitudes reflecting field pilots.
• Closed-loop feedback: Demand reductions and subsequent load persistence dynamically influence next-step prices (configurable price elasticity), and customer fatigue penalizes repeated solicitation.

Figure 2: Detailed simulator architecture, highlighting modular components and information flow.

Multi-objective and Risk-aware Reward Specification

DR-Gym's reward is explicitly multi-objective: agents optimize a weighted combination of utility revenue, consumer cost, grid stress (computed from demand, price, and temperature), and an optional risk measure (e.g., CVaR of customer bills). This plug-and-play configuration allows benchmarking risk-neutral algorithms as well as risk-aware policy gradients, aligning the environment with core open questions in safe and equitable RL.

Empirical Validation of Simulator Dynamics

The authors validate the physical realism and statistical fidelity of each simulator component through direct comparison with public market and demand benchmarks.

• Wholesale price modeling: The regime-switching model replicates autocorrelation, price volatility, spike clustering, and distributional tails as observed in ERCOT and CAISO data, including stress testing for extreme events analogous to the 2021 ERCOT Uri storm.
• Building load: CityLearn/ResStock data yields aggregate load patterns with observed bimodal ramps and variance across episodes/buildings, validated against empirical smart meter traces.
• Consumer behavior: Acceptance probabilities and fatigue decay curves across archetypes are calibrated to survey-based and pilot program findings, ensuring that DR policy evaluation is meaningfully grounded in observed human participation.

RL Benchmarks and Learnability

To demonstrate the environment's learnability and fidelity, a series of RL experiments are conducted with PPO as the main RL agent, compared against four policy baselines drawn from the literature (no credit, uniform credit, stress-based thresholds, and budget-aware rule-based). Quantitative evaluation indicates:

• Policy learning: PPO agents rapidly exceed baseline rewards, optimizing credit allocation to reserve budget for high-impact events while protecting revenue.
• Budget efficiency: Learned policies use budget more selectively than uniform/baseline policies and achieve positive average utility revenue across simulations, even under repeated price storms.
• Risk reduction: With risk-aware reward configuration, PPO reduces tail risk in customer bills (CVaR$_{0.95}$) by 18–24% relative to the no-credit baseline, demonstrating the environment's suitability for safe RL research.

Comparative Analysis to Prior Environments

Existing RL environments—such as CityLearn, Sinergym, and Grid2Op—target device-level controls (e.g., HVAC scheduling, transmission operations) and largely omit market-level, consumer-protection objectives and dynamic consumer heterogeneity. DR-Gym is distinctive by:

• Modeling market-level sequential decisions under realistic price (and demand) feedback.
• Embedding statistically calibrated risk and operational constraints (e.g., rolling budgets).
• Providing access to risk-aware reward shaping (including CVaR), facilitating research at the nexus of utility profit, grid reliability, and consumer equity.

Limitations and Future Research Directions

Key limitations include the reliance on defaults for some feedback parameters; further calibration to specific datasets (e.g., Pecan Street, LBNL pilot data) would increase realism. The agent benchmarks are initial: systematic evaluation of risk-sensitive RL variants (CVaR-PPO, distributional RL, and others) is a critical direction. Incorporating customer-level behavioral adaptation, grid constraints, and integration with broader market models remain open avenues for future development.

Conclusion

DR-Gym introduces a rigorously validated, modular, and open-source RL environment for market-level DR credit optimization, addressing gaps unaddressed by prevailing simulators. By exposing a diverse, realistic, and risk-sensitive state/action/reward landscape, it establishes a strong testbed for the development and benchmarking of RL algorithms targeting grid flexibility, affordability, and consumer protection. The implications extend both to practical DR program design—enabling safer, more equitable market participation—and to theoretical advances in multi-objective, risk-aware sequential decision-making for energy systems.