- The paper develops a capacity-constrained DDQN framework that links service-provider incentive learning with appliance-level home energy management and reduces peak load by about 36%, mean load by 17%, and PAR by 22.82%.
- The framework uses real Austin residential data and ERCOT prices to target incentives near capacity limits, keeping demand below approximately 7 kW while avoiding the rebound peaks seen in a multi-agent reinforcement-learning baseline.
- The paper shows that the comfort–incentive weighting factor determines financial viability: end users achieve positive aggregate profits only at higher values of ρ, particularly 0.7 and 0.9, highlighting the importance of fair surplus allocation.
Overview
This paper proposes a capacity-constrained reinforcement learning-based demand response framework (CCRL-DR) for residential smart grids, in which a service provider (SP) learns real-time hourly incentive rates under explicit grid capacity limits. The framework couples a Double Deep Q-Network (DDQN) agent at the SP level with appliance-level home energy management system (HEMS) models at the end-user (EU) level, capturing heterogeneous comfort preferences through dissatisfaction costs. Using real-world residential load data (Pecan Street Dataport, Austin, Texas) and ERCOT wholesale prices, the authors report a peak reduction of roughly 36%, a mean-load reduction of about 17%, and a peak-to-average ratio (PAR) reduction of approximately 22.82% relative to the no-demand-response case, while avoiding the rebound peaks observed in a prior multi-agent RL baseline.
The authors position the work against two limitations of the existing incentive-based demand response (IBDR) literature. First, model-based approaches—coupon schemes, Stackelberg games, MILP formulations, and elasticity-based methods—require accurate parameter estimation, often operate offline or day-ahead, and frequently assume homogeneous users. Second, existing RL-based IBDR studies typically model consumer response at an aggregate level, decouple appliance-level flexibility from incentive design, or treat grid capacity constraints implicitly. The paper argues that a gap remains between data-driven incentive learning and practical, capacity-aware residential DR with device-level modelling. Notably, the closest related work—a multi-agent RL scheme with a uniform incentive and a disjunctively constrained knapsack formulation—serves as the qualitative rebound-peak comparison point, while the quantitative benchmark is an elasticity-based load reduction (EBLR) model.
System model
The architecture is a three-level hierarchy: a grid operator announces hourly wholesale prices and a capacity limit; the SP purchases energy wholesale and sets per-EU hourly incentive rates λn,h; each EU's HEMS responds by curtailing or shifting flexible appliances. The SP maximises wholesale cost savings minus incentive payments, with incentives bounded by contractual limits. Each EU maximises a weighted objective combining incentive income and dissatisfaction cost, controlled by a weighting factor ρ∈[0,1].
Appliances are partitioned into power-controllable (PC), interruptible time-shiftable (TS,I), non-interruptible time-shiftable (TS,NI), and non-shiftable sets. PC appliances offer discrete curtailment levels with quadratic dissatisfaction in the curtailed fraction; TS appliances incur quadratic dissatisfaction in accumulated delay, with non-interruptible appliances constrained to contiguous operating blocks and completion deadlines. Dissatisfaction coefficients βn,a are sampled per appliance type from household-specific distributions (e.g., mean 3.50 with standard deviation 2.00 for air conditioners versus 0.05 for EVs), encoding the assumption that thermal comfort matters far more than charging delay.
The global objective sums SP and EU profits per hour, augmented in implementation by a reward-shaping term Φh with four components: a bonus for issuing no incentives when no reduction is needed, a penalty for unnecessary incentive issuance, a penalty on unmet reduction (doubled when no incentive is offered despite a requirement), and a mild over-reduction penalty. This shaping term is essential to the practical behaviour of the learned policy, and the authors are explicit that it leaves the economic structure of the objective unchanged.
The decision problem is cast as a finite-horizon MDP with hourly steps, where the state comprises forecasts, current load, and capacity information, and each action is a vector of incentive rates. The reward is the hourly contribution of the global objective. A DDQN with decoupled action selection and evaluation mitigates Q-value overestimation; the target network is updated by soft updates. The real-time pipeline pre-computes daily wholesale price and per-household load trajectories with LSTM forecasters (2 layers, 64 hidden units, dropout 0.2, window 24, one-step horizon rolled over the day), which are embedded in the RL state. Forecast accuracy is reported as MAE/MAPE of 1.87/0.53 for price and, for the three households, 0.90/0.93, 0.72/0.78, and 0.41/0.24 respectively—adequate but noticeably weaker for household 1, a point the paper does not further analyse.
Experimental results
The case study uses three households over July 2018, with the aggregate capacity threshold set to 75% (about 7 kW) of the average daily peak. The DDQN agent (two hidden layers of 128 and 64 ReLU units, γ=0.99, replay buffer 50,000, batch 256, 2,500 training episodes) converges to a stable policy with no evident overfitting, though residual reward variability persists due to environment stochasticity and ε-greedy exploration.
On the representative test day (27 July 2018), incentives are activated primarily when aggregate load approaches the threshold, and the post-DR aggregate profile stays mostly below 7 kW without a significant rebound peak—an explicit contrast with the MARL-iDR baseline. Averaged over July, the results are:
| Metric |
No DR |
CCRL-DR |
| Peak load (kW) |
11.89 |
7.60 |
| Mean load (kW) |
6.46 |
5.34 |
| PAR |
1.84 |
1.42 |
The mean-load reduction being smaller than the peak reduction indicates that the scheme predominantly shifts flexible demand rather than curtailing it uniformly. The DDQN agent also tailors incentives to household heterogeneity: households with larger dissatisfaction coefficients or more inflexible appliances receive lower incentives and exhibit smaller adjustments.
The sensitivity analysis over ρ is the paper's most consequential financial finding. For ρ=0.1 and ρ=0.3, aggregate EU profit is strongly negative (−114.36¢ and −84.56¢ respectively), meaning the DR programme is economically unattractive to users even though SP profit is high (67.37¢ and 68.17¢). Only at ρ=0.7 and ρ∈[0,1]0 do all three EUs achieve positive net profit (aggregate EU profit 9.58¢ and 28.62¢) while the SP remains profitable (35.61¢ and 39.62¢). This establishes ρ∈[0,1]1 as the policy lever that allocates the DR surplus between SP and EUs, and it implies that participation sustainability in this framework requires ρ∈[0,1]2 to be set well above 0.5—a design constraint rather than a free parameter.
Against the EBLR benchmark, CCRL-DR produces reductions concentrated around congested hours and keeps the aggregate profile at or below the capacity threshold during critical periods, whereas EBLR yields diffuse, elasticity-proportional reductions that can still approach or exceed the limit at peak and waste reductions in non-critical hours. CCRL-DR attains the lowest PAR among the three operating modes.
Limitations and open questions
Several limitations are conceded or evident. The evaluation covers only three households and a single month, so scalability of the single-agent formulation to realistic population sizes remains untested; the action space of per-EU incentive vectors grows linearly with ρ∈[0,1]3, and the authors propose cluster-based aggregation as future work without evaluating it. The capacity threshold is fixed and deterministic, and the LSTM forecasts—particularly for household 1—are imperfect, yet the paper does not quantify the DR performance degradation attributable to forecast error. The dissatisfaction coefficients are user-defined parameters sampled from assumed distributions rather than calibrated from measured behavioural data, so the reported comfort outcomes depend on the plausibility of those distributions. The ρ∈[0,1]4 parameter is assumed identical across EUs, and the interaction between heterogeneous ρ∈[0,1]5 values and incentive fairness is unexplored. Finally, the comparison with EBLR uses elasticity and EU parameters "loosely adapted" from prior illustrative settings, so the quantitative margin of superiority over the benchmark is sensitive to benchmark tuning.
Conclusion
The paper demonstrates that embedding an explicit capacity limit in the state and reward of a DDQN incentive-setting agent, coupled with appliance-level HEMS response models, yields capacity-aware peak shaving and load smoothing on real residential data, with a 22.82% PAR reduction and no rebound peaks. The financial analysis shows that the comfort–incentive weighting factor determines whether participation is economically viable for end users, making it a central design parameter. The principal open question left by the paper is whether the single-agent, per-EU action formulation scales beyond a handful of households, which the authors propose to address through cluster-based incentive policies that preserve appliance-level comfort modelling.