Feature-driven Improvement of Renewable Energy Forecasting and Trading (1907.07580v3)

Abstract: Inspired from recent insights into the common ground of machine learning, optimization and decision-making, this paper proposes an easy-to-implement, but effective procedure to enhance both the quality of renewable energy forecasts and the competitive edge of renewable energy producers in electricity markets with a dual-price settlement of imbalances. The quality and economic gains brought by the proposed procedure essentially stem from the utilization of valuable predictors (also known as features) in a data-driven newsvendor model that renders a computationally inexpensive linear program. We illustrate the proposed procedure and numerically assess its benefits on a realistic case study that considers the aggregate wind power production in the Danish DK1 bidding zone as the variable to be predicted and traded. Within this context, our procedure leverages, among others, spatial information in the form of wind power forecasts issued by transmission system operators (TSO) in surrounding bidding zones and publicly available in online platforms. We show that our method is able to improve the quality of the wind power forecast issued by the Danish TSO by several percentage points (when measured in terms of the mean absolute or the root mean square error) and to significantly reduce the balancing costs incurred by the wind power producer.

• The paper presents a data-driven LP framework that minimizes opportunity costs by integrating historical energy data and spatial forecasts within a newsvendor problem structure.
• It demonstrates that combining TSO and neighboring region forecasts can improve forecasting accuracy, reducing MAE by up to 8.5% over baseline models.
• The two-step trading model effectively adjusts bids to hedge imbalance costs, achieving a reduction in Average Opportunity Loss by up to 2.13% compared to benchmarks.

This paper introduces a data-driven method to enhance both the accuracy of renewable energy forecasts and the profitability of renewable energy producers participating in electricity markets with a dual-price settlement for imbalances. The core idea is to leverage readily available data, particularly forecasts from neighboring regions and other relevant features, within a framework inspired by the newsvendor problem. This approach translates into a computationally efficient linear programming (LP) model.

Problem Addressed:

Renewable energy producers face uncertainty in their production levels. When participating in day-ahead markets with dual-price imbalance settlements (where under- and overproduction relative to the day-ahead bid are penalized differently), inaccurate forecasts lead to significant financial losses (opportunity costs). Traditional methods often require complex probabilistic forecasts or stochastic models for energy production and market prices.

Proposed Methodology:

The paper builds upon the newsvendor problem framework, where the goal is to minimize the expected opportunity cost of deviating from the day-ahead bid $E^D$:

$$\min_{E^D} \mathbb{E}[\psi^{-}(E^D - E)^{+} + \psi^{+}(E - E^D)^{+}]$$

where $E$ is the actual production, and $\psi^{-}$ and $\psi^{+}$ are the marginal opportunity costs for under- and overproduction, respectively.

Instead of relying on probabilistic forecasts for $E$, $\psi^{-}$, and $\psi^{+}$, the authors propose a data-driven approach using historical data points $(E_t, \psi_t^{-}, \psi_t^{+}, \mathbf{x}_t)$ where $\mathbf{x}_t$ is a vector of features available at the time of bidding for period $t$. They employ a linear decision rule for the bid: $E^D_t(\mathbf{x}_t) = \sum_{j=1}^p q^j x^j_t$.

The optimization problem becomes finding the coefficients $\mathbf{q}$ that minimize the average historical opportunity cost over a training set $\mathcal{T}$:

$$\min_{\mathbf{q}} \frac{1}{|\mathcal{T}|} \sum_{t\in\mathcal{T}} \left[ \psi_t^{-}\left(\sum_{j=1}^p q^j x^j_t - E_t\right)^{+} + \psi_t^{+}\left(E_t - \sum_{j=1}^p q^j x^j_t\right)^{+} \right]$$

subject to $0 \leq \sum_{j=1}^p q^j x^j_t \leq \overline{E}$ (capacity limits). This is reformulated as a linear program using auxiliary variables.

Applications:

1. Renewable Energy Forecasting: By setting $\psi_t^{-} = \psi_t^{+} = 1$ for all $t$ in the training set, the LP minimizes the Mean Absolute Error (MAE) of the linear combination $\sum q^j x^j_t$ with respect to the actual production $E_t$. This effectively learns the conditional median of the energy production given the features. If an existing forecast (e.g., from a TSO) is included as one of the features $x^j$, the model learns to combine it with other features (like spatial forecasts from neighboring regions) to produce an improved forecast.
2. Renewable Energy Trading: Directly using the full LP with historical $\psi_t^{-}$ and $\psi_t^{+}$ values was found to perform poorly due to the low predictability of these costs and potential overfitting. Therefore, a two-step approach is proposed:
   • Step 1: Generate an improved forecast $\hat{w}_t$ using the forecasting LP (setting $\psi_t^{-} = \psi_t^{+} = 1$) with relevant features.
   • Step 2: Use a simpler LP model to find a single scaling factor $a$ that adjusts the improved forecast for trading: $E^D_t = a \hat{w}_t$. This simpler model minimizes the historical opportunity cost using only the improved forecast $\hat{w}_t$ as input: $$\min_{a} \frac{1}{|\mathcal{T}|} \sum_{t\in\mathcal{T}} [\psi_t^{-}(a \hat{w}_t - E_t)^{+} + \psi_t^{+}(E_t - a \hat{w}_t)^{+}]$$. This captures the learnable mid-term patterns in the relative costs ($\psi^{-}$ vs $\psi^{+}$) without overfitting to noise.

Implementation and Case Study:

• Data: The paper uses publicly available data (ENTSO-e Transparency Platform, Energinet.dk) for the DK1 bidding zone in Denmark (onshore wind power) from Aug 2015 to Apr 2019.
• Features: Various features were tested, including the Danish TSO's DK1 onshore forecast (benchmark, BM), DK1 offshore forecast, DK2 forecasts, forecasts from neighboring zones (NO2, SE3, SE4, DE-AT-LU), DK1 load/solar/generation forecasts, and time-based categorical features (hour, day of week).
• Models: Several forecasting models (FM1-FM5) were built by adding feature sets incrementally. "Utopian" models (UM1, UM2) used actual outcomes instead of forecasts for some features to gauge potential improvement limits. A trading model (TM) was built using the output of the best-performing practical forecasting model (FM3) as input $\hat{w}_t$.
• Training: A rolling-window approach was used, retraining the models daily. The optimal training set length (hyperparameter) was determined via validation (6 months was chosen for both forecasting and trading). Features were scaled.
• Metrics: Forecasts were evaluated using MAE and RMSE improvement over the benchmark (BM). Trading performance was assessed by the reduction in Average Opportunity Loss (AOL) compared to bidding the benchmark forecast directly.

Results:

• Forecasting: The proposed LP method significantly improved the Danish TSO's forecast. Using just the DK1 offshore forecast (FM1) yielded ~7% MAE reduction. Adding DK2 forecasts (FM3) increased this to ~8.5% MAE reduction. Adding forecasts from further zones (FM4, FM5) provided minimal extra benefit but raised concerns about data time-consistency. The model effectively learned to combine the TSO's forecast with spatial information. Time-based and other DK1 features (FM2) did not add value. The interpretability of the linear coefficients showed the importance of the original TSO forecast and nearby regional forecasts.
• Trading: Directly bidding the improved forecast (FM3) reduced AOL by 1.30% compared to the benchmark bid. Using the two-step trading model (TM) further improved this, achieving a 2.13% AOL reduction. The learned coefficient $a$ adapted over time to the relative average historical costs of under- vs. overproduction, often scaling the bid slightly above the forecast (>1) to hedge against higher average penalties for overproduction observed in the DK1 data.

Conclusions:

The paper presents a practical, easy-to-implement linear programming framework based on the data-driven newsvendor model that effectively leverages available features (especially spatial forecasts) to improve both renewable energy forecasting accuracy (reducing MAE/RMSE) and trading performance (reducing AOL) in dual-price imbalance markets. The two-step approach for trading balances model flexibility with the limited predictability of imbalance costs. The method demonstrated significant gains using real-world data for Danish wind power.