- The paper introduces a novel Task-Aware Random Weighting (TARW) method that significantly boosts forecast stability while maintaining accuracy in N-BEATS models.
- Methodological innovations include the comparative evaluation of DLW techniques like GradNorm and Gradient Cosine Similarity using benchmark M3 and M4 datasets.
- Experimental results show that DLW methods, particularly TARW, reduce forecast instability without compromising performance compared to static loss weighting.
Using Dynamic Loss Weighting to Boost Improvements in Forecast Stability
The paper "Using Dynamic Loss Weighting to Boost Improvements in Forecast Stability," authored by Daan Caljon and colleagues at KU Leuven, investigates the enhancement of forecast stability using dynamic loss weighting in the context of time series forecasting with the N-BEATS model. Forecast stability is crucial in many practical scenarios where forecast updates based on new data can lead to substantial and potentially costly changes in planned activities.
Background and Motivation
Forecast instability, or Rolling Origin Forecast Instability (ROFI), arises when updates to forecasts, triggered by new data points, cause variability in forecasts for a specific period. Van Belle et al. (2023) previously addressed this challenge by extending the N-BEATS model to include forecast stability as an optimization objective, in addition to accuracy. They introduced N-BEATS-S, which uses a composite loss function to balance forecast error and instability.
Despite the promising results, the authors hypothesized that dynamic loss weighting (DLW) algorithms, which adjust loss weights during training, could further improve forecast stability without compromising accuracy. This is because DLW methods can potentially prevent the model from getting trapped in local optima and address the varying learning dynamics of different tasks.
Methodology
The authors explored several existing DLW methods and proposed a novel variant tailored for their specific context, namely Task-Aware Random Weighting (TARW). The methods evaluated include:
- GradNorm: Balances training rates by adjusting loss weights based on the gradient norms of different tasks.
- Uncertainty Weighting (UW): Adjusts weights based on the learned relative uncertainties of the tasks.
- Random Weighting (RW): Randomly samples loss weights from a uniform distribution.
- Gradient Cosine Similarity (GCosSim) and Weighted GCosSim: Adjust weights based on the cosine similarity between the gradients of accuracy and stability losses.
For their proposed method, TARW, the authors sample loss weights from a capped uniform distribution to prevent excessive emphasis on stability.
Experimental Design
The authors tested their methods on the monthly data sets from the M3 and M4 competitions, which are well-known benchmarks in time series forecasting research. They employed a rolling origin evaluation scheme to measure both forecast accuracy (sMAPE) and stability (sMAPC). The N-BEATS-S models with static and dynamic loss weighting were compared against traditional forecasting methods (ETS, ARIMA, and THETA).
Results
The empirical results indicated that all DLW methods improved forecast stability compared to the static loss weighting approach used in N-BEATS-S. However, only specific methods, such as GradNorm, Weighted GCosSim, and TARW, managed to enhance stability without significantly sacrificing accuracy.
- For the M3 data set:
- TARW achieved the highest accuracy among the DLW variants and also outperformed both N-BEATS and N-BEATS-S.
- GradNorm and Weighted GCosSim also maintained accuracy close to that of N-BEATS and N-BEATS-S.
- For the M4 data set:
- TARW again performed robustly, achieving accuracy comparable to N-BEATS-S.
- TARW, along with GradNorm and Weighted GCosSim, significantly improved forecast stability compared to static weighting.
Statistical tests confirmed these findings, emphasizing that TARW's stochastic nature allowed for better exploration and potential escape from local optima compared to static loss weight tuning.
Implications and Future Work
The paper presents strong numerical evidence that DLW methods can enhance forecast stability without compromising accuracy. This research has practical implications for industries reliant on stable forecasts for planning and operational adjustments, such as supply chain management.
Future research could explore the incorporation of forecast stability in different modeling approaches or extend these findings to other tasks within the multi-task learning domain. The proposed TARW method's general applicability to various multi-task learning problems also presents a promising direction for further investigation.
Conclusion
This paper addresses the significant issue of forecast instability in time series forecasting. By leveraging dynamic loss weighting methods, the authors demonstrate enhanced stability in forecasts generated by the N-BEATS-S model, ultimately providing a more robust forecasting tool for practical applications. The proposed TARW method, in particular, shows promise for future developments in both theoretical and applied contexts within forecasting and broader multi-task learning paradigms.