OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensembling (2309.12659v1)

Abstract: Online updating of time series forecasting models aims to address the concept drifting problem by efficiently updating forecasting models based on streaming data. Many algorithms are designed for online time series forecasting, with some exploiting cross-variable dependency while others assume independence among variables. Given every data assumption has its own pros and cons in online time series modeling, we propose \textbf{On}line \textbf{e}nsembling \textbf{Net}work (OneNet). It dynamically updates and combines two models, with one focusing on modeling the dependency across the time dimension and the other on cross-variate dependency. Our method incorporates a reinforcement learning-based approach into the traditional online convex programming framework, allowing for the linear combination of the two models with dynamically adjusted weights. OneNet addresses the main shortcoming of classical online learning methods that tend to be slow in adapting to the concept drift. Empirical results show that OneNet reduces online forecasting error by more than $\mathbf{50\%}$ compared to the State-Of-The-Art (SOTA) method. The code is available at \url{https://github.com/yfzhang114/OneNet}.

• The paper introduces OneNet, a dynamic online ensembling method that integrates dual-model strategies and reinforcement learning to counter concept drift.
• The paper employs a two-stream architecture combining cross-time and cross-variable models to capture temporal and inter-variable dependencies for improved resilience.
• The paper demonstrates robust performance improvements, reducing MSE by 59.2% and MAE by 63.0% on benchmark datasets.

Understanding OneNet: A Novel Approach for Time Series Forecasting under Concept Drift

The paper "OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensembling" introduces OneNet, an innovative method for enhancing time series forecasting models by addressing the challenges posed by concept drift. Concept drift, defined as changes in the data distribution over time, is a well-known issue in time series forecasting, particularly affecting real-world applications where the relationship between input and output data is not constant. OneNet proposes a dynamic ensemble learning framework combining multiple predictive models to adapt effectively to these shifts, enhancing the forecasting accuracy significantly.

Key Contributions

1. Two-Stream Architecture: OneNet employs a dual-model approach using two different forecasting strategies—cross-time and cross-variable forecasters. The cross-time model focuses on temporal dependencies, treating each variable independently, which boosts robustness to concept drift. On the other hand, the cross-variable model captures inter-variable dependencies, which are crucial for datasets with a few variables where cross-variable interactions are more pronounced. This dual-model strategy allows OneNet to benefit from the advantages of both dependency assumptions.
2. Online Convex Programming (OCP) with Reinforcement Learning: OneNet integrates a novel OCP framework incorporating reinforcement learning to dynamically adjust model combination weights. Traditional methods like Exponentiated Gradient Descent (EGD) are slow to adapt to rapid changes in the data distribution. To overcome this, OneNet introduces a reinforcement learning-based bias term that captures short-term environmental changes more effectively, thus mitigating the "slow switch phenomenon."
3. Robust Performance Improvements: Empirical results across four datasets demonstrate that OneNet significantly outperforms state-of-the-art methods, reducing the average cumulative mean-squared errors (MSE) and mean-absolute errors (MAE) by substantial margins. Particularly for the ECL dataset, OneNet achieves a remarkable reduction in MSE by 59.2% and MAE by 63.0%, illustrating its efficacy in handling concept drift.

Implications and Future Directions

The proposed OneNet framework holds significant implications for both theoretical and practical aspects of time series forecasting:

• Theoretical Insights: OneNet challenges the conventional wisdom that focuses on single-model strategies by effectively combining the strengths of multiple dependency assumptions. This work prompts a reevaluation of the model bias present in time series forecasting, demonstrating the potential for multi-model ensembles to better adapt to dynamic environments.
• Practical Applications: The ability of OneNet to adapt more swiftly to concept drift makes it particularly valuable for real-world applications such as energy load forecasting, where data distributions may shift due to seasonal changes, economic factors, or technological advancements.
• Future Research Directions: Future work could explore integrating other advanced architectures into OneNet’s ensemble, refining the reinforcement learning component for even faster adaptation, and reducing model complexity for more parameter-efficient solutions. Additionally, discovering a more optimal normalization technique that enhances both distribution shift mitigation and rapid adaptation remains an open challenge.

In conclusion, OneNet represents a substantial advancement in time series forecasting under concept drift, offering a robust, adaptable framework that sets a new benchmark for accuracy and efficiency. Its incorporation of online ensembling and reinforcement learning provides a promising direction for future improvements in time series model performance.