- The paper presents a generalized, automated demand forecasting pipeline designed to enable small to medium-sized enterprises (SMEs) without dedicated data scientists to implement advanced sales forecasting.
- The proposed pipeline integrates automated data preparation, training across diverse models (including statistical, tree-based, deep learning, and GAM), and selection of the best performing model based on validation results.
- Testing with SMEs showed that model performance varied significantly depending on the company's data, that no single model was universally best, and that automated selection improved accuracy for some, with HWES surprisingly often recommended in practice.
This paper presents a generalized, automated demand forecasting pipeline designed for small to medium-sized enterprises (SMEs) that lack the resources of dedicated data scientists. The core idea is to automate time series sales forecasting, encompassing data preparation, model training, and model selection based on validation results, thereby providing SMEs with an advanced planning tool.
Problem Statement:
SMEs often lack the resources to develop and implement customized sales forecasting methods, hindering their ability to optimize production, purchasing, and logistics. Existing ERP systems often lack sophisticated forecasting tools that can be easily used by individuals without machine learning expertise.
Proposed Solution:
The authors propose an automated demand forecasting pipeline comprising two main phases: model preselection and forecasting pipeline development.
- Model Preselection: Several state-of-the-art methods were tested on a showcase dataset to identify suitable candidates for the pipeline. The showcase dataset contained sales data for 51 products over a period of up to 6 years. The best six performing models (ARIMA, SARIMAX, Holt-Winters Exponential Smoothing, Regression Tree, Dilated Convolutional Neural Network, and Generalized Additive Model) were selected for the forecasting pipeline. Long-Short-Term Memory (LSTM) was excluded due to insufficient prediction accuracy for the 18-month horizon, and Facebook Prophet was excluded due to compatibility issues with the production environment despite its robustness.
- Forecasting Pipeline: The proposed forecasting pipeline was tested with SMEs in the food and electric industries. The pipeline automates the process of training and evaluating different forecasting models on the SME's sales data, then recommends the model with the best validation performance.
Methods:
The paper employs a variety of time series forecasting methods:
- Classical Statistical Models: ARIMA, SARIMAX, Holt-Winters Exponential Smoothing (HWES).
- Tree-Based Methods: Regression Tree (XGBoost, Random Forest).
- Deep Learning: Dilated Convolutional Neural Networks (CNN).
- Generalized Additive Model (GAM): A decompositional GAM is developed, incorporating linear and exponential trends, seasonality modeled using Fourier transformation, and user-specific external features. Lasso penalization is used for coefficient estimation.
- Ensemble Methods: The pipeline includes an ensemble prediction of the above models, using a simple mean or median function to aggregate the predictions.
The pipeline includes data preprocessing steps such as validity checks and formatting for weekly and monthly sales data. Model validation is performed using a one-year holdout period, and models are evaluated using Root Mean Squared Error (RMSE). After validation, all models are retrained using the full dataset, and forecasts are generated for the next 18 months or 78 weeks.
Experimental Setup and Results:
- Model Preselection Phase: 33 different models and specifications were compared. The best performing models were primarily deep learning approaches (Dilated CNN, CNN), followed by XGBoost.
- User Testing Phase: The automated pipeline was tested with five companies from the food and electric industries. The baseline model for comparison was a naive model that predicted future values based on the mean of previous years. The model's performance was evaluated by comparing its predictions to the actual sales data from 2023, using normalized RMSE (nRMSE) and MAPE.
- Results:
- The model performance varied significantly across different companies. For a company with project-based sales, the pipeline provided no benefit, while other companies experienced more realistic and superior sales forecasts compared to naive estimators.
- No single model was universally superior. The best model for each company depended on the specific characteristics of its sales data.
- Automated model selection based on internal validation significantly enhanced forecasting accuracy for some SMEs, but not for all. The Wilcoxon signed-rank test showed that the recommended models performed significantly better than the naive estimator in four out of five companies.
- In practice, \acs{HWES} was recommended dominantly and showed the best model performance, which was not reflected in prior studies.
- The CNNs that performed well in the preselection phase, did not perform as well in the end-to-end pipeline evaluation.
Contributions:
- A generalized automated demand forecasting pipeline suitable for SMEs.
- Integration of automatic validation and model selection.
- An evaluation framework that allows the pipeline to adapt to various business contexts.
- Demonstration of the importance of model diversity.
- Evidence of the advantages of automatic validation to address the unique needs of each business.
Limitations and Future Work:
- Data quality variations in real-world ERP systems can significantly impact model performance.
- Long prediction horizons increase bias and can affect the validation process.
- Future work could include additional features (marketing expenses, market growth), improved data preprocessing and validation techniques, and exploration of robust model selection approaches.
- Future works could also focus on data from rare out-of-stock events or delayed deliveries are not collected, but might influence the practical relevance. With that, the data may not match the actual demand.
Conclusion:
The research provides a thorough assessment of an automated demand forecasting pipeline for SMEs, demonstrating its potential to address key real-world challenges. It offers valuable insights into model performance and introduces a viable approach for automated model evaluation. It is a step towards more efficient and scalable forecasting solutions.
