Kaggle forecasting competitions: An overlooked learning opportunity (2009.07701v1)

Abstract: Competitions play an invaluable role in the field of forecasting, as exemplified through the recent M4 competition. The competition received attention from both academics and practitioners and sparked discussions around the representativeness of the data for business forecasting. Several competitions featuring real-life business forecasting tasks on the Kaggle platform has, however, been largely ignored by the academic community. We believe the learnings from these competitions have much to offer to the forecasting community and provide a review of the results from six Kaggle competitions. We find that most of the Kaggle datasets are characterized by higher intermittence and entropy than the M-competitions and that global ensemble models tend to outperform local single models. Furthermore, we find the strong performance of gradient boosted decision trees, increasing success of neural networks for forecasting, and a variety of techniques for adapting machine learning models to the forecasting task.

• The paper analyzes Kaggle forecasting competitions, revealing their unique characteristics and identifying effective methods compared to traditional M-competitions.
• Top Kaggle solutions demonstrate that complex global models like GBDT and neural networks significantly outperform simple benchmarks.
• Access to exogenous variables significantly improves model accuracy, and ensemble methods consistently prove successful in top Kaggle submissions.

Learning from Kaggle's Forecasting Competitions: A Review

The paper "Learnings from Kaggle's Forecasting Competitions" by Casper Solheim Bojer and Jens Peder Meldgaard presents an analysis of six Kaggle competitions that feature real-world forecasting tasks. Unlike the highly regarded M-competitions, the Kaggle competitions remain underexplored in academic literature, despite offering a distinct and potentially valuable perspective on forecasting methods, particularly in business contexts. The paper scrutinizes the characteristics of the Kaggle datasets, benchmarks their top-performing solutions, and derives hypotheses for future competitions like the M5.

Key Findings

The authors identify higher levels of intermittence and entropy in the Kaggle datasets compared to those used in M-competitions. Traditional M-competitions, like the M3 and M4, generally focus on time series that are largely continuous and from diverse business domains, whereas the Kaggle datasets pertain directly to real-world events and incorporate intermittency. This allows the Kaggle datasets to provide a closer representation of actual business forecasting tasks, fostering the development of more applicable forecasting models.

1. Superior Models: The evaluation shows that complex, global models outperform simple, local ones across the board. Particularly, gradient boosted decision trees (GBDT) and neural networks are prevalent in successful solutions, indicating a shift in the efficacy of machine learning methods in forecasting tasks.
2. Performance Metrics: The paper presents robust numerical results, revealing that top Kaggle models outperform simple benchmarks like naïve and seasonal naïve methods by margins of 25% to 74% in forecast error reduction. This suggests that the sophisticated use of data and modeling in Kaggle submissions adds substantial value.
3. Role of Exogenous Variables: Access to exogenous variables, such as business hierarchy, promotions, events, and holidays has been shown to substantially enhance the accuracy of models. Nonetheless, variables requiring forecasts themselves, like weather and macroeconomic predictors, do not consistently provide significant advantages.
4. Ensembling and Cross-Learning: Consistent with M4 competition insights, the success of ensemble methods that blend multiple forecasting models is evident. Additionally, the utilization of global models points to the benefits of cross-learning from multiple time series.

Implications and Future Prospects

The research highlights several core implications for both practitioner and academic audience. Practically, the takeaways from Kaggle competitions suggest incorporating methods such as GBDT and neural networks in business forecasting systems, especially for tasks involving daily or weekly data and available exogenous information. Theoretically, the paper underscores the need for further investigation into the adaptability of ML models from the Kaggle ecosystem to traditional forecasting paradigms, echoing findings observed in past M-competitions.

For future exercises like the M5 competition, the authors hypothesize that Kaggle's dataset characteristics — with their typical intermittency, high entropy, and availability of exogenous data — will enable a diverse range of modeling strategies to flourish. This particularly points towards the burgeoning potential of GBDT and neural networks, with emphasis on the nuanced capabilities of ML methodology in handling prediction uncertainty and hierarchical data architecture.

Conclusion

Ultimately, the paper presses upon the importance of opening forecasting competitions to the methods refined on platforms like Kaggle. The conclusions drawn by Bojer and Meldgaard suggest that the traditional forecasting community could greatly benefit from integrating more agile and data-responsive ML methods that have demonstrated significant empirical success in addressing complex forecasting problems. With the upcoming M5 competition, the forecasting community is presented with an opportune moment to validate the learnings from Kaggle's experiences in more structured settings and to fuel ongoing research into hybrid forecasting approaches.