EpiCastBench: Datasets and Benchmarks for Multivariate Epidemic Forecasting

Abstract: The increasing adoption of data-driven decision-making in public health has established epidemic forecasting as a critical area of research. Recent advances in multivariate forecasting models better capture complex temporal dependencies than conventional univariate approaches, which model individual series independently. Despite this potential, the development of robust epidemic forecasting methods is constrained by the lack of high-quality benchmarks comprising diverse multivariate datasets across infectious diseases and geographical regions. To address this gap, we present EpiCastBench, a large-scale benchmarking framework featuring 40 curated (correlated) multivariate epidemic datasets. These publicly available datasets span a wide range of infectious diseases and exhibit diverse characteristics in terms of temporal granularity, series length, and sparsity. We analyze these datasets to identify their global features and structural patterns. To ensure reproducibility and fair comparison, we establish standardized evaluation settings, including a unified forecasting horizon, consistent preprocessing pipelines, diverse performance metrics, and statistical significance testing. By leveraging this framework, we conduct a comprehensive evaluation of 15 multivariate forecasting models spanning statistical baselines to state-of-the-art deep learning and foundation models. All datasets and code are publicly available on Kaggle (https://www.kaggle.com/datasets/aimltsf/epicastbench) and GitHub (https://github.com/aimltsf/EpiCastBench).

• The paper introduces a unified dataset with 40 multivariate epidemic incidence datasets covering diverse diseases and temporal granularities.
• It implements a standardized benchmarking protocol evaluating 15 forecasting models using both scale-independent and scale-dependent metrics.
• Foundation models like TimesFM and Chronos-2 consistently outperform others, demonstrating robust performance in long-term epidemic forecasting.

EpiCastBench: A Unified Dataset and Benchmarking Framework for Multivariate Epidemic Forecasting

Motivation and Problem Statement

Robust epidemic forecasting is fundamental to public health, supporting proactive intervention planning and resource allocation. While data-driven models—spanning statistical baselines, machine learning, deep learning, and foundation models—have demonstrated increasing efficacy, systematic progress is stymied by the absence of large-scale, curated, and diverse multivariate epidemic datasets. Existing repositories are either univariate, geographically constrained, or lack structural and epidemiological diversity, limiting the generalizability and comparability of methodological advances.

Framework and Dataset Design

EpiCastBench introduces a comprehensive, large-scale benchmarking framework comprised of 40 publicly available, multivariate epidemic incidence datasets. These datasets cover a spectrum of infectious diseases (COVID-19, chickenpox, chikungunya, dengue, influenza, measles, tuberculosis, Zika) across 27 regions, with significant variability in temporal granularity (daily, weekly, monthly), dataset length (ranging from 60 to 2088 observations), dimensionality (4 to 107 time series per dataset), and prevalence of zero inflation. Features such as long memory, trend, non-stationarity, seasonality, spikiness, entropy, and curvature are quantified for each time series, exposing marked heterogeneity in structural complexity across diseases and transmission modalities. Geographical and epidemiological breadth is ensured by curating data from official repositories and prior peer-reviewed studies, supporting both reproducibility and standardization.

Figure 1: Overview of the EpiCastBench framework.

Global feature analysis using normalized metrics illustrates, for instance, that airborne diseases (COVID-19, measles, tuberculosis) present pronounced nonlinearity and non-stationarity, while vector-borne diseases (dengue, chikungunya, Zika) display high spikiness and trend components—engineered toward capturing the full range of epidemiological temporal behaviors relevant for model assessment.

Figure 2: Radar plot comparing global features of epidemic datasets across transmission channels.

Standardized Benchmarking Protocol

EpiCastBench stipulates a rigorous, reproducible experimental pipeline:

• Model Spectrum: 15 forecasting models are evaluated, encompassing statistical baselines (e.g., Naive), machine learning (Random Forest, XGBoost, KAN, DLinear, TSMixer), deep neural networks (LSTM, DeepAR, TCN, NBEATS, NHiTS, Transformer, TiDE), and foundation models (Chronos-2, TimesFM).
• Forecasting Horizons: Short, medium, and long-term prediction tasks tailored for operational, tactical, and strategic planning scenarios, respectively.
• Evaluation Metrics: Both scale-independent (MASE, SMAPE) and scale-dependent (MAE, RMSE) metrics are employed, with careful normalization to allow for fair cross-dataset/model comparison.
• Statistical Significance: The Friedman test and Multiple Comparisons with the Best (MCB) post-hoc analyses are implemented to robustly quantify model performance differences under non-normality and heteroscedasticity.

All preprocessing and model training is conducted using unified, open-source libraries with predefined hyperparameters to maximize comparability.

Empirical Findings: Model Comparison

Overall Model Performance

All foundation models, particularly TimesFM and Chronos-2, report consistently lowest median errors and reduced interquartile variability across the full span of datasets and forecast horizons, especially excelling under long-term forecast scenarios and in datasets with complex, non-stationary, or sparse temporal structure.

Figure 3: Boxplots comparing model performance using MASE (top) and RMSE (bottom) across long (left), medium (middle), and short-term (right) forecast horizons.

Pairwise win-count heatmaps further clarify the dominance of foundation models; they outperform both classical and deep learning baselines in the vast majority of datasets and horizons. Notably, classical ensemble methods (KAN, Random Forest, XGBoost) attain horizon-dependent competitiveness, particularly for short-term forecasting and in datasets with high sparsity or limited length, occasionally outperforming foundation models in extremely data-constrained, zero-inflated or highly stochastic regimes.

Figure 4: Win-count heatmaps based on the MASE metric for long (left), medium (middle), and short-term (right) forecasting horizons.

Execution Time and Computational Trade-offs

Execution profiling on modern GPUs shows Naive and DLinear models are computationally trivial, while ensemble and conventional deep learning models incur higher training costs. Foundation models require lower training time due to their zero-shot architecture but higher inference time from autoregressive decoding and contextual conditioning. Despite this, their scalability and efficiency per forecast quality remain optimal for resource-constrained or high-throughput applications.

Figure 5: Training and inference times of all models for the short-term forecasting task on the Belgium COVID-19 dataset, run on an H100 High-RAM GPU using Google Colab. Models on the x-axis are ranked according to MASE performance, while the y-axis shows computation time (in seconds) on a logarithmic scale.

Statistical Significance and Model Robustness

The statistical testing using Friedman and MCB procedures confirms that performance differences between TimesFM and Chronos-2 are not statistically significant for most evaluation horizons and metrics, whereas classical and deep learning baselines—though occasionally competitive—show statistically inferior consistency across the suite of tasks. Nontrivial gaps between the Naive baseline and all data-driven approaches underscore the non-triviality of epidemic sequence forecasting.

Figure 6: MCB test result for long (left), medium (middle), and short-term (right) forecasting horizons based on MASE. In the plot, `TimesFM - 2.21' indicates the average rank of the TimesFM framework for long-term forecasting is 2.21; a similar interpretation holds for other models and horizons.

Additional analyses using MAE and SMAPE metrics corroborate these findings, further establishing the cross-metric robustness of the main empirical results.

Figure 7: Boxplots comparing the performance of different models based on SMAPE (upper panel) and MAE (lower panel) across long (left), medium (middle), and short-term (right) forecasting horizons.

Figure 8: Multiple comparisons with the best (MCB) test based on the (a) RMSE, (b) MAE, and (c) SMAPE metrics across long (left), medium (middle), and short-term (right) forecasting horizons. In the plot, `TimesFM - 3.30' indicates the average rank of the TimesFM framework for long-term forecasting is 3.30 based on the RMSE metric; a similar interpretation holds for other models, metrics and horizons.

Implications for AI and Epidemic Forecasting

Practical Considerations

• Model Selection: When forecasting horizons are long, or datasets exhibit high non-stationarity and structural complexity, foundation models are preferred. However, for small-scale or highly sparse regimes (e.g., region-specific, low-frequency, or early-stage outbreaks), ensemble methods may retain performance advantages.
• Zero Inflation and Sparsity: Foundation models demonstrate notable robustness to prevalent zero-inflation, except in extremely data-limited regimes. This observation is critical for public health surveillance settings characterized by periodic lulls and bursty outbreak phases.

Theoretical Implications and Future Directions

• The introduction of EpiCastBench establishes a reproducible scaffold for genuine out-of-distribution generalization studies, cross-task model transfer, hierarchical spatiotemporal extension, and the integration of mechanistic priors into foundation model training.
• The observed stability of foundation models across diverse epidemic regimes raises important research questions regarding scaling laws, meta-learning, and in-context adaptation in the low-data, nonstationary limit, and the interplay between pretraining strategy and downstream robustness.
• The current framework is limited to deterministic forecasts and does not capture explicitly spatial or network propagation effects, motivating extendable benchmarks for probabilistic, spatiotemporal, and hybrid mechanistic-ML approaches.

Conclusion

EpiCastBench constitutes a substantial step forward by unifying dataset diversity, model benchmarking, standardized evaluation, and statistical rigor for multivariate epidemic forecasting. Its public availability, granular feature analysis, and empirical findings provide a foundation for reproducible research, informed model selection, and future advances in applied epidemic forecasting and time series AI. The empirical evidence within the framework underscores the consistency and superiority of foundation models, while also highlighting settings where conventional models remain relevant—thereby facilitating both methodological innovation and operational translation in public health informatics.