MetaboNet-Bench: A Multi-modal Benchmark for Glucose Forecasting in Type 1 Diabetes

Abstract: Glucose forecasting algorithms are an important aspect of glycemic control management in type 1 diabetes. So far, the research community has developed numerous algorithms and models for forecasting. However, it is well-recognized that the lack of standardized model performance evaluation benchmarks makes fair comparison difficult and hinders further innovation, and thus benchmark standardization is in urgent need. Furthermore, many published glucose forecasting algorithms are limited to CGM data alone, ignoring other multimodal signals such as insulin dosing and carbohydrate intake. Here, we introduce MetaboNet-Bench, a benchmark for multimodal glucose forecasting for patients with type 1 diabetes that provides an extensible open-source evaluation framework for comparison of glucose forecasting algorithms that leverage glucose, insulin, and carbohydrate data. We then demonstrate its utility by benchmarking several recently published glucose forecasting models and a custom multimodal time-series model, representing different model architectures. The results show that the benefit of adding data modalities is conditioned on the complexity of the model and that incorporating more clinical metrics helps identify meaningful gaps to fill for future research.

• The paper introduces an open-source, multi-modal benchmark, MetaboNet-Bench, that consolidates data from 13 public studies to standardize glucose forecasting in T1D management.
• It evaluates a range of models—statistical, ML, and deep learning—with GluForecast achieving superior performance on RMSE, MARD, and clinical DTS error metrics.
• The study demonstrates the clinical value of integrating multimodal data for improved postprandial and correction forecasts, with detailed subgroup and perturbation analyses.

MetaboNet-Bench: A Comprehensive Multi-Modal Benchmark for Glucose Forecasting in Type 1 Diabetes

Introduction and Motivation

Glucose forecasting is a core challenge in T1D management and AID system design, with recent progress driven by CGM-based time series analysis. The field, however, has suffered from a lack of standardized, extensible benchmarks facilitating systematic and fair comparison across models and datasets, particularly those incorporating multimodal data (glucose, insulin, carbohydrate intake), which significantly influence glycemic control. Previous benchmarks focused on small or fragmented datasets and were limited primarily to glucose-only prediction (2606.18640).

MetaboNet-Bench directly addresses these gaps by offering an open-source, multimodal benchmarking platform underpinned by the public MetaboNet dataset. This resource integrates data from 13 public studies, comprising 1,895 subjects and 1,464 subject-years, substantially increasing both scale and diversity relative to prior efforts. This extensibility enables not only cross-model comparisons on standard metrics but also fine-grained clinical evaluations and subgroup analyses. The benchmark supports research progress through transparent methodology, reproducibility, and ease of extension.

Dataset Design and Preprocessing

MetaboNet-Bench leverages the public subset of the MetaboNet dataset, subject to explicit criteria: presence of CGM, rapid-acting insulin, and carbohydrate intake data. Inputs are systematically filtered—removing MDI users and standardizing missing data imputation (zero-filling up to 1 hour for insulin, 12 hours for carbs). CGM gaps of up to 30 minutes are addressed via interpolation; sequences shorter than 16 hours are excluded to ensure consistency for model input windows (15h context, up to 1h horizon prediction). The train/test split is patient-based, configured as both a fully novel patient split (Task 1) and a "known patient" timewise split (Task 2), enabling explicit evaluation of model generalization to new individuals versus those with prior data in training.

Benchmarking Models

A panel of models is evaluated, spanning statistical, classic ML, and modern deep architectures:

• ZOH and Linear Extrapolation (LE): Glucose-only statistical baselines.
• Ridge Regression and LightGBM: ML baselines with lagged features.
• LSTM: Deep time series model with multimodal capability.
• UniTS: Pretrained, domain-agnostic transformer-based foundation model.
• Gluformer: State-of-the-art transformer for glucose forecasting.
• GluForecast: Custom multimodal temporal transformer, exploiting glucose, insulin, and carbohydrate data with a causal, horizon-specific architecture.

Hyperparameter selection is performed via Optuna with distinct configurations per ablation and modality.

Evaluation Protocol and Metrics

MetaboNet-Bench evaluates models across standard metrics (RMSE, MARD) and clinical relevance using the DTS error grid, which zones prediction-reference pairs by clinical risk—A (safe) through E (dangerous). Performance is stratified by prediction horizon (every 5 minutes, up to 1 hour) and across subpopulations (glycemic region, age, gender, meal, correction context). Ablation studies quantify the incremental benefit of each modality (CGM only; +carb; +insulin; +carb+insulin) for each model.

Results

Aggregate Model Performance

GluForecast exhibits best-in-class accuracy (RMSE at 30-min horizon: 19.95 mg/dL; MARD: 10.76%; DTS zone A: 87.65%) relative to Gluformer, UniTS, LightGBM, and statistical baselines. LE, while competitive at very short horizons and in hypoglycemic prediction, loses efficacy at longer horizons, where complex, multimodal models dominate.

Figure 1: Linear Extrapolation and GluForecast DTS error grids at 30 min; RMSE for all models and horizons. Clinical risk is mapped by grid color, emphasizing A zone as safe.

Performance rankings are consistent across novel and known patient splits, with negligible gains for known-patient exposure, indicating persistent generalization challenges—partly attributable to the heterogeneity and physiological complexity across subjects.

Figure 2: RMSE for novel versus known patient splits across all horizons, highlighting minimal benefit from prior patient data in training.

Glycemic-Region-Specific Performance

There are pronounced discrepancies across glycemic ranges. In hypoglycemic states, simple models (LE) match or outperform deep/multimodal models at short horizons. RMSE rises for all models in both hypo- and hyperglycemic ranges, reflecting both class imbalance and greater intrinsic physiological unpredictability.

Figure 3: Distribution and RMSE by glycemic region for 30- and 60-minute horizons; LE excels in hypoglycemia at 30 min, while GluForecast leads overall at longer horizons.

A comprehensive heatmap further reveals horizon- and range-dependent weaknesses in all models (Figure 4).

Clinical Perturbation and Event Analysis

Models leveraging multimodal inputs (notably GluForecast and UniTS) show pronounced gains in postprandial (meal) and correction bolus contexts, underscoring the importance and value of insulin and carbohydrate features for practical deployment in real-world management scenarios.

Figure 5: Ablation study—RMSE deltas under postprandial and correction conditions versus control. Multimodal models exploit perturbation signals more effectively.

In particular, insulin improves performance post-correction, while carbohydrates primarily benefit postprandial forecasts.

Figure 6: RMSE with and without recent meals. Multimodal models are more robust in postprandial windows.

Figure 7: Model performance during hyperglycemia, with and without recent insulin corrections. Insulin input is critical for accurate correction prediction.

Subpopulation Analyses

Robustness across subpopulations is assessed by age, weight, height, and gender. All models suffer reduced accuracy in pediatric (age <20) subgroups, consistent with higher glycemic volatility. Model rankings are preserved across demographics; GluForecast maintains superior performance.

Figure 8: RMSE by age subpopulations. Prediction error increases substantially in pediatric groups.

Figure 9: RMSE by gender, aggregated over prediction horizons. No anomalous model ranking shifts detected; inter-gender differences are minor.

Scaling and Ablation Trends

Model parameter count is correlated with capacity to exploit multimodal input. Shallow models (e.g., Ridge) derive minimal benefit from additional signals; large-scale learners (e.g., UniTS, GluForecast) achieve the largest marginal gains with insulin and carbohydrate data.

Theoretical and Practical Implications

MetaboNet-Bench demonstrates that clinically relevant advances in glucose forecasting now hinge less on incremental improvements in RMSE/MARD and more on consistency and reliability across diverse clinical scenarios, subpopulations, and perturbation responses. Notably, traditional metrics often fail to reflect critical failures in hypoglycemia prediction—a key unsolved problem for safe algorithm deployment. The resource sets a new standard for reproducibility, extensibility, and clinical relevance, supporting future benchmarking and research in multimodal and personalized AI for T1D management.

Further, robust postprandial and perturbation event prediction is only feasible with high-fidelity multimodal data—emphasizing the necessity of comprehensive integration in both dataset construction and model architecture. Scaling laws suggest further gains may come from even larger foundation models and better exploitation of latent individual signals.

Limitations

The benchmark is currently restricted to CGM, insulin, and carbohydrate modalities; external physiological signals (physical activity, heart rate) remain underutilized due to sporadic availability. The representativeness of the dataset may be biased toward populations with comprehensive logging practices. No claims are made with respect to end-to-end safety or regulatory compliance in clinical deployment, and benchmarking does not substitute for prospective validation. Additionally, the current study does not encompass model fine-tuning at the individual patient level—a direction for further work enabled by the split protocol.

Conclusion

MetaboNet-Bench establishes a rigorous standard for multimodal glucose forecasting evaluation in T1D. Through systematic integration of diverse data, open methodology, and clinical relevance at its core, it illuminates model strengths and substantial remaining weaknesses—particularly in rare but high-risk events and under-served subpopulations. Future directions include incorporation of broader physiological and behavioral signals, model adaptation/fine-tuning paradigms, and deeper integration with deployment-focused safety evaluation.

---

Reference:

"MetaboNet-Bench: A Multi-modal Benchmark for Glucose Forecasting in Type 1 Diabetes" (2606.18640)