MetaboNet-Bench: Glucose Forecasting Benchmark

• MetaboNet-Bench is a benchmark that standardizes multimodal glucose forecasting in type 1 diabetes by unifying diverse datasets and preprocessing protocols.
• It defines two task settings (novel and known patient splits) and evaluates models using standard metrics like RMSE as well as clinically meaningful DTS error grids.
• The benchmark highlights the benefits of multimodality and exposes research gaps in glycemic extremes while offering an extensible, reproducible open-source evaluation framework.

Searching arXiv for the benchmark paper and closely related source papers to ground the article. MetaboNet-Bench is a benchmark for multimodal glucose forecasting for patients with type 1 diabetes (T1D), introduced to standardize model performance evaluation in a literature where results are often reported on different datasets, with different preprocessing rules, different train/test splits, and different metrics (Jeffries et al., 17 Jun 2026). It is built on the public subset of MetaboNet, a consolidated T1D dataset created to overcome fragmentation, heterogeneity in format, and uneven access conditions across existing sources (Wolff et al., 16 Jan 2026). In practical terms, MetaboNet-Bench standardizes the full evaluation cycle—data selection, preprocessing, task definition, model comparison, and metric reporting—around a public multimodal dataset and an extensible open-source software framework, with explicit support for glucose, insulin, carbohydrate, and timestamp-derived inputs (Jeffries et al., 17 Jun 2026).

1. Benchmark foundation and scope

MetaboNet-Bench is anchored in the public MetaboNet release, which spans 13 source datasets and contains 1,895 subjects and 1,464 subject-years, with over 153 million samples (Jeffries et al., 17 Jun 2026). MetaboNet itself was designed as a unified, standardized, multi-source T1D dataset because prior datasets differ substantially in structure/format, access procedure, feature availability, cohort composition, study purpose, and observational duration, which directly impedes algorithm development, comparability, and generalization (Wolff et al., 16 Jan 2026).

For benchmark use, MetaboNet-Bench further restricts the public release to data appropriate for multimodal glucose forecasting. The dataset-level inclusion criterion is overlapping CGM and insulin data from T1D patients, and the benchmark additionally requires carbohydrate availability. As a result, CTR3, Loop Observational Study, PEDAP, ReplaceBG, AZT1D, BrisT1D, HUPA-UCM, Shanghai T1DM, and T1D-UOM are retained, whereas DCLP3, DCLP5, Flair, and IOBP2 are excluded because they lack carbohydrate values between meals (Jeffries et al., 17 Jun 2026).

The benchmark excludes the private DUA-governed subset of MetaboNet in order to remain frictionless and community-accessible. Its implemented modalities are CGM/glucose, insulin dosing, carbohydrate intake, and timestamps/time embeddings. The paper presents this multimodal scope as a key advance over prior benchmarks such as GlucoBench, which focus on CGM-only forecasting (Jeffries et al., 17 Jun 2026).

This design places MetaboNet-Bench within a broader benchmarking tradition that emphasizes standardized preprocessing, synchronized evaluation, and extensible software abstractions. This suggests a methodological affinity to frameworks such as BenchML, although MetaboNet-Bench is specialized to multimodal T1D time-series forecasting rather than materials or molecular representation benchmarking (Poelking et al., 2021).

2. Data selection, filtering, and split construction

The benchmark workflow is: retrieve data, preprocess and filter, segment into windows, run model inference, then compute quantitative and clinical evaluations (Jeffries et al., 17 Jun 2026). The MetaboNet download already comes with train and test sets, and MetaboNet-Bench uses these predefined splits while formalizing two task settings.

The first task is the Novel patients split (Task 1), in which about 10% of subjects in each dataset are assigned exclusively to the test set. This evaluates generalization to entirely unseen patients. The second is the Known patients split (Task 2), in which 10% of the remaining training subjects are selected and the time-sorted second half of their records is assigned to the test set. This evaluates performance when earlier data from the same patient appear in training. The full benchmark reports the combined test set, merging both portions; the test set is approximately 15% of the total dataset (Jeffries et al., 17 Jun 2026).

Several filtering and imputation rules are fixed by the benchmark. Missing carbohydrate values are filled with 0 for up to 12 hours following a valid carbohydrate entry. Missing insulin values are filled with 0 for up to 1 hour following a valid insulin entry. Rows still missing timestamps, carb data, or insulin are removed. Patients on multiple daily injections (MDI) are excluded, because long-acting insulin has effects extending beyond the 15-hour context window used by the benchmark and would confound the modeling assumptions. These steps reduce the number of per-patient time slices from 22.1 million to 17.3 million (Jeffries et al., 17 Jun 2026).

CGM is standardized to a 5-minute cadence as much as possible. Gaps up to 30 minutes between valid CGM measurements are linearly interpolated, both to harmonize devices with sampling intervals longer than 5 minutes and to fill short signal losses. Remaining gaps define breaks between contiguous sequences. Each contiguous segment receives a unique sequence ID, and sequences shorter than 16 hours are discarded so that there is enough history for the 15-hour context plus forecasting targets up to 1 hour (Jeffries et al., 17 Jun 2026).

3. Forecasting task and model classes

MetaboNet-Bench defines a multi-horizon glucose forecasting task at 5-minute increments from 5 to 60 minutes ahead, using 15 hours of historical context. The benchmark defines 30 minutes as the primary horizon because of its clinical relevance, especially for automated insulin delivery (AID) systems, but full horizon tables are reported through 60 minutes (Jeffries et al., 17 Jun 2026).

The benchmarked models span naïve baselines, classical ML, recurrent sequence models, domain-specific forecasting architectures, and a general-purpose foundation model. The included models are ZOH (zero-order hold), LE (linear extrapolation), Ridge regression, LightGBM, LSTM, Gluformer, UniTS, and GluForecast (Jeffries et al., 17 Jun 2026). Their input configurations differ materially:

• GluForecast: timestamps, CGM, insulin, carbs; input length 167
• Gluformer: timestamps, CGM only; input length 180
• LE: CGM only; input length 1
• LightGBM: timestamps, CGM, insulin, carbs; input length 24
• LSTM: CGM, insulin, carbs; input length 180
• Ridge: timestamps, CGM, insulin, carbs; input length 24
• UniTS: CGM, insulin, carbs; input length 180
• ZOH: CGM only; input length 1

For GluForecast, the paper specifies cyclical day/week embeddings. Let $t$ be a timestamp in seconds. Then

$D = 24 \cdot 60 \cdot 60, \qquad W = 7D$

and

$t_{day} = t \bmod D, \qquad t_{week} = t \bmod W.$

With angular phases $\theta_{\mathrm{day}}(t)$ and $\theta_{\mathrm{week}}(t)$, the time embedding is

$$\phi(t)= \begin{bmatrix} \sin(\theta_{\mathrm{day}}(t)) \ \cos(\theta_{\mathrm{day}}(t)) \ \sin(\theta_{\mathrm{week}}(t)) \ \cos(\theta_{\mathrm{week}}(t)) \end{bmatrix}.$$

GluForecast predicts glucose changes for each horizon and is trained with a weighted multi-horizon loss,

$$\mathcal{L} = \sum_{h=1}^{H} w_h \sum_{t=c}^{T-h} \ell\!\left(y_t + \hat{y}_{h,t},\, y_{t+h}\right), \qquad w_h = 10 + h,$$

with loss calculated only for predictions with at least 3 hours of context, i.e. after 36 values (Jeffries et al., 17 Jun 2026).

4. Evaluation protocol and clinical metrics

MetaboNet-Bench evaluates models with both standard forecasting metrics and clinically motivated error analyses (Jeffries et al., 17 Jun 2026). The standard metrics are RMSE and MARD. The appendix gives

$$RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat y_i)^2}.$$

The MARD expression printed in the paper is malformed; the denominator is omitted in the typesetting, but MARD is still used throughout as a conventional error metric (Jeffries et al., 17 Jun 2026).

A distinctive feature of the benchmark is its use of the Diabetes Technology Society (DTS) error grid. Each prediction-reference pair is mapped into clinically meaningful risk zones: Zone A, Zone B, and Zones C–E combined. The motivation is explicit: two models can have similar aggregate RMSE but differ materially in whether their errors are clinically dangerous, especially in hypo- or hyperglycemia (Jeffries et al., 17 Jun 2026).

The benchmark also defines perturbation subsets. A sample is labeled Postprandial if current carbs or carbs reported in any 5-minute interval within the previous 30 minutes are nonzero. A sample is labeled Correction bolus if current CGM is $>250$ mg/dL and insulin $>2$ IU appears in any interval within the previous 30 minutes. These analyses make it possible to compare models specifically around meals and insulin corrections, where forecasting is clinically consequential (Jeffries et al., 17 Jun 2026).

Subgroup analyses are also supported by the scale of the underlying dataset. The paper stratifies by age, weight, height, and gender, and uses the same benchmark outputs to assess whether model rankings are stable across subpopulations (Jeffries et al., 17 Jun 2026).

5. Empirical results and benchmark-derived findings

At the primary 30-minute horizon, the benchmark reports the following results (Jeffries et al., 17 Jun 2026):

Model	RMSE / MARD	DTS A
GluForecast	19.95 / 10.76	87.65
LSTM	20.06 / 10.89	87.44
UniTS	20.44 / 10.93	87.25
LightGBM	20.84 / 11.27	86.55
Gluformer	21.97 / 11.69	85.69
Ridge	22.03 / 11.94	85.34
ZOH	26.28 / 13.80	80.77
LE	30.60 / 15.54	77.03

Across all horizons, GluForecast has the best RMSE and MARD. At 5 minutes it achieves RMSE 3.57 mg/dL, compared with 3.69 for LightGBM and 3.85 for Ridge; at 60 minutes it remains best at 33.17, versus 33.41 for LSTM, 34.14 for UniTS, and 36.35 for Gluformer (Jeffries et al., 17 Jun 2026).

The benchmark also shows that simple models remain competitive in specific regimes. For horizons up to 15 minutes, linear extrapolation can match or even outperform more complex models. The paper further notes that LE performs especially well in the lowest glycemic ranges for prediction horizons up to 30 minutes, outperforming GluForecast in hypoglycemia. This is a central scientific result of the benchmark: average metrics do not imply uniform superiority in safety-critical regions (Jeffries et al., 17 Jun 2026).

The split-based analysis shows only slight gains for the known-patient task relative to the novel-patient task, with unchanged model rankings. At 30 minutes, GluForecast RMSE is 20.06 on novel patients versus 19.70 on known patients; LSTM is 20.16 versus 19.82; UniTS is 20.55 versus 20.17. Because even ZOH and LE show the same pattern, the authors caution that the difference may reflect differing variability between subsets more than a true personalization effect (Jeffries et al., 17 Jun 2026).

The ablation study evaluates five models under four input configurations: CGM only, CGM + carbs, CGM + insulin, and CGM + insulin + carbs. At 30 minutes, all models except Ridge gain some benefit from multimodality, but the magnitude depends strongly on model class. GluForecast improves from 20.66 to 19.95 RMSE; UniTS from 21.10 to 20.44; LSTM from 20.59 to 20.06; LightGBM from 21.17 to 20.84; Ridge from 22.16 to 22.03. DTS A percentages move in the same direction. The authors interpret this as evidence that larger or more expressive models can better exploit non-CGM inputs in the setting of a large benchmark-scale dataset (Jeffries et al., 17 Jun 2026).

The perturbation analyses refine this conclusion. Around meals, adding carbohydrates is slightly more useful than adding insulin, while around hyperglycemic correction boluses, insulin is much more important. More broadly, insulin seems to help most models more than carbohydrates, which the paper plausibly links to data fidelity: insulin from pumps is logged every 5 minutes, while carbohydrates are self-reported and sparse (Jeffries et al., 17 Jun 2026).

6. Reproducibility, extensibility, and limitations

The benchmark is fully open source, with code hosted on GitHub and model artifacts available either on Hugging Face Hub or in the repository (Jeffries et al., 17 Jun 2026). Reproducibility is supported through fixed public data splits, shared preprocessing logic, shared task definitions, common metrics, and open implementations. Hyperparameter tuning is standardized with Optuna for several models; LightGBM uses 3-fold CV and Optuna search, Ridge uses a standardized pipeline with StandardScaler and Optuna over $D = 24 \cdot 60 \cdot 60, \qquad W = 7D$0, and UniTS, LSTM, and GluForecast each undergo separate tuning for each ablation condition (Jeffries et al., 17 Jun 2026).

A notable engineering property is extensibility. The paper states that a new Hugging Face model can typically be added by writing one implementation file and registering it in the supported model list, and that existing implementation files are usually 25 lines of code or less. This makes the benchmark a reusable evaluation substrate rather than a one-off comparison. Training was done mostly on a single H200 GPU with 141 GB VRAM for deep models, while simpler models ran on CPU; end-to-end benchmarking, metrics, and figure generation take about one hour on H200 (Jeffries et al., 17 Jun 2026).

The limitations are explicit. Despite MetaboNet containing additional modalities, the benchmark currently evaluates only CGM, insulin, and carbohydrate data. Behavioral and physiological signals such as wearables or exercise are not benchmarked yet. The benchmark uses only the subset of MetaboNet with overlapping CGM, insulin, and carb data, so it may represent a biased slice of the broader T1D population. Residual missingness may still affect modeling. The ablation study covers only five models and does not include a multimodal version of Gluformer. Some architectures may not be fully optimized. Finally, the protocol currently evaluates generalized population-wide models only; explicit personalized fine-tuning is not yet built into the benchmark, even though the known-patient split indicates that such an extension would be valuable (Jeffries et al., 17 Jun 2026).

In that sense, MetaboNet-Bench is best understood as a full benchmark ecosystem for multimodal T1D glucose forecasting rather than merely a dataset or a leaderboard. It standardizes evaluation, foregrounds multimodal inputs, and expands assessment beyond aggregate error metrics to clinically meaningful analyses. Its principal contribution is therefore not only the ranking of current models, but the exposure of concrete research gaps: poor performance in glycemic extremes, limited robustness after meals and correction boluses, weak exploitation of multimodality by shallow models, and only modest gains from known-patient exposure without explicit personalization (Jeffries et al., 17 Jun 2026).