SWE-MIMIC-Bench: MIMIC-IV ICU Benchmark

• The paper presents SWE-MIMIC-Bench, a standardized benchmark for MIMIC-IV ICU data on Chronic Kidney Disease using a reproducible extraction and preprocessing pipeline.
• It regularizes irregular EHR time series into fixed 2-hour bins and handles missing data with forward fill and mean imputation to ensure consistent feature representation.
• Benchmark experiments using XGBoost, LSTM, and TCN on in-ICU mortality and LOS > 3 days tasks demonstrate competitive performance under a unified evaluation protocol.

Searching arXiv for the benchmark paper and closely related benchmark references. Search query: (Bui et al., 2024) SWE-MIMIC-Bench denotes the concrete MIMIC-IV benchmarking configuration described in "Benchmarking with MIMIC-IV, an irregular, sparse clinical time series dataset" (Bui et al., 2024). It is a reproducible benchmark for structured electronic health record time series that focuses on ICU patients admitted for Chronic Kidney Disease, uses the first 48 hours after ICU admission, regularizes irregular observations into 2-hour bins, and evaluates two binary prediction tasks: in-ICU mortality and length of stay greater than 3 days. Within this configuration, diagnoses are harmonized by converting ICD-9 to ICD-10 and then grouped, labs and vitals are selected as dynamic features, and missingness is handled by forward fill and mean imputation (Bui et al., 2024).

1. Motivation and benchmark scope

The benchmark is motivated by the observation that clinical EHR time series are high-dimensional, irregularly sampled, and sparse, with many missing values and variable sampling frequencies across measurements. Models that assume fixed-size, regularly sampled inputs struggle in this regime, which makes preprocessing choices central to any benchmark definition (Bui et al., 2024).

A central gap addressed by the benchmark is the absence of a standardized MIMIC-IV benchmark analogous to prior MIMIC-III efforts. Earlier MIMIC-III benchmarks, including those associated with Purushotham et al. and Harutyunyan et al., used different pipelines, tasks, and metrics. SWE-MIMIC-Bench instead targets MIMIC-IV with a consistent extraction and preprocessing pipeline and a common evaluation setup. The paper also emphasizes that MIMIC-IV is newer, larger, and modular, so direct transfer of MIMIC-III benchmark pipelines is reduced by differences in data organization, identifiers, and coding systems (Bui et al., 2024).

The benchmark makes an explicit design choice to regularize irregular data into fixed 2-hour bins and to apply straightforward imputation. This suggests a preference for a transparent, replicable baseline over a benchmark that is tightly coupled to a particular irregular-time modeling formalism.

2. MIMIC-IV substrate and cohort construction

The benchmark uses MIMIC-IV v2.2, released in January 2023, covering 2008–2019. The paper notes that MIMIC-IV contains data for 383,220 patients admitted to an ICU or the emergency department. Its two major modules are hosp, which contains hospital-wide EHR data, and icu, which contains ICU-specific information system data. Key identifiers include subject_id, hadm_id, and stay_id; the paper states that stay_id supersedes icustay_id and can track stays across different hospital areas (Bui et al., 2024).

The coding environment differs materially from MIMIC-III. MIMIC-IV contains both ICD-9 and ICD-10, whereas MIMIC-III contained ICD-9 only. The paper cites counts from Nguyen et al., 2023 of 209,359 admissions with ICD-9 and 122,317 with ICD-10. In the benchmark, this mixed-coding setting is addressed by converting ICD-9 to ICD-10 and then grouping diagnoses to reduce dimensionality (Bui et al., 2024).

For benchmark experiments, only ICU data are used. The cohort is filtered to admissions "Admitted for Chronic Kidney Disease," and the input window is the first 48 hours of data after ICU admission. The dynamic time-series component consists of labs and vitals, while diagnoses supply static or categorical information. For the mortality task, the benchmark uses 407 labs/vitals features and 1034 diagnosis features; for the LOS task, it uses 408 labs/vitals features and 1117 diagnosis features (Bui et al., 2024).

The dataset is public and de-identified, and access is via PhysioNet. The paper does not detail the access workflow and directs readers to PhysioNet for current terms and requirements.

3. Extraction and preprocessing pipeline

SWE-MIMIC-Bench is built on Gupta et al. (2022), "An Extensive Data Processing Pipeline for MIMIC-IV," which the paper identifies as the exact foundation for extraction, preprocessing, and evaluation. Figure 1 in the paper reproduces the pipeline overview, and the authors report successful use with MIMIC-IV v2.2 even though the original pipeline natively supported v2.0 (Bui et al., 2024).

The configuration summarized in Table 1 is the operational definition of the benchmark. Extraction is restricted to MIMIC-IV v2.2, the ICU module only, the Chronic Kidney Disease cohort, and diagnoses together with labs/vitals tables. Preprocessing comprises ICD-9 to ICD-10 conversion, diagnosis grouping, feature summary reports of mean frequency and missingness, feature selection for labs/vitals, outlier removal with threshold 0.98, time-series regularization to 2-hour bins, forward fill and mean imputation, and restriction to the first 48 hours of admission (Bui et al., 2024).

No new mathematical formulation for imputation or decay is introduced. The paper explicitly notes that it does not provide new formulas for imputation or learned decay; instead, it uses straightforward forward fill and mean imputation. Irregularity is therefore handled at the data representation level rather than by adding masks, time-delta channels, or continuous-time latent dynamics.

This preprocessing choice has methodological consequences. A plausible implication is that benchmark comparability is strengthened because all baselines consume the same regularized representation, but some of the information present in the original irregular sampling process may be suppressed by construction.

4. Prediction tasks and baseline models

The benchmark contains two tasks. The first is in-ICU mortality prediction, with the goal of predicting risk of in-ICU death from the first 48 hours of ICU data at 2-hour resolution. The second is binary length-of-stay prediction, framed as whether ICU LOS exceeds 3 days. LOS is defined as days in ICU computed from discharge minus admission, and Table 3 reports the positive class as LOS $> 3$ (Bui et al., 2024).

Readmission prediction is not included. The authors state that readmission typically requires text or note processing with LLMs, which lies outside the benchmark scope (Bui et al., 2024).

The evaluated baselines are XGBoost, LSTM, and TCN. XGBoost uses the binary:logistic objective. The LSTM hyperparameters are: input size $= 152$, hidden size $= 256$, layers $= 2$, dropout $= 0.8$, learning rate $= 0.001$, batch size $= 128$, and epochs $= 20$. The TCN hyperparameters are: layers $= 4$, filters $= 128$, kernel size $= 152$0, dropout $= 152$1, learning rate $= 152$2, batch size $= 152$3, and epochs $= 152$4 (Bui et al., 2024).

The benchmark does not evaluate irregular-time state-of-the-art methods in its experiments, even though the related-work discussion surveys GRU-D, T-LSTM, ODE-RNN, mTAN, HeTVAE, and STraTS. The models actually run are standard XGBoost, LSTM, and TCN implementations on the binned and imputed data.

5. Evaluation protocol and empirical results

Evaluation uses 5-fold cross-validation, with each fold split into 80% train and 20% test. For deep learning models, 10% of the training fold is reserved as validation at the end of each epoch. The paper does not specify whether splitting is performed at patient level or encounter level, and it does not report leakage checks; it states that it uses the Gupta et al. pipeline configurations (Bui et al., 2024).

The reported metrics are ROC-AUC and PR-AUC. No additional metrics such as calibration measures, Brier score, or expected calibration error are reported, and no statistical tests or confidence intervals are provided.

The average results across 5-fold cross-validation are as follows:

Model	In-ICU mortality	LOS $= 152$5 days
XGBoost	ROC-AUC 0.87; PR-AUC 0.52	ROC-AUC 0.76; PR-AUC 0.73
LSTM	ROC-AUC 0.83; PR-AUC 0.45	ROC-AUC 0.72; PR-AUC 0.69
TCN	ROC-AUC 0.82; PR-AUC 0.43	ROC-AUC 0.73; PR-AUC 0.73

The population sizes used in training are 7,910 admissions for mortality and 12,604 admissions for LOS. XGBoost achieves the best performance on both tasks. LSTM and TCN are close to one another: LSTM is slightly better on mortality, while TCN is slightly better on LOS, and TCN matches XGBoost on PR-AUC for LOS (Bui et al., 2024).

Because confidence intervals and formal significance tests are absent, the empirical ranking should be read as descriptive rather than inferential. The benchmark nonetheless highlights the competitiveness of a strong tabular baseline after aggressive regularization and imputation of the original irregular time series.

6. Relation to prior MIMIC benchmarks

The paper positions SWE-MIMIC-Bench against the heterogeneous MIMIC-III benchmark literature. Prior MIMIC-III efforts covered tasks such as in-hospital mortality, short-term and long-term mortality, phenotyping, decompensation, readmission, and LOS, but they differed in window definitions, task formulations, and model sets. Examples named in the paper include logistic regression, random forest, SAPS II, SOFA, feed-forward networks, RNN/GRU/LSTM variants, and multitask learning setups (Bui et al., 2024).

What is new for MIMIC-IV, and consequential for benchmarking, is the modular structure (hosp and icu), updated identifiers (stay_id), longer coverage from 2008 to 2019, ICD-10 availability alongside ICD-9, and improved table organization. These characteristics require retooled extraction, grouping, and harmonization. SWE-MIMIC-Bench addresses these requirements through the Gupta et al. pipeline and the explicit configuration reported in Table 1 (Bui et al., 2024).

The benchmark therefore occupies a distinct position: it is not merely a rerun of MIMIC-III tasks on a newer database, but a re-specified benchmark whose preprocessing assumptions are adapted to MIMIC-IV’s organization and coding environment.

7. Reproducibility, limitations, and extensions

The reproducibility recipe is explicit. To reproduce the benchmark, one must obtain MIMIC-IV v2.2 through PhysioNet, configure the Gupta et al. pipeline for ICU-only extraction, the Chronic Kidney Disease cohort, diagnoses plus labs/vitals, ICD-9 to ICD-10 mapping with diagnosis grouping, labs/vitals feature selection, outlier removal threshold 0.98, 2-hour bins over the first 48 hours, and forward fill plus mean imputation; construct labels for in-ICU mortality and LOS $= 152$6; split into 5 folds with 80% train and 20% test and reserve 10% inner validation for deep learning models; train XGBoost, LSTM, and TCN with the reported settings; and evaluate using ROC-AUC and PR-AUC averaged across folds (Bui et al., 2024).

The paper does not provide a dedicated code repository for the benchmark or pretrained models. It relies on Gupta et al.’s pipeline and on the concrete settings in Tables 1 and 2. Seeds and compute are not reported (Bui et al., 2024).

Several limitations are explicit. The cohort is disease-specific, limited to ICU patients with Chronic Kidney Disease, which may constrain generalizability to broader ICU or hospital populations. Only structured diagnoses and labs/vitals are used; notes and other modalities such as images are excluded. The benchmark covers only two binary tasks, mortality and LOS $= 152$7, and does not include phenotyping, decompensation, or readmission. Bias and fairness audits and label noise analyses are not reported. The split strategy is described, but patient-level separation and time-wise leakage risks are not explicitly discussed (Bui et al., 2024).

The benchmark is also intentionally extensible. The paper recommends keeping the extraction, preprocessing, and cross-validation protocol unchanged when adding new baselines, and ensuring that new models consume the same 2-hour binned sequences and feature sets to preserve comparability. For irregular-time models such as GRU-D, T-LSTM, ODE-RNN, STraTS, mTAN, or HeTVAE, the paper notes that one may bypass heavy imputation by adding masks, time-deltas, or continuous-time encoders, while still reporting results under the same cohort, time window, and splits. It also recommends standardized reporting of ROC-AUC and PR-AUC for each fold and the mean across folds, together with the cohort, window, binning, feature groups, imputation method, outlier handling, and training hyperparameters (Bui et al., 2024).