MIMIC-Multimodal: A Clinical Data Ecosystem

• MIMIC-Multimodal is a comprehensive ecosystem of clinical datasets that aligns structured and unstructured data for joint inference in ICU settings.
• It employs diverse fusion architectures and representation learning techniques to improve prediction accuracy and interpretability despite partial observability.
• Empirical results consistently show that integrating modalities such as clinical notes, radiology, and vital signs enhances performance on key tasks like mortality and length-of-stay prediction.

MIMIC-Multimodal denotes a family of datasets, benchmarks, and model formulations that align heterogeneous clinical modalities within the MIMIC ecosystem for joint inference. In current usage, the term may refer narrowly to a benchmark built on MIMIC‑IV v2.2 with four modalities and 73,181 ICU stays, or more broadly to multimodal learning on linked resources such as MIMIC‑III, MIMIC‑IV, MIMIC‑CXR, MIMIC‑IV‑Note, MIMIC‑III Waveform Database Matched Subset, MIMIC‑IV‑ECG, and external cohorts such as eICU. Across these works, the common objective is to couple structured time series with unstructured notes, images, waveforms, or ECG-derived artifacts in order to improve prediction, representation learning, offline decision making, and interpretability under clinically realistic missingness (Yu et al., 20 Jul 2025, Xu et al., 2024, Jorf et al., 7 Aug 2025, Liang et al., 23 Apr 2026, Zhang et al., 21 Jul 2025, Li et al., 2023).

1. Scope, resources, and data organization

In the clinical literature, “MIMIC-Multimodal” is not a single raw database but an organizing label for several aligned resources. One benchmark constructs an ICU-stay-level master dataset from the Hospital, ICU, Chest X-ray, and Notes modules of MIMIC‑IV v2.2, yielding 73,181 ICU stays, 50,920 unique patients, and 66,239 hospital admissions, with four key modalities: structured demographics, time-series vital signs, chest X-ray images, and free-text radiology reports (Yu et al., 20 Jul 2025). FlexCare defines a unified prediction setting over MIMIC‑IV, MIMIC‑CXR‑JPG, and MIMIC‑IV‑NOTE for six tasks with pervasive missing modalities and non-overlapping labels (Xu et al., 2024). MedPatch constructs a four-way fusion setting over MIMIC‑IV, MIMIC‑CXR, and MIMIC‑IV‑Note using structured EHR, chest X-rays, radiology reports, and discharge notes (Jorf et al., 7 Aug 2025). CareBench standardizes cohorts from MIMIC‑IV v2.2 and MIMIC‑CXR into a base cohort of 26,947 ICU stays and a matched EHR+CXR subset of 7,149 ICU stays (Yin et al., 27 Feb 2026). MEETI extends MIMIC‑IV‑ECG into a four-way aligned ECG corpus with 784,680 ECG records from 160,597 patients (Zhang et al., 21 Jul 2025). X‑MMP links MIMIC‑III with the MIMIC‑III Waveform Database Matched Subset and constructs a tri-modal mortality cohort of 4,729 ICU stays with discrete events, clinical notes, and vital sign waveforms (Li et al., 2023).

Resource	Modalities	Primary use
MIMIC‑Multimodal benchmark	Demographics, vitals, CXR, radiology reports	Mortality, LOS
FlexCare	Time-series, CXR, notes	Multitask prediction
MedPatch	EHR, CXR, radiology reports, discharge notes	Mortality, phenotyping
MEETI	ECG waveform, ECG image, beat-level features, interpretation text	ECG multimodal learning
X‑MMP cohort	Discrete events, notes, waveforms	Mortality, explainability

This resource diversity implies that MIMIC-Multimodal is best understood as an ecosystem of aligned clinical multimodal benchmarks rather than a single canonical schema. A plausible implication is that comparability across papers depends as much on alignment policy and cohort construction as on model architecture.

2. Modalities, temporal alignment, and episode construction

The underlying modalities span several temporal and structural regimes. In ICU sepsis modeling, structured measurements comprise 8 vitals, 8 labs, and static features, while textual streams include nursing notes, radiology reports, and microbiology reports; time is discretized into 4-hour decision steps over the first 72 hours in ICU, $H=18$, with irregular within-step times $\mathcal{T}_h$ (Liang et al., 23 Apr 2026). In the MIMIC‑Multimodal benchmark, only the first 24 hours of ICU stay are used, with irregularly sampled measurements transformed into fixed-dimensional representations by averaging values over 1-hour intervals or encoded by GRU or Moment (Yu et al., 20 Jul 2025). MedPatch follows a 48-hour ICU horizon for mortality and a full-stay setting for phenotyping, pairing EHR, chest radiographs, and reports under partial modality availability (Jorf et al., 7 Aug 2025). CareBench also fixes the first 48 hours as the observation window for phenotyping, in-hospital mortality, and remaining length-of-stay prediction, aligning the most recent frontal AP CXR before the prediction time point (Yin et al., 27 Feb 2026). X‑MMP uses the first 24 hours after ICU admission, resamples discrete events hourly, and resamples vital-sign waveforms to once every three minutes, producing 480 values per channel over 24 hours (Li et al., 2023). MEETI operates on 10-second 12-lead ECGs at $f_s = 500$ Hz, aligning raw waveforms, 300 dpi plotted images, beat-level FeatureDB outputs, and GPT‑4o interpretations by preserved subject_id and study_id identifiers (Zhang et al., 21 Jul 2025).

These alignment choices are not interchangeable. The literature therefore treats multimodality not merely as heterogeneous feature concatenation, but as the joint handling of heterogeneous sampling densities, distinct observation processes, and episode-level linkage constraints.

3. Fusion architectures and representation learning

Model families in this area range from simple late fusion to task-conditioned cross-modal encoders and latent-state models. The MIMIC‑Multimodal benchmark adopts a deliberately modular baseline: unimodal embeddings are extracted independently, concatenated, and passed to logistic regression, making representation quality directly comparable across modality-specific foundation models (Yu et al., 20 Jul 2025). FlexCare replaces parallel multitask prediction with asynchronous single-task prediction and introduces a task-agnostic multimodal information extraction module, modality-combination tokens, a masked Transformer encoder, a task/modality-aware Mixture of Experts, and task-guided hierarchical multimodal fusion (Xu et al., 2024). MedPatch uses a multi-stage fusion strategy that combines unimodal predictors, confidence-guided token patching, joint fusion, a missingness-aware module, and late fusion over $\hat{y}_{\text{high}}$, $\hat{y}_{\text{low}}$, $\hat{y}_{\text{miss}}$, and unimodal outputs (Jorf et al., 7 Aug 2025). EMERGE first generates a task-relevant textual summary through retrieval-augmented generation and then performs bi-directional cross-attention between the time-series representation and the fused text representation derived from raw notes and the generated summary (Zhu et al., 2024). X‑MMP uses three modality-specific Transformer encoders for discrete events, ClinicalBERT-based note embeddings, and vital signs, concatenates the pooled modality representations, and predicts mortality with a feed-forward network (Li et al., 2023). In resilient vision-tabular learning, intermediate fusion is implemented by concatenating modality tokens and applying masked self-attention in a multimodal Transformer, rather than switching among separate unimodal and multimodal models (Caruso et al., 12 May 2026).

A recurring pattern is the move from static late fusion toward architectures that preserve modality-specific encoders while learning controlled cross-modal interactions. This suggests that the central design problem is not whether to fuse, but where and under what constraints fusion should occur.

4. Informative missingness, modality absence, and latent state modeling

A defining technical feature of MIMIC-Multimodal research is that missingness is treated as endogenous. In multimodal sepsis modeling, observation processes are explicitly not random: sicker patients receive more frequent vitals and labs and more frequent and more urgent notes, and structured and textual modalities arise under different recording processes. The framework therefore augments GRU‑D with explicit monitoring statistics and introduces a documentation-process factor $F_h^{\text{doc}}$ built from modality-level presence indicators, note recency, and documentation density (Liang et al., 23 Apr 2026). FlexCare quantifies this problem directly: for IHM, Missing Img is 76.40% and Missing Note is 7.49%; for LOS, Missing Img is 85.16%; for DIA, Missing TS is 76.34% and Missing Note is 32.56% (Xu et al., 2024). MedPatch adds a missingness indicator vector $\mathbf{a} \in \{0,1\}^4$ and a dedicated classifier $$\hat{y}_{\text{miss}} = g_{\text{miss}}(\mathbf{a})$$, thereby allowing the pattern of observed modalities itself to contribute to prediction (Jorf et al., 7 Aug 2025). Resilient vision-tabular learning uses masked self-attention so that missing tokens neither influence others nor receive influence, blocks gradients through missing image modalities, and applies modality-dropout regularization with $p=0.3$ to simulate partial observability during training (Caruso et al., 12 May 2026). CareBench further shows that multimodal gains on EHR+CXR rapidly degrade under realistic missingness unless methods are explicitly designed to handle incomplete inputs (Yin et al., 27 Feb 2026).

The most elaborate state-space treatment appears in the sepsis offline RL setting, where the learned patient state is $\mathcal{T}_h$0 with $\mathcal{T}_h$1, and $\mathcal{T}_h$2 is an action-conditioned latent belief state governed by a VAE-style latent dynamics model. That work argues that if $\mathcal{T}_h$3 does not depend on action $\mathcal{T}_h$4, then the policy gradient from future rewards with respect to current action is zero, so action-conditioned latent dynamics are necessary for credit assignment under terminal-only rewards (Liang et al., 23 Apr 2026).

5. Tasks, benchmarks, and empirical results

The task spectrum is broad. The MIMIC‑Multimodal benchmark evaluates in-hospital mortality and ICU stay longer than 3 days from the first 24 hours, and finds that integrating multimodal data consistently improved predictive performance compared to models trained exclusively on structured data, with the strongest gains coming from adding radiology text rather than images (Yu et al., 20 Jul 2025). FlexCare addresses six tasks—In-hospital mortality, Length of stay, Decompensation, Phenotyping, 30-day readmission, and Radiology diagnosis—and reports, for example, AUROC 0.8823 for IHM, AUROC 0.9538 for DEC, AUROC 0.7680 for REA, and macro-AUROC 0.6845 for DIA (Xu et al., 2024). MedPatch reports state-of-the-art performance on two benchmark tasks: in-hospital mortality with AUROC 0.876 and AUPRC 0.558 in the trimodal setting, and phenotyping with AUROC 0.862 and AUPRC 0.614 in the quatrimodal setting, outperforming strong baselines and ensembles (Jorf et al., 7 Aug 2025). EMERGE improves multimodal EHR prediction on MIMIC‑III and MIMIC‑IV; on MIMIC‑IV it reports AUROC 89.50 and AUPRC 63.11 for mortality, and AUROC 80.61 and AUPRC 57.28 for 30-day readmission (Zhu et al., 2024).

A distinct branch of the literature frames the problem as sequential decision making rather than static risk prediction. In sepsis treatment, the multimodal patient representation with informative missingness is used for state learning, offline policy learning with Implict Q‑Learning and expectile regression, and post‑72h in-hospital mortality prediction. On MIMIC‑III, the learned policy reaches FQE 0.679 versus 0.528 for clinician behavior, and the adverse outcome predictor reaches AUROC 0.886 for post‑72-hour mortality prediction; on MIMIC‑IV the corresponding FQE is 0.634 versus 0.521 for clinician behavior (Liang et al., 23 Apr 2026). MEETI, by contrast, is primarily a dataset-enabling contribution rather than a benchmark of one predictive model: it supplies 784,680 ECG records from 160,597 patients, 10-second 12-lead waveforms at 500 Hz, high-resolution plotted images, beat-level quantitative parameters for each lead, and detailed interpretation text, thereby supporting multimodal ECG classification, cross-modal retrieval, report generation, and multimodal foundation or pretraining tasks (Zhang et al., 21 Jul 2025).

Taken together, these results indicate that multimodality is most consistently useful when the additional modality supplies nonredundant clinical signal—especially clinical notes and diagnostic text—and when the model explicitly addresses partial observability rather than assuming complete pairing.

6. Interpretability, fairness, and terminological boundaries

Interpretability and trustworthiness have become explicit evaluation axes. X‑MMP introduces Layer‑Wise Propagation to Transformer, defines multimodal relevance conservation by

$\mathcal{T}_h$5

and further decomposes relevance at the feature level through

$\mathcal{T}_h$6

Its case studies highlight salient respiratory rate spikes, note tokens such as “unresponsive” and “dnr/dni,” and waveform tachycardia peaks as positive contributors to mortality prediction (Li et al., 2023). The MIMIC‑Multimodal benchmark evaluates interpretability with SHAP and logistic regression coefficients and reports that time-series data emerged as the most influential modality across both benchmark tasks, while the contribution of imaging increased substantially in the subset of patients without missing modalities (Yu et al., 20 Jul 2025). In the sepsis multimodal state-space model, attention over text emphasizes clinically intuitive phrases and the gating mechanism is highest for “worsening” or “new critical finding” note categories, especially when documentation is fresh or dense (Liang et al., 23 Apr 2026).

Fairness findings are less uniform. One benchmark concludes that incorporating multiple data modalities leads to consistent improvements in predictive performance without introducing additional bias, with demographic parity and equalized odds remaining stable across subgroups (Yu et al., 20 Jul 2025). CareBench reaches a different conclusion: multimodal fusion does not inherently improve fairness, and subgroup disparities mainly arise from unequal sensitivity across demographic groups (Yin et al., 27 Feb 2026). This suggests that fairness behavior is benchmark-dependent and may be governed by cohort construction, missingness patterns, and the relative dominance of EHR versus imaging modalities rather than by multimodality alone.

The term itself also has terminological boundaries. In arXiv usage, “MIMIC” or “MiMIC” can denote unrelated multimodal work such as vision–LLM inversion, universal multimodal retrieval, or a financial earnings-call dataset (Jain et al., 11 Aug 2025, Li et al., 23 Apr 2026, Ghosh et al., 12 Apr 2025). Within clinical machine learning, however, MIMIC-Multimodal specifically denotes multimodal research built on the MIMIC family of critical-care and diagnostic resources, with its central technical themes being alignment, partial observability, fusion under modality imbalance, and clinically grounded evaluation.