---
title: 'HGDC-Fuse: Clinical Multi-modal Fusion Framework'
url: https://www.emergentmind.com/topics/hgdc-fuse
type: topic
---

# HGDC-Fuse: Clinical Multi-modal Fusion Framework

Searching arXiv for the primary paper and a key related baseline.
I’ll look up the cited arXiv entries to ground the article in the current literature.
HGDC-Fuse is a clinical multi-modal fusion framework for **multi-disease prediction** from **electronic health records (EHR)** and **chest X-ray images (CXR)** in realistic ICU settings. Introduced in "Clinical Multi-modal Fusion with Heterogeneous Graph and Disease Correlation Learning for Multi-Disease Prediction" [2509.15852], it is designed to make multi-modal prediction robust to **missing modalities, noisy/incomplete data, temporal asynchrony, and conflicting evidence across modalities for different diseases**. Its architecture combines a **patient-centric multi-modal heterogeneous graph**, a **heterogeneous graph learning module**, and a **disease correlation-guided attention** mechanism, with the aim of producing disease-specific fused representations rather than a single generic multimodal embedding.

## 1. Clinical task and motivating failure modes

HGDC-Fuse is evaluated in an ICU setting built from **MIMIC-IV** and **MIMIC-CXR**, where each patient has one **time-series EHR** record from the first **48 hours** of ICU stay and zero, one, or multiple **CXRs** acquired irregularly during the stay. The prediction target is **multi-label disease prediction** over **25 disease phenotypes** using **17 EHR variables**, CXR images, or both modalities together [2509.15852].

The framework is motivated by three failure modes that the underlying paper treats as central obstacles to real-world deployment. The first is **modality missingness and noise**: some patients have no CXR, some have sparse EHR variables, and both modalities can be noisy. The second is **temporal asynchrony**: EHR and imaging are not collected on the same schedule, yet many prior methods fuse EHR only with the **latest** CXR and thereby ignore the CXR sequence, cross-modal temporal dependencies, and disease progression cues in image timelines. The third is **modality inconsistency across diseases**: EHR and CXR do not support all diseases equally, and a modality that is informative for one phenotype may be weak or misleading for another.

Within this formulation, HGDC-Fuse is explicitly positioned against fusion strategies that impute or synthesize missing modalities, disentangle shared and modality-specific features, or rely on generic attention-based weighting. The paper argues that such approaches can amplify noise, discard useful modality information, or fail to represent disease-specific modality importance. A plausible implication is that HGDC-Fuse treats robustness not as a preprocessing problem but as a representational problem, embedding incompleteness, asynchrony, and inconsistency directly into the graph and fusion design.

## 2. Patient-centric heterogeneous graph formulation

For each patient \(s\), HGDC-Fuse constructs a personalized heterogeneous graph
\[
\mathcal{G}_s = (\mathcal{V}_s, \mathcal{E}_s).
\]
The graph contains two node types: an **EHR node** \(n_s^E\), which is the target node for the patient, and one or more **CXR nodes** \(n_s^C\), with one node per available chest X-ray image [2509.15852].

The EHR sequence \(E_s \in \mathbb{R}^{T_n \times J}\) is encoded with a **Transformer** to obtain \(\mathbf{h}_s^{ehr}\). Each image \(C_s^k\) is encoded with **ResNet-50** to obtain \(\mathbf{h}_s^{cxr,k}\). This encoding stage is followed by graph construction in which the EHR node functions as the central target node and receives information from two distinct sources: **intra-patient CXR nodes**, used to capture cross-modal temporal evidence, and **inter-patient EHR neighbors**, used to exploit patient similarity.

Two edge families define the graph structure. First, **CXR \(\rightarrow\) EHR** edges connect each available image to the patient’s EHR node:
\[
\mathcal{E}^{cxr \rightarrow ehr}_s = \left\{ (n_i, v_i, \Delta T(n_i)) \mid n_i \in \mathcal{V}_s^{cxr}, v_i = n_s^E \right\}.
\]
Here \(\Delta T(n_i)\) denotes relative acquisition time and is the mechanism by which temporal asynchrony is encoded. Second, **EHR–EHR** edges connect the patient to similar patients in the same batch:
\[
\mathcal{E}^{ehr\text{-}ehr}_s = \left\{ (n_s^E, n_{s'}^E) \mid \cos(\mathbf{h}_s^{ehr}, \mathbf{h}_{s'}^{ehr}) > \delta \right\},
\]
where \(\delta\) is a similarity threshold.

The immediate significance of this graph is architectural separation of heterogeneous evidence sources. Rather than collapsing EHR and imaging into a single synchronous sequence, HGDC-Fuse preserves modality semantics and treats missingness and asynchrony as graph-topological properties. This design is described as especially useful when one modality is missing or sparse.

## 3. Message passing, temporal weighting, and robustness to incomplete modalities

The heterogeneous graph learning module aggregates information from two neighbor types: similar patients’ EHR nodes and the patient’s own CXR nodes. It produces two messages, \(\mathbf{m}_s^{E \leftarrow E}\) from similar-patient EHR nodes and \(\mathbf{m}_s^{E \leftarrow C}\) from intra-patient CXR nodes [2509.15852].

For EHR–EHR aggregation, HGDC-Fuse uses multi-head attention over the neighbor set \(\mathcal{N}^E(n_s^E)\):
\[
\mathbf{m}_s^{E \leftarrow E} = \bigg\|_{i=1}^H \sum_{n_{s'}^E \in \mathcal{N}^E(s)} \alpha_{ss'}^{(i)} \mathbf{W}_E^{(i)} \mathbf{h}_{s'}^{ehr}.
\]
The corresponding attention coefficients are
\[
e_{ss'}^{(i)} = \text{LeakyReLU}\left(\mathbf{a}^{(i)\top} \left[\mathbf{W}_E^{(i)} \mathbf{h}_s^{ehr} \, \| \, \mathbf{W}_E^{(i)} \mathbf{h}_{s'}^{ehr} \right] \right),
\]
\[
\alpha_{ss'}^{(i)} = \frac{\exp(e_{ss'}^{(i)})}{\sum_{n_{s''}^E \in \mathcal{N}^E(s)} \exp(e_{ss''}^{(i)})},
\]
with \(H\) the number of attention heads, \(\mathbf{a}^{(i)}\) a learnable attention vector, and \(\mathbf{W}_E^{(i)}\) a projection matrix. The paper characterizes this as similar to a graph attention mechanism applied to patient neighbors.

For CXR–EHR aggregation, each image embedding is projected as \(\mathbf{W}_C \mathbf{h}_s^{cxr,j}\) and weighted according to acquisition time:
\[
w_j^{(s)} = \frac{\exp( \Delta T(n_s^{C,j}))}{\sum_{k=1}^{K} \exp( \Delta T(n_s^{C,k}))},
\]
\[
\mathbf{m}_s^{E \leftarrow C} = \sum_{j=1}^{K} w_j^{(s)} \cdot \mathbf{W}_C \mathbf{h}_s^{cxr,j}.
\]
This mechanism uses **all available CXRs** rather than only the latest image and makes contribution depend on time.

Missing CXR data are handled through masking in the fusion layer. If a patient has no CXR, the CXR branch is masked by setting
\[
m_{s,3} = -\infty.
\]
This prevents the absent modality from influencing the fused representation. The model’s robustness to incomplete or noisy modalities is attributed to three interacting factors: **similar-patient aggregation** from EHR–EHR edges, **type-specific aggregation** that preserves modality semantics, and **disease-specific attention** that reduces the effect of inconsistent or noisy modalities. In the paper’s interpretation, separate message channels avoid mixing heterogeneous semantics too early.

## 4. Disease correlation-guided attention and disease-specific fusion

The component that differentiates HGDC-Fuse most sharply from generic multimodal attention is the **disease correlation-guided attention layer**, introduced to address **modality inconsistency** across diseases. The central idea is that each disease should attend differently to the patient’s own EHR representation, the message aggregated from similar patients, and the message aggregated from the CXR timeline, while disease correlations should guide that weighting [2509.15852].

To model label relations, the method constructs a **disease correlation graph** in which each node is a disease label. Initial label embeddings are represented as one-hot word embedding vectors:
\[
\mathbf{Z} = [\mathbf{z}_1, \dots, \mathbf{z}_N]^\top \in \mathbb{R}^{N \times d}.
\]
The disease correlation matrix \(\mathbf{A}\) is estimated from training-set conditional co-occurrence statistics:
\[
A_{ij} = \frac{\text{co-occur}(i, j)}{\text{count}(i)}, \quad i \neq j,
\]
where \(\text{co-occur}(i,j)\) is the number of samples containing both labels and \(\text{count}(i)\) is the number of samples with label \(i\). Weak or noisy correlations are removed by thresholding:
\[
A_{ij} =
\begin{cases}
1, & \text{if } A_{ij} \ge \tau \\
0, & \text{otherwise}.
\end{cases}
\]
A two-layer GCN is then applied over the normalized correlation matrix to produce \(\tilde{\mathbf{Z}} \in \mathbb{R}^{N \times d'}\), which the paper describes as **disease-aware prototypes** encoding higher-order disease co-occurrence semantics.

For patient \(s\), the three latent feature sources are
\[
\mathbf{h}_s^{ehr}, \quad \mathbf{m}_s^{E \leftarrow E}, \quad \mathbf{m}_s^{E \leftarrow C}.
\]
They are stacked as
\[
\mathbf{T}_s = [\mathbf{h}_s^{ehr}, \mathbf{m}_s^{E \leftarrow E}, \mathbf{m}_s^{E \leftarrow C}] \in \mathbb{R}^{3 \times d}.
\]
For each disease \(n\), the label embedding \(\mathbf{z}_n\) is used as the query, with projections
\[
\mathbf{q}_n = \mathbf{W}_q \mathbf{z}_n,\quad \mathbf{K} = \mathbf{W}_k \mathbf{T}_s,\quad \mathbf{V} = \mathbf{W}_v \mathbf{T}_s.
\]
The attention weights are computed with scaled dot-product attention and masking:
\[
\boldsymbol{\alpha}_n = \mathrm{softmax}\!\left(\frac{\mathbf{K}\mathbf{q}_n^\top}{\sqrt{d} + \mathbf{m}_s} \right),
\]
where \(\mathbf{m}_s \in \{0,-\infty\}^3\) and \(m_{s,3} = -\infty\) when CXR is missing. The resulting disease-specific fused representation is
\[
\tilde{\mathbf{h}}_n = \boldsymbol{\alpha}_n^\top \mathbf{V} \in \mathbb{R}^d.
\]

Prediction for disease \(n\) is performed by a feedforward classifier,
\[
\hat{y}_n = \psi_n(\tilde{\mathbf{h}}_n),
\]
and optimization uses binary cross-entropy summed over labels:
\[
\mathcal{L} = \sum_{n=1}^{N} y_n \log(\hat{y}_n) + (1 - y_n) \log(1 - \hat{y}_n).
\]

A salient conceptual consequence of this design is that HGDC-Fuse does not learn one shared multimodal patient representation for all labels. Instead, each disease receives its own fused representation, conditioned by label correlation structure. The paper argues that correlated diseases often share evidence patterns, so disease graph embeddings provide a disease-aware prior for modality selection.

## 5. Experimental protocol, baselines, and reported performance

The empirical study uses **59,344** ICU stays with EHR, of which **10,630** also have CXRs, with an average of **1.89 CXR images per stay**. Two evaluation settings are reported: **Full**, using all stays, and **Matched**, using only EHR + CXR stays. The split is **7:1:2** for train, validation, and test. Training is implemented in **PyTorch 2.5.1** on an **NVIDIA GeForce RTX 4090**. Batch size is **256** on the full dataset and **64** on the matched dataset. The similarity threshold for the patient graph is \(\delta = 0.6\), and the disease correlation threshold is \(\tau = 0.4\). When training on the matched subset, **30% of samples with CXR in each batch** are randomly removed to improve robustness under missing-modality conditions. Because the labels are highly imbalanced, the main metric is **macro-average PRAUC** [2509.15852].

The reported baselines are **Transformer** as an EHR-only unimodal baseline, and the multimodal methods **MMTM**, **DAFT**, **MedFuse**, **MedFuse-II**, and **DrFuse**. Among these, **DrFuse** is identified as the strongest prior multimodal baseline and as directly addressing missing modality and modality inconsistency to some extent.

HGDC-Fuse achieved the best macro-PRAUC in all reported settings.

| Setting | DrFuse | HGDC-Fuse |
|---|---:|---:|
| trained/testing on matched subset | 0.450 | 0.470 |
| trained on full, tested on matched | 0.470 | 0.489 |
| trained on full, tested on full | 0.419 | 0.434 |

The paper reports relative improvements over DrFuse of **4.4%** on the matched subset, **4.0%** on full→matched, and **3.5%** on full→full. The accompanying interpretation is that HGDC-Fuse is especially strong under incomplete modality conditions. Since the full setting includes patients without CXR, this suggests that the masking and graph-based aggregation mechanisms contribute to robustness when multimodal availability is nonuniform.

## 6. Disease-wise behavior, ablations, scope, and terminology

The disease-wise analysis indicates that EHR and CXR contribute differently depending on the disease. The paper notes that **DrFuse** sometimes underperforms the best unimodal baseline on several diseases, whereas **HGDC-Fuse** usually improves on the best unimodal score across nearly all labels. Two highlighted examples are **other upper respiratory disease** and **gastrointestinal hemorrhage**, for which HGDC-Fuse shows improvements around **55%** relative to the best single modality. The paper presents this as evidence that disease-aware fusion is essential [2509.15852].

The ablation study, conducted on the matched subset, isolates the contributions of three design choices: similar-patient EHR neighbors, temporal multi-CXR modeling, and disease correlation-guided attention.

| Model | PRAUC matched | PRAUC full |
|---|---:|---:|
| w/o HER-EHR | 0.4500 | 0.3812 |
| w/o multi-cxr | 0.4506 | 0.3698 |
| w/o CGA | 0.4548 | 0.3760 |
| HGDC-Fuse | 0.4698 | 0.3860 |

The reported interpretation is direct. Removing **similar-patient neighbors** degrades performance, indicating that inter-patient graph context is useful. Using only the **latest CXR** rather than the full CXR sequence degrades performance, indicating that temporal modeling of CXRs matters. Replacing **disease correlation-guided attention** with generic self-attention also degrades performance, indicating that label-correlation priors are important for resolving modality inconsistency.

The paper’s stated contributions are threefold: a **patient-centric heterogeneous graph for multi-modal clinical fusion** that captures temporal asynchrony and missing modalities through cross-modal and inter-patient edges; **disease correlation-guided attention** that models label co-occurrence and learns disease-specific modality weights; and **strong empirical performance** on MIMIC-IV/MIMIC-CXR. The paper does not deeply discuss limitations, but a plausible reading of the reported method is that it is currently demonstrated on **two modalities** only, that disease correlations estimated from training-set co-occurrence may be imperfect, that similar-patient graph construction depends on batch-level cosine similarity and thresholding, and that the formulation is focused on ICU prediction and may require adaptation for other clinical workflows.

A common terminological confusion is possible because the word “fuse” also appears in an unrelated HCI and cognitive-science context in "From Flow to Fuse: A Cognitive Perspective" [2107.08268]. In that literature, **fuse** denotes the “fusion of activity-related sensory stimuli and awareness,” whereas in HGDC-Fuse the term names a clinical multimodal fusion framework. The overlap is lexical rather than methodological.

Taken as a whole, HGDC-Fuse is presented as a move beyond generic multimodal fusion toward a design that is explicitly **patient-centric**, **temporally aware**, and **disease-aware**. Within the reported evidence, its advantage derives from combining graph learning for incomplete and asynchronous inputs, temporal weighting over the full imaging history, and disease-correlated attention for disease-specific modality selection.

Source: https://www.emergentmind.com/topics/hgdc-fuse