HGDC-Fuse: Clinical Multi-modal Fusion Framework

• The paper introduces HGDC-Fuse, a clinical multi-modal fusion framework that creates disease-specific fused representations using a patient-centric heterogeneous graph.
• It aggregates temporal CXR data and similar-patient EHR information via multi-head attention and temporal weighting to handle modality missingness and asynchrony.
• The framework employs disease correlation-guided attention to resolve modality inconsistency, resulting in robust multi-label prediction performance across 25 disease phenotypes.

Searching arXiv for the primary paper and a key related baseline. I’ll look up the cited arXiv entries to ground the article in the current literature. HGDC-Fuse is a clinical multi-modal fusion framework for multi-disease prediction from electronic health records (EHR) and chest X-ray images (CXR) in realistic ICU settings. Introduced in "Clinical Multi-modal Fusion with Heterogeneous Graph and Disease Correlation Learning for Multi-Disease Prediction" (Jiang et al., 19 Sep 2025), it is designed to make multi-modal prediction robust to missing modalities, noisy/incomplete data, temporal asynchrony, and conflicting evidence across modalities for different diseases. Its architecture combines a patient-centric multi-modal heterogeneous graph, a heterogeneous graph learning module, and a disease correlation-guided attention mechanism, with the aim of producing disease-specific fused representations rather than a single generic multimodal embedding.

1. Clinical task and motivating failure modes

HGDC-Fuse is evaluated in an ICU setting built from MIMIC-IV and MIMIC-CXR, where each patient has one time-series EHR record from the first 48 hours of ICU stay and zero, one, or multiple CXRs acquired irregularly during the stay. The prediction target is multi-label disease prediction over 25 disease phenotypes using 17 EHR variables, CXR images, or both modalities together (Jiang et al., 19 Sep 2025).

The framework is motivated by three failure modes that the underlying paper treats as central obstacles to real-world deployment. The first is modality missingness and noise: some patients have no CXR, some have sparse EHR variables, and both modalities can be noisy. The second is temporal asynchrony: EHR and imaging are not collected on the same schedule, yet many prior methods fuse EHR only with the latest CXR and thereby ignore the CXR sequence, cross-modal temporal dependencies, and disease progression cues in image timelines. The third is modality inconsistency across diseases: EHR and CXR do not support all diseases equally, and a modality that is informative for one phenotype may be weak or misleading for another.

Within this formulation, HGDC-Fuse is explicitly positioned against fusion strategies that impute or synthesize missing modalities, disentangle shared and modality-specific features, or rely on generic attention-based weighting. The paper argues that such approaches can amplify noise, discard useful modality information, or fail to represent disease-specific modality importance. A plausible implication is that HGDC-Fuse treats robustness not as a preprocessing problem but as a representational problem, embedding incompleteness, asynchrony, and inconsistency directly into the graph and fusion design.

2. Patient-centric heterogeneous graph formulation

For each patient $s$, HGDC-Fuse constructs a personalized heterogeneous graph

$\mathcal{G}_s = (\mathcal{V}_s, \mathcal{E}_s).$

The graph contains two node types: an EHR node $n_s^E$, which is the target node for the patient, and one or more CXR nodes $n_s^C$, with one node per available chest X-ray image (Jiang et al., 19 Sep 2025).

The EHR sequence $E_s \in \mathbb{R}^{T_n \times J}$ is encoded with a Transformer to obtain $\mathbf{h}_s^{ehr}$. Each image $C_s^k$ is encoded with ResNet-50 to obtain $\mathbf{h}_s^{cxr,k}$. This encoding stage is followed by graph construction in which the EHR node functions as the central target node and receives information from two distinct sources: intra-patient CXR nodes, used to capture cross-modal temporal evidence, and inter-patient EHR neighbors, used to exploit patient similarity.

Two edge families define the graph structure. First, CXR $\rightarrow$ EHR edges connect each available image to the patient’s EHR node: $$\mathcal{E}^{cxr \rightarrow ehr}_s = \left\{ (n_i, v_i, \Delta T(n_i)) \mid n_i \in \mathcal{V}_s^{cxr}, v_i = n_s^E \right\}.$$ Here $\mathcal{G}_s = (\mathcal{V}_s, \mathcal{E}_s).$0 denotes relative acquisition time and is the mechanism by which temporal asynchrony is encoded. Second, EHR–EHR edges connect the patient to similar patients in the same batch: $\mathcal{G}_s = (\mathcal{V}_s, \mathcal{E}_s).$1 where $\mathcal{G}_s = (\mathcal{V}_s, \mathcal{E}_s).$2 is a similarity threshold.

The immediate significance of this graph is architectural separation of heterogeneous evidence sources. Rather than collapsing EHR and imaging into a single synchronous sequence, HGDC-Fuse preserves modality semantics and treats missingness and asynchrony as graph-topological properties. This design is described as especially useful when one modality is missing or sparse.

3. Message passing, temporal weighting, and robustness to incomplete modalities

The heterogeneous graph learning module aggregates information from two neighbor types: similar patients’ EHR nodes and the patient’s own CXR nodes. It produces two messages, $\mathcal{G}_s = (\mathcal{V}_s, \mathcal{E}_s).$3 from similar-patient EHR nodes and $\mathcal{G}_s = (\mathcal{V}_s, \mathcal{E}_s).$4 from intra-patient CXR nodes (Jiang et al., 19 Sep 2025).

For EHR–EHR aggregation, HGDC-Fuse uses multi-head attention over the neighbor set $\mathcal{G}_s = (\mathcal{V}_s, \mathcal{E}_s).$5: $\mathcal{G}_s = (\mathcal{V}_s, \mathcal{E}_s).$6 The corresponding attention coefficients are

$\mathcal{G}_s = (\mathcal{V}_s, \mathcal{E}_s).$7

$\mathcal{G}_s = (\mathcal{V}_s, \mathcal{E}_s).$8

with $\mathcal{G}_s = (\mathcal{V}_s, \mathcal{E}_s).$9 the number of attention heads, $n_s^E$0 a learnable attention vector, and $n_s^E$1 a projection matrix. The paper characterizes this as similar to a graph attention mechanism applied to patient neighbors.

For CXR–EHR aggregation, each image embedding is projected as $n_s^E$2 and weighted according to acquisition time: $n_s^E$3

$n_s^E$4

This mechanism uses all available CXRs rather than only the latest image and makes contribution depend on time.

Missing CXR data are handled through masking in the fusion layer. If a patient has no CXR, the CXR branch is masked by setting

$n_s^E$5

This prevents the absent modality from influencing the fused representation. The model’s robustness to incomplete or noisy modalities is attributed to three interacting factors: similar-patient aggregation from EHR–EHR edges, type-specific aggregation that preserves modality semantics, and disease-specific attention that reduces the effect of inconsistent or noisy modalities. In the paper’s interpretation, separate message channels avoid mixing heterogeneous semantics too early.

4. Disease correlation-guided attention and disease-specific fusion

The component that differentiates HGDC-Fuse most sharply from generic multimodal attention is the disease correlation-guided attention layer, introduced to address modality inconsistency across diseases. The central idea is that each disease should attend differently to the patient’s own EHR representation, the message aggregated from similar patients, and the message aggregated from the CXR timeline, while disease correlations should guide that weighting (Jiang et al., 19 Sep 2025).

To model label relations, the method constructs a disease correlation graph in which each node is a disease label. Initial label embeddings are represented as one-hot word embedding vectors: $n_s^E$6 The disease correlation matrix $n_s^E$7 is estimated from training-set conditional co-occurrence statistics: $n_s^E$8 where $n_s^E$9 is the number of samples containing both labels and $n_s^C$0 is the number of samples with label $n_s^C$1. Weak or noisy correlations are removed by thresholding: $n_s^C$2 A two-layer GCN is then applied over the normalized correlation matrix to produce $n_s^C$3, which the paper describes as disease-aware prototypes encoding higher-order disease co-occurrence semantics.

For patient $n_s^C$4, the three latent feature sources are

$n_s^C$5

They are stacked as

$n_s^C$6

For each disease $n_s^C$7, the label embedding $n_s^C$8 is used as the query, with projections

$n_s^C$9

The attention weights are computed with scaled dot-product attention and masking: $E_s \in \mathbb{R}^{T_n \times J}$0 where $E_s \in \mathbb{R}^{T_n \times J}$1 and $E_s \in \mathbb{R}^{T_n \times J}$2 when CXR is missing. The resulting disease-specific fused representation is

$E_s \in \mathbb{R}^{T_n \times J}$3

Prediction for disease $E_s \in \mathbb{R}^{T_n \times J}$4 is performed by a feedforward classifier,

$E_s \in \mathbb{R}^{T_n \times J}$5

and optimization uses binary cross-entropy summed over labels: $E_s \in \mathbb{R}^{T_n \times J}$6

A salient conceptual consequence of this design is that HGDC-Fuse does not learn one shared multimodal patient representation for all labels. Instead, each disease receives its own fused representation, conditioned by label correlation structure. The paper argues that correlated diseases often share evidence patterns, so disease graph embeddings provide a disease-aware prior for modality selection.

5. Experimental protocol, baselines, and reported performance

The empirical study uses 59,344 ICU stays with EHR, of which 10,630 also have CXRs, with an average of 1.89 CXR images per stay. Two evaluation settings are reported: Full, using all stays, and Matched, using only EHR + CXR stays. The split is 7:1:2 for train, validation, and test. Training is implemented in PyTorch 2.5.1 on an NVIDIA GeForce RTX 4090. Batch size is 256 on the full dataset and 64 on the matched dataset. The similarity threshold for the patient graph is $E_s \in \mathbb{R}^{T_n \times J}$7, and the disease correlation threshold is $E_s \in \mathbb{R}^{T_n \times J}$8. When training on the matched subset, 30% of samples with CXR in each batch are randomly removed to improve robustness under missing-modality conditions. Because the labels are highly imbalanced, the main metric is macro-average PRAUC (Jiang et al., 19 Sep 2025).

The reported baselines are Transformer as an EHR-only unimodal baseline, and the multimodal methods MMTM, DAFT, MedFuse, MedFuse-II, and DrFuse. Among these, DrFuse is identified as the strongest prior multimodal baseline and as directly addressing missing modality and modality inconsistency to some extent.

HGDC-Fuse achieved the best macro-PRAUC in all reported settings.

Setting	DrFuse	HGDC-Fuse
trained/testing on matched subset	0.450	0.470
trained on full, tested on matched	0.470	0.489
trained on full, tested on full	0.419	0.434

The paper reports relative improvements over DrFuse of 4.4% on the matched subset, 4.0% on full→matched, and 3.5% on full→full. The accompanying interpretation is that HGDC-Fuse is especially strong under incomplete modality conditions. Since the full setting includes patients without CXR, this suggests that the masking and graph-based aggregation mechanisms contribute to robustness when multimodal availability is nonuniform.

6. Disease-wise behavior, ablations, scope, and terminology

The disease-wise analysis indicates that EHR and CXR contribute differently depending on the disease. The paper notes that DrFuse sometimes underperforms the best unimodal baseline on several diseases, whereas HGDC-Fuse usually improves on the best unimodal score across nearly all labels. Two highlighted examples are other upper respiratory disease and gastrointestinal hemorrhage, for which HGDC-Fuse shows improvements around 55% relative to the best single modality. The paper presents this as evidence that disease-aware fusion is essential (Jiang et al., 19 Sep 2025).

The ablation study, conducted on the matched subset, isolates the contributions of three design choices: similar-patient EHR neighbors, temporal multi-CXR modeling, and disease correlation-guided attention.

Model	PRAUC matched	PRAUC full
w/o HER-EHR	0.4500	0.3812
w/o multi-cxr	0.4506	0.3698
w/o CGA	0.4548	0.3760
HGDC-Fuse	0.4698	0.3860

The reported interpretation is direct. Removing similar-patient neighbors degrades performance, indicating that inter-patient graph context is useful. Using only the latest CXR rather than the full CXR sequence degrades performance, indicating that temporal modeling of CXRs matters. Replacing disease correlation-guided attention with generic self-attention also degrades performance, indicating that label-correlation priors are important for resolving modality inconsistency.

The paper’s stated contributions are threefold: a patient-centric heterogeneous graph for multi-modal clinical fusion that captures temporal asynchrony and missing modalities through cross-modal and inter-patient edges; disease correlation-guided attention that models label co-occurrence and learns disease-specific modality weights; and strong empirical performance on MIMIC-IV/MIMIC-CXR. The paper does not deeply discuss limitations, but a plausible reading of the reported method is that it is currently demonstrated on two modalities only, that disease correlations estimated from training-set co-occurrence may be imperfect, that similar-patient graph construction depends on batch-level cosine similarity and thresholding, and that the formulation is focused on ICU prediction and may require adaptation for other clinical workflows.

A common terminological confusion is possible because the word “fuse” also appears in an unrelated HCI and cognitive-science context in "From Flow to Fuse: A Cognitive Perspective" (Jalife et al., 2021). In that literature, fuse denotes the “fusion of activity-related sensory stimuli and awareness,” whereas in HGDC-Fuse the term names a clinical multimodal fusion framework. The overlap is lexical rather than methodological.

Taken as a whole, HGDC-Fuse is presented as a move beyond generic multimodal fusion toward a design that is explicitly patient-centric, temporally aware, and disease-aware. Within the reported evidence, its advantage derives from combining graph learning for incomplete and asynchronous inputs, temporal weighting over the full imaging history, and disease-correlated attention for disease-specific modality selection.