Is Clinical Text Enough? A Multimodal Study on Mortality Prediction in Heart Failure Patients

Abstract: Accurate short-term mortality prediction in heart failure (HF) remains challenging, particularly when relying on structured electronic health record (EHR) data alone. We evaluate transformer-based models on a French HF cohort, comparing text-only, structured-only, multimodal, and LLM-based approaches. Our results show that enriching clinical text with entity-level representations improves prediction over CLS embeddings alone, and that supervised multimodal fusion of text and structured variables achieves the best overall performance. In contrast, LLMs perform inconsistently across modalities and decoding strategies, with text-only prompts outperforming structured or multimodal inputs. These findings highlight that entity-aware multimodal transformers offer the most reliable solution for short-term HF outcome prediction, while current LLM prompting remains limited for clinical decision support.

• The paper demonstrates that incorporating entity-level text embeddings and structured data into multimodal transformers improves HF mortality prediction, achieving up to 48.5% F1 and 81.5% AUC.
• It systematically compares unimodal and multimodal approaches, revealing that advanced fusion strategies, such as gated attention, deliver statistically robust performance enhancements over text-only models.
• The study also highlights limitations of current instruction-tuned LLMs, which underperform and show inconsistent adherence when applied to multimodal clinical risk prediction.

Multimodal Mortality Prediction in Heart Failure: Integrating Clinical Text, Structured Data, and LLMs

Introduction

Short-term mortality prediction in heart failure (HF) is inherently challenging due to heterogeneous signals spanning structured EHR variables and unstructured clinical notes. "Is Clinical Text Enough? A Multimodal Study on Mortality Prediction in Heart Failure Patients" (2604.01924) specifically investigates the relative and joint contributions of French-language clinical text and structured variables, and evaluates modern transformer-based and generative LLM approaches. The work is distinguished by its systematic comparison of unimodal and multimodal models, as well as entity-level enrichment of textual representations and a critical analysis of LLM instruction-following and modality adaptation.

The study leverages a closed French hospital dataset, incorporating rigorous NER annotation, model ablation, and paired statistical significance testing.

Dataset and Preprocessing

The dataset consists of 2,254 hospitalizations for HF from a Parisian cardiology department, each annotated with a 3-month post-discharge mortality label. The cohort is characterized by significant class imbalance: ~11% positive cases and a median patient age of 81 years. Each admission contains physician-authored clinical notes (free text) and 115 structured variables (demographics, labs, treatments, procedures), refined to 41 features through LASSO-based feature selection.

Figure 1: Distribution of clinical notes per size, indicating substantial variability in narrative text length across the cohort.

Clinical notes underwent an advanced entity recognition pipeline, beginning with heuristic-based pre-annotation (leveraging UMLS, French medical lexicons, regex, and morphosyntactic triggers), followed by dual-phase manual correction (expert refinement, larger-scale non-expert corrections) and subsequent supervised NER (using fine-tuned BERT models). High inter-annotator agreement was achieved for salient categories. This entity-enriched text corpus is central to the proposed multimodal fusion strategies.

Methodological Framework

The central machine learning task is formulated as binary classification: predicting $y=1$ for post-discharge mortality within three months and $y=0$ otherwise.

Unimodal Text Approaches

Two French biomedical transformer models—CamemBERT-bio and DrBERT—are deployed to represent clinical notes. Baseline textual encoding uses the CLS token’s contextual embedding. Augmented approaches incorporate entity-level embeddings from NER, with five fusion strategies: average, sum, concatenation, learned weighted fusion, and gated fusion (sigmoid gating per entity type). Logistic regression serves as the classification head.

Unimodal Structured Approaches

Structured variables are evaluated via two logistic regression models (reference and LASSO-selected variables), with optimal preprocessing (imputation, scaling) and class weighting.

Multimodal Fusion

Three fusion classes are compared:

• Simple (feature concatenation, late fusion by averaging or stacking)
• Learnable gating (trainable weights, MLP classifier)
• Cross-modal (dual cross-attention and variants, notably bi-directional/symmetric attention with entities and structured data)

Textual input can be either CLS-only or entity-augmented.

Generative LLM Evaluation

Three instruction-tuned LLMs (Mistral-7B, Qwen2.5-7B, MedGemma-4B) are prompted to predict outcomes using: (i) text only, (ii) structured input, or (iii) both. Both direct tabular and templated textual encodings of structured data are tested. Generation strategies include greedy and output-constrained decoding.

Results

Text Representations

The inclusion of entity-level embeddings improves all metrics over the CLS-only baseline. Weighted and gated fusions with CamemBERT-bio yield the highest F1 (39.5%) and ROC-AUC (75.8%). The gain is less pronounced for DrBERT, suggesting model and corpus sensitivity to domain and language variants.

Structured-Only Models

Structured variables alone (41-variable logistic regression) outperform text-only counterparts, reaching AUC = 79.0%. Key risk factors (age, urea, CRP, troponin; negative coefficients for evidence-based medications) align with clinical prior knowledge, reinforcing the structured model's external validity.

Multimodal Fusion

All fusion strategies outperform their unimodal baselines. The strongest F1 and AUC are consistently achieved with learnable (gated or attention-based) fusion utilizing entity-level text, with CamemBERT-bio gated fusion (F1 = 48.5%, AUC = 81.5%) and DrBERT Gated Dual Cross Attention (F1 = 46.8%, AUC = 79.7%). Late fusion (stacking) provides the highest AUC (83.8%) for CamemBERT-bio, demonstrating the importance of post-hoc ensemble mechanisms. This demonstrates that computationally tractable multimodal transformers leveraging fine-grained clinical semantics systematically improve discrimination for short-term mortality.

Paired significance tests confirm improvements from multimodal late fusion over best unimodal models are statistically robust (p < 0.05).

LLMs and Prompted Inference

All LLMs perform substantially worse than supervised models. Even for text-only queries, the highest F1 is ~26%. Multimodal prompts induce single-class collapse; for example, Qwen with output constraints always predicts class 1. Instruction adherence varies by model and input type, with some LLMs outputting extraneous text in violation of prompts. Transcribing structured data to templated text only marginally improves interpretability and adherence, with little to no effect on actual predictive quality.

LLMs remain unreliable for binary outcome prediction given current prompt engineering and model sizes, particularly when applied to structured or multimodal inputs. Novel interaction and fine-tuning methods are needed to close this gap.

Practical and Theoretical Implications

The evidence supports several essential conclusions:

• Entity-level augmentation: Incorporating entity-level representations into clinical text encoding significantly enhances predictive performance compared to CLS-only representations, especially in multimodal settings.
• Multimodal fusion benefit: Multimodal models leveraging both entity-enhanced text and structured variables outperform all unimodal approaches in short-term HF mortality prediction.
• Limitations of LLMs: Contemporary instruction-tuned generative LLMs do not constitute viable standalone alternatives for clinical risk prediction—even with tailored prompting and modality adaptation.
• Generalizability: While the findings are specific to French clinical notes and a single-center cohort, they underscore the non-sufficiency of clinical text alone and reinforce the need for clinically meaningful, multimodal representation in outcome modeling.

Limitations and Future Directions

Model performance and robustness are contingent upon NER quality, annotation schema, and generalizability to multilingual/multicenter cohorts. Larger LLMs may alter these findings, but constraints imposed by clinical environments and computational resources remain nontrivial. Improving LLM reliability on structured data inputs, exploring more effective cross-modal pretraining, and advancing interpretability for deployment are critical next steps.

One theoretical direction is the principled integration of externally pre-trained LLM representations and task-specific multimodal fusions, potentially with continuous fine-tuning or retrieval-augmented methods. Advances in model evaluation metrics emphasizing reliability, interpretability, and uncertainty quantification are also required for safe clinical application.

Conclusion

The study robustly demonstrates that French clinical notes alone are insufficient for high-fidelity short-term HF mortality prediction. Multimodal transformers incorporating both structured data and entity-level textual signals, especially using attention-based or gating fusion, yield the best discriminative performance. Currently available instruction-tuned LLMs are inconsistent and unreliable for this purpose, especially in multimodal settings. The practical implication is the clear superiority and robustness of entity-aware multimodal architectures for clinical risk prediction in HF, with future work required to improve generalizability, interpretability, and LLM reasoning over structured data.