MultiSurv: Multimodal Deep Survival Analysis

• The paper demonstrates that MultiSurv integrates heterogeneous modalities using a DeepHit-based backbone to estimate full survival distributions in oncology.
• MultiSurv employs modality-specific encoders to generate 128-dimensional embeddings that are fused via an inter-modality cross-attention mechanism.
• Evaluation shows promising C-index performance in prostate and bladder cancer tasks, while highlighting challenges like generalization gaps and dataset shifts.

MultiSurv is a multimodal deep survival model introduced by Wahab et al. for prediction of right-censored time-to-event outcomes in oncology, integrating heterogeneous patient data—clinical covariates, radiology (MRI), transcriptomics (RNA-seq), and histopathology (whole-slide images, WSI)—to estimate individualised time-to-biochemical recurrence in prostate cancer and time-to-cancer recurrence in bladder cancer (Wahab et al., 5 Sep 2025). The framework utilises DeepHit with a projection layer and inter-modality cross-attention, was evaluated in the CHIMERA Grand Challenge across two of the three provided tasks, and is presented as broadly applicable to survival prediction tasks involving heterogeneous biomedical data.

1. Clinical endpoints and task definition

MultiSurv is designed for two survival tasks. In Task 1 of CHIMERA, it predicts time to biochemical recurrence (BCR) in prostate cancer. In Task 3, it predicts time to tumor recurrence in bladder cancer. Both endpoints are right-censored time-to-event outcomes, measured in months from definitive therapy.

The framework is explicitly multimodal, but the available modalities differ by task. For prostate cancer, the model incorporates clinical covariates, T2-weighted MRI, and WSI-derived pathology features. For bladder cancer, it incorporates clinical covariates, RNA-seq, and WSI-derived pathology features. This task-specific modality structure is central to the design: the system is not a single fixed-input network, but a common fusion-and-survival architecture instantiated over the subset of modalities available for a given disease setting.

The stated objective is to capture complementary prognostic signals across modalities and to estimate individualised survival distributions rather than a single risk score. In the terminology of the paper, the framework targets personalised risk stratification through multimodal integration with deep survival learning (Wahab et al., 5 Sep 2025).

2. Modality-specific representation learning

MultiSurv begins by extracting fixed-length feature vectors from each data modality and projecting them into a common latent space of dimension $d=128$. This common dimensionality is the basis for subsequent inter-modality attention.

For clinical data, the input is a set of $C$ real-valued or one-hot categorical variables, after dropping features with missing values. The encoder is a single fully connected layer with weights $W_{\mathrm{clin}} \in \mathbb{R}^{128 \times C}$ and bias $b_{\mathrm{clin}} \in \mathbb{R}^{128}$, followed by ReLU and dropout. The resulting embedding is

$$h_{\mathrm{clin}} = \mathrm{Dropout}\bigl(\mathrm{ReLU}(W_{\mathrm{clin}} x_{\mathrm{clin}} + b_{\mathrm{clin}})\bigr) \in \mathbb{R}^{128}.$$

For MRI, which is used in the prostate task only, the inputs are pre-segmented and cropped T2-weighted MRI volumes, 3D, single-channel. The encoder is a pretrained MedicalNet ResNet10 backbone, with weights frozen or finetuned, producing a 512-dimensional embedding from the final average-pooling layer. A linear reduction then maps that representation into the shared latent space:

$$h_{\mathrm{MRI}} = \mathrm{Dropout}\bigl(\mathrm{ReLU}(W_{\mathrm{MRI}} e_{\mathrm{MRI}} + b_{\mathrm{MRI}})\bigr) \in \mathbb{R}^{128},$$

where $e_{\mathrm{MRI}} \in \mathbb{R}^{512}$.

For RNA-seq, which is used in the bladder task only, the inputs are gene expression counts of dimension 19,359. The encoder is a PCA projection to 128 dimensions,

$$h_{\mathrm{RNA}} = \mathrm{Dropout}\bigl(\mathrm{ReLU}(W_{\mathrm{PCA}} x_{\mathrm{RNA}} + b_{\mathrm{PCA}})\bigr) \in \mathbb{R}^{128},$$

with $W_{\mathrm{PCA}} \in \mathbb{R}^{128 \times 19359}$ and $b_{\mathrm{PCA}} \in \mathbb{R}^{128}$. The summary notes that these parameters are learned jointly with the rest of the network, while also remarking that one may pretrain PCA and then fix $C$0.

For whole-slide pathology, the inputs are gigapixel WSIs tiled at $C$1 into $C$2 patches. A tile-level feature extractor, the CONCH visual-LLM, maps each tile to a 768-dimensional vector. A slide-level aggregator, Titan, pools the set of tile embeddings into a single 512-dimensional vector $C$3 by attention-based pooling. A linear projection then produces the shared embedding:

$C$4

This modality-specific design separates feature extraction from multimodal fusion. A plausible implication is that the architecture can accommodate encoders with very different inductive biases—tabular, volumetric imaging, molecular, and slide-level pathology—without requiring them to share a common upstream representation.

3. Projection layers and inter-modality cross-attention

Once each modality $C$5 has produced a 128-dimensional embedding $C$6, MultiSurv projects the embeddings into query, key, and value spaces and performs cross-attention across modalities.

For each modality embedding $C$7 and attention head $C$8, the model computes

$C$9

where $W_{\mathrm{clin}} \in \mathbb{R}^{128 \times C}$0, $W_{\mathrm{clin}} \in \mathbb{R}^{128 \times C}$1, and the biases are in $W_{\mathrm{clin}} \in \mathbb{R}^{128 \times C}$2.

The per-modality queries are concatenated into a matrix $W_{\mathrm{clin}} \in \mathbb{R}^{128 \times C}$3, where $W_{\mathrm{clin}} \in \mathbb{R}^{128 \times C}$4 is the number of available modalities for the task; similarly for $W_{\mathrm{clin}} \in \mathbb{R}^{128 \times C}$5 and $W_{\mathrm{clin}} \in \mathbb{R}^{128 \times C}$6. Cross-attention is then computed as

$W_{\mathrm{clin}} \in \mathbb{R}^{128 \times C}$7

Each row of $W_{\mathrm{clin}} \in \mathbb{R}^{128 \times C}$8 is the attended representation for one modality.

The outputs of all $W_{\mathrm{clin}} \in \mathbb{R}^{128 \times C}$9 heads are concatenated and collapsed across modalities, for example by averaging or a small MLP, to obtain a fused representation

$b_{\mathrm{clin}} \in \mathbb{R}^{128}$0

typically with $b_{\mathrm{clin}} \in \mathbb{R}^{128}$1.

The paper identifies the projection-plus-cross-attention design as a strength because it flexibly incorporates any subset of modalities at inference. In architectural terms, fusion is therefore not merely concatenation; it is an interaction mechanism that allows one modality’s latent state to reweight another’s contribution.

4. DeepHit adaptation, loss function, and optimisation

The survival head adapts DeepHit to multimodal input. DeepHit formulates survival as learning a discrete probability mass function over $b_{\mathrm{clin}} \in \mathbb{R}^{128}$2 pre-defined time bins. Let $b_{\mathrm{clin}} \in \mathbb{R}^{128}$3 index the bins. The network predicts

$b_{\mathrm{clin}} \in \mathbb{R}^{128}$4

The cumulative distribution function is

$b_{\mathrm{clin}} \in \mathbb{R}^{128}$5

and the survival function is

$b_{\mathrm{clin}} \in \mathbb{R}^{128}$6

The summary notes that, optionally, one would learn separate heads per event for competing risks; here there is a single event of interest.

Following Lee et al. (DeepHit), the total loss is

$b_{\mathrm{clin}} \in \mathbb{R}^{128}$7

The likelihood term is a negative log-likelihood over the event bin. The ranking loss encourages correct ordering of risk scores over comparable pairs

$b_{\mathrm{clin}} \in \mathbb{R}^{128}$8

with margin $b_{\mathrm{clin}} \in \mathbb{R}^{128}$9. The diversity loss discourages collapse of predictions across bins. Hyperparameters $$h_{\mathrm{clin}} = \mathrm{Dropout}\bigl(\mathrm{ReLU}(W_{\mathrm{clin}} x_{\mathrm{clin}} + b_{\mathrm{clin}})\bigr) \in \mathbb{R}^{128}.$$0, $$h_{\mathrm{clin}} = \mathrm{Dropout}\bigl(\mathrm{ReLU}(W_{\mathrm{clin}} x_{\mathrm{clin}} + b_{\mathrm{clin}})\bigr) \in \mathbb{R}^{128}.$$1, and $$h_{\mathrm{clin}} = \mathrm{Dropout}\bigl(\mathrm{ReLU}(W_{\mathrm{clin}} x_{\mathrm{clin}} + b_{\mathrm{clin}})\bigr) \in \mathbb{R}^{128}.$$2 are tuned on validation data (Wahab et al., 5 Sep 2025).

Training uses $$h_{\mathrm{clin}} = \mathrm{Dropout}\bigl(\mathrm{ReLU}(W_{\mathrm{clin}} x_{\mathrm{clin}} + b_{\mathrm{clin}})\bigr) \in \mathbb{R}^{128}.$$3 bins covering 0–60 months. Clinical and RNA features undergo z-score standardisation using train-fold statistics only. MRI preprocessing uses standard MRI intensity normalisation and cropping around the prostate. WSI preprocessing uses tissue segmentation with Grandqc, tile extraction at 1024 px and $$h_{\mathrm{clin}} = \mathrm{Dropout}\bigl(\mathrm{ReLU}(W_{\mathrm{clin}} x_{\mathrm{clin}} + b_{\mathrm{clin}})\bigr) \in \mathbb{R}^{128}.$$4, and CONCH descriptors. Optimisation uses Adam with initial learning rate $$h_{\mathrm{clin}} = \mathrm{Dropout}\bigl(\mathrm{ReLU}(W_{\mathrm{clin}} x_{\mathrm{clin}} + b_{\mathrm{clin}})\bigr) \in \mathbb{R}^{128}.$$5 and weight decay $$h_{\mathrm{clin}} = \mathrm{Dropout}\bigl(\mathrm{ReLU}(W_{\mathrm{clin}} x_{\mathrm{clin}} + b_{\mathrm{clin}})\bigr) \in \mathbb{R}^{128}.$$6. Dropout is $$h_{\mathrm{clin}} = \mathrm{Dropout}\bigl(\mathrm{ReLU}(W_{\mathrm{clin}} x_{\mathrm{clin}} + b_{\mathrm{clin}})\bigr) \in \mathbb{R}^{128}.$$7 in modality projection heads and $$h_{\mathrm{clin}} = \mathrm{Dropout}\bigl(\mathrm{ReLU}(W_{\mathrm{clin}} x_{\mathrm{clin}} + b_{\mathrm{clin}})\bigr) \in \mathbb{R}^{128}.$$8 in the fusion MLP. Batch size is 16, early stopping monitors validation C-index with patience 10 epochs, and the maximum training duration is 100 epochs. Cross-validation is 5-fold stratified on the event indicator and repeated 10 times with different seeds.

The DeepHit backbone is identified in the paper as a strength because it provides full survival distributions and handles censoring without proportional hazards assumptions. This distinguishes MultiSurv from architectures that predict only scalar hazards or assume proportionality.

5. Evaluation protocol and reported performance

Evaluation was performed with concordance index (C-index) throughout. The abstract reports, for Task 1, a C-index of 0.843 on 5-folds cross-validation and 0.818 on the CHIMERA development set, and for Task 3, a C-index of 0.662 on 5-folds cross-validation and 0.457 on the development set (Wahab et al., 5 Sep 2025). The detailed breakdown below shows that the reported values depend strongly on the modality configuration.

Task	Configuration	Reported C-index
Task 1: prostate BCR	Clinical only	0.843 ± 0.08 (CV), 0.818 (development set)
Task 1: prostate BCR	WSI only	0.846 ± 0.09 (CV), no dev-set submission
Task 1: prostate BCR	Clinical + WSI	0.877 ± 0.08 (CV), 0.6446 (development set)
Task 1: prostate BCR	Clinical + MRI + WSI	0.833 ± 0.11 (CV)
Task 3: bladder recurrence	Clinical only	0.749 ± 0.08 (CV), 0.587 (Cox PH baseline on dev)
Task 3: bladder recurrence	RNA only	0.664 ± 0.08 (CV)
Task 3: bladder recurrence	WSI only	0.649 ± 0.06 (CV)
Task 3: bladder recurrence	Clinical + WSI	0.733 ± 0.06 (CV), 0.4565 (development set)
Task 3: bladder recurrence	Clinical + RNA	0.689 ± 0.07 (CV)

The ablation summary states that, in both tasks, the dominant single modality was clinical features, and that WSI added most complementary signal in Task 1. It also states that the ranking and diversity losses improved C-index by approximately 2–3%.

The evaluation section is notable for the mismatch between cross-validation and development-set behaviour for several multimodal combinations. The paper itself identifies generalisation gaps between cross-validation and the held-out development set, especially for multimodal combinations, and states that these suggest potential overfitting or dataset shift. The same section nevertheless characterises the results as indicating adaptability and potential for clinical translation, while also emphasizing the challenge of robust multimodal generalisation.

6. Limitations, proposed extensions, and name ambiguity

The paper identifies several strengths. The projection and cross-attention design flexibly incorporates any subset of modalities at inference. The DeepHit backbone provides full survival distributions and handles censoring without proportional hazards assumptions. It also reports strong CV performance in prostate BCR, with C-index approximately 0.84, and states that this demonstrates that WSI features can complement clinical covariates (Wahab et al., 5 Sep 2025).

The limitations are stated explicitly. Generalisation gaps between cross-validation and the held-out development set, especially for multimodal combinations, suggest potential overfitting or dataset shift. The relatively small sample sizes—95 prostate and 176 bladder—limit the capacity to fully exploit high-dimensional imaging and transcriptomic inputs. Task 3 performance on the challenge development set, with C-index = 0.456, is described as poor and likely due to late access for submissions and restricted tuning.

The proposed extensions are also explicit: pre-training the entire cross-attention fusion on large-scale unlabeled multimodal cancer cohorts to improve generalisation; incorporating continuous-time survival models such as neural ODEs to avoid bin discretisation; extending to competing risks by adding parallel DeepHit heads per event; and applying the same framework to other cancers, including lung and breast, or to non-oncology domains where multimodal time-to-event prediction is needed. These proposals position MultiSurv as a template for multimodal survival analysis rather than a disease-specific architecture.

A separate source introduces a different use of the label “MultiSurv” in multi-objective Bayesian optimisation. In Chugh’s “Mono-surrogate vs Multi-surrogate in Multi-objective Bayesian Optimisation” (Chugh, 2022), “MultiSurv” denotes a multi-surrogate methodology that builds one Gaussian-process surrogate per objective, uses Expected Improvement on an augmented weighted Tchebycheff scalarisation, and models the resulting scalarising distribution as non-Gaussian, with a GEV or Gumbel approximation. This is unrelated to Wahab et al.’s multimodal oncology survival framework. This suggests that references to “MultiSurv” require domain disambiguation: in cancer prognosis it denotes a DeepHit-based multimodal survival model, whereas in Bayesian optimisation it refers to a multi-surrogate acquisition strategy (Chugh, 2022).