Latent World Recovery for Multimodal Learning with Missing Modalities

Abstract: We study multimodal learning under missing modalities, with particular motivation from bioscience applications in which heterogeneous modalities are often only partially available when decisions need to be made. We propose Latent World Recovery (LWR), a framework built on two key ideas: (i) modality-specific embeddings from different modalities are aligned in a shared latent space, and (ii) a unified representation is constructed by fusing only the embeddings of the modalities that are actually available at both training and inference time. Rather than imputing missing modalities or requiring a fixed modality set, LWR treats each modality as a partial perception of an underlying latent state and performs availability-aware representation learning directly from the observed modalities. This combination of neighbor-based latent alignment and availability-aware modality fusion enables robust multimodal prediction under partial observation, while avoiding error propagation from explicit reconstruction of missing modalities. We evaluate the proposed framework on real-world incomplete multi-omics benchmarks and demonstrate that it provides an effective approach to downstream tasks such as cancer phenotype classification and survival prediction.

• The paper introduces a latent-state recovery method that bypasses missing modality reconstruction by leveraging available observations to form unified biological representations.
• It employs modality-specific variational encoders and an attention-based fusion mechanism, validated on multi-omics datasets for cancer classification, survival prediction, and reconstruction.
• Neighbor-based alignment preserves local sample relationships and prevents modal collapse, resulting in robust, interpretable embeddings for precision oncology.

Latent World Recovery for Multimodal Learning with Missing Modalities

Introduction and Motivation

Latent World Recovery (LWR) addresses the core challenge in multimodal learning under missing modalities, particularly in high-dimensional and highly incomplete bioscience datasets (e.g., multi-omics). Unlike earlier approaches that either require complete modality sets, impute missing data, or rely on decision-level fusion with limited cross-modality interaction, LWR is predicated on a latent-state recovery paradigm: it treats each observed modality as a (potentially noisy) partial view of an unobserved underlying biological state. The principal design is not to reconstruct missing modalities but to directly learn unified sample-level representations robust to arbitrary missingness by leveraging only observed modalities.

Methodology

LWR deploys separate modality-specific variational encoders that map each available modality into a shared latent space. These modality representations are subsequently fused into a sample-level embedding using an attention-based, availability-aware latent fusion mechanism—which computes embedding weights only over the modalities actually present for a given sample. This ensures that the aggregation process is strictly conditioned on available data, without substituting imputation, zeros, or mask tokens.

Figure 1: LWR pipeline: each available modality is variationally encoded, fused with attention conditioned on actual availability, and the resulting embedding is jointly supervised via observed-modality reconstruction and neighbor-based alignment.

Central to LWR’s stability and biological effectiveness is a neighbor-based alignment objective. Instead of enforcing strict coordinate-wise similarity across modality embeddings for matched samples (which can suppress biologically relevant modality-specific variations and lead to modal collapse), LWR encourages the learned sample embedding to preserve local neighborhood structures induced by each modality. This is formulated via a stop-gradient KL divergence between modality-specific and fused-space sample neighborhood distributions, constructed with cosine similarity and a temperature-controlled exponential kernel. The method only considers the observed sample pairs for each modality and batch, further respecting the missingness pattern inherent to the data.

Crucially, LWR eschews any objective or model component devoted to explicit missing-modality reconstruction. Observed-modality reconstruction is used purely as a self-supervision signal, and missing modalities do not appear in the loss at any stage.

Empirical Evaluation

Classification, Survival Prediction, and Reconstruction

LWR is evaluated on multiple multi-omics benchmarks, notably TCGA (17 cancer cohorts, up to five omics modalities each), the Childhood Cancer Model Atlas (CCMA), and the Cancer Cell Line Encyclopedia (CCLE). Three tasks are used to assess the learned representations:

• Cancer phenotype classification: Downstream XGBoost classifiers on LWR embeddings outperform or match specialized multi-omics methods (MIND, JASMINE, IntegrAO, MSNE) on most TCGA cohorts, with clear wins on CCMA.
• Survival prediction: LWR achieves top-2 performance (C-index) on key TCGA cohorts and secures the best overall ranking on multiple key datasets, demonstrating that fused representations encode clinical outcome-relevant variance.
• Masked data reconstruction: Pearson correlations between original and reconstructed values (with 10% random masking per modality) are consistently higher for LWR compared to MIND and JASMINE, highlighting strong information preservation.

Ablation Studies

Rigorous ablations reveal that coordinate-level alignment (naive pairwise loss) induces latent collapse, severely degrading performance on all tasks. The neighbor-based alignment robustly shields against representation collapse while preserving cross-modality relational structure. Further, the comparison between attention-based versus mean fusion shows task dependency: dynamic attention is optimal for phenotype classification and information preservation, while mean fusion acts as a regularizer in survival prediction settings.

Clustering-Based Survival Stratification

A case study applies clustering (varying $k$) over LWR representations in the TCGA cohorts, then interrogates whether resultant clusters stratify patient survival. Using $-\log_2(p)$ from the log-rank test as a separation metric, LWR’s latent space provides meaningful and stable survival stratification across cancers, validated against simplified baselines.

Figure 2: Clustering of learned representations on TCGA cohorts; clusters derived from LWR embeddings display strong and consistent survival separation ($-\log_2(p)$).

Figure 3: Mortality rates for patient clusters derived from LWR, illustrating survival stratification correspondence with established molecular subtypes.

Further biological annotation indicates that unsupervised LWR clusters are aligned with known high-risk genotypic or molecular tumor subgroups (e.g., IDH status in LGG, copy-number subtypes in UCEC), with differentiated mortality and clinical profiles—supporting the biological interpretability of the method.

Implications and Future Directions

LWR suggests that the prevalent focus on missing-modality reconstruction for multimodal data integration in computational biology is suboptimal, and that robust, discriminative representations can be learned directly from arbitrary partial observations. The introduction of a neighbor-based (as opposed to coordinate-wise) alignment provides a practical, scalable means of aligning cross-modal relational structures without forcing modal collapse.

The attention-based availability-aware fusion mechanism also reveals how dynamic, data-driven weighting of observed modalities enables the model to adapt its integration to biologically relevant contexts, improving interpretability. The alignment of unsupervised clusters with molecular subtypes is of direct translational interest in oncology and precision medicine.

Extending LWR to support uncertainty-aware fusion, more complex neighborhoods (such as those derived from biological graphs), or application to diverse multimodal domains (e.g., medical imaging and omics) represents important future work. The separation between representation learning and downstream prediction—central to the LWR design—provides a flexible scaffold for transfer learning and domain adaptation in clinical data science.

Conclusion

LWR establishes an effective and theoretically motivated approach to incomplete multimodal learning by fusing only available modalities with adaptive attention, regularizing sample-level embeddings via neighbor-based alignment, and avoiding explicit missing-modality generation. The approach achieves superior or on-par empirical performance in multi-omics classification, survival, and reconstruction while producing interpretable, biologically meaningful latent spaces. LWR’s design philosophy—of partial-observation state recovery over data imputation—marks a significant conceptual step toward robust, flexible, and interpretable multimodal learning in high-missingness, high-heterogeneity biological data.

Reference: "Latent World Recovery for Multimodal Learning with Missing Modalities" (2606.12362)