Missing Modality Reconstruction

• Missing modality reconstruction is a set of techniques that infers unobserved modalities from available data to maintain semantic consistency and robust generalization.
• It utilizes methods such as masked autoencoders, diffusion models, and cross-modal translation to address challenges like semantic drift and arbitrary missing patterns.
• Applications include action recognition, medical imaging, and remote sensing, where accurate restoration of missing signals improves overall task performance.

Missing modality reconstruction refers to the set of algorithmic and architectural techniques for inferring the unobserved outputs of one or more modalities in multimodal systems, from the information present in the remaining observed modalities. The objective is not mere imputation, but the restoration of semantically consistent, information-rich features or signals of the missing streams, ensuring robust inference and generalization even when some modalities are systematically or randomly absent during training or deployment. The problem pervades heterogeneous data fusion settings in action recognition, emotion and sentiment analysis, medical imaging, event-based vision, remote sensing, federated analytics, and large pre-trained multimodal foundation models.

1. Theoretical Frameworks and Challenges

Missing modality reconstruction assumes an underdetermined mapping $f: \{x_i\}_{i\in I} \to \{x_j\}_{j\in J}$ where $I$ indexes the observed modalities and $J$ indexes the missing ones, with $I \cup J = \mathcal{A}$ (the full set of $M$ modalities), $I \cap J = \emptyset$. Reconstruction is ill-posed in general, as information may not be recoverable from the observed streams alone due to conditional independence, sampling, or corruption. Core challenges include:

• Semantic Consistency: Preserving structural and semantic correlations between available and missing modalities, avoiding mode collapse or semantic drift (Lu et al., 2023, Zhang et al., 8 Jul 2025).
• Robustness to Arbitrary Patterns: Handling arbitrary missingness—random, block, pattern-dependent, or even all-stage (missing at both train and test)—without requiring retraining or specialized networks for each missing case (Zhao et al., 2024, Kebaili et al., 22 Jan 2025).
• Efficient Inference and Adaptation: Achieving efficient inference, low memory, and rapid adaptation to new missing patterns in large-scale or distributed systems (Zhao et al., 2024, Liu et al., 14 Apr 2025).
• Generalization Beyond Seen Data: Ensuring that reconstructed features enhance zero-shot generalization and downstream performance, not just pixel-level similarity (Dai et al., 3 Feb 2026).

These challenges have led to the development of a diverse set of algorithms spanning masked autoencoding, diffusion-based conditional generative models, invertible or content-preserving prompts, cross-modal translation, optimal transport, variational inference, and flow-based models.

2. Architectural Paradigms for Reconstruction

2.1 Masked Autoencoders and Predictive Coding

Masked autoencoding approaches, such as ActionMAE (Woo et al., 2022) and the multimodal MAE framework (Zhao et al., 2024), randomly drop entire modality tokens and/or feature patches during training, then reconstruct them via a lightweight transformer or U-Net architecture. A memory/global context token is typically prepended to aggregate global cues for reconstruction. Objective functions mix classification and reconstruction losses, often balanced 1:1.

2.2 Diffusion and Flow-Based Generative Models

Conditional diffusion models—for example, AMM-Diff (Kebaili et al., 22 Jan 2025), ADMC (Zhang et al., 8 Jul 2025), and scalable diffusion for foundation VLMs (Dai et al., 3 Feb 2026)—perform stochastic restoration of missing features in the latent or pixel space by simulating forward noising and reverse denoising, conditioned on observed modalities. Adaptive techniques (e.g., FiLM layers with modality indicators, dynamic gating) enable fully variable input configurations. Normalizing flow models (e.g., RealNVP-style coupling layers (Sun et al., 2024)) offer invertible, likelihood-based translation between Gaussian-distributed feature spaces, enabling precise and uncertainty-aware imputation.

2.3 Cross-Modal Translation and Knowledge Transfer

Cross-modal translation systems map the feature space of available modalities (visual, language, acoustic) to the missing one via dedicated translation networks and fuse multiple reconstructions, typically refining the result via cross-modal attention (Liu et al., 2023). These methods rely heavily on the learned inter-modality mappings and are often coupled with consistency losses.

2.4 Invertible Prompt Learning and Memory Prompt Pools

Invertible prompt learning (IPL) leverages content-preserving invertible layers to generate substitute prompts for the missing modality within frozen deep backbones (Lu et al., 2023). This bidirectional mapping ensures both prompt utility and full reconstrucibility of the original features. Memory-prompt based approaches, such as RebQ (Zhao et al., 2024), introduce small prompt pools for each modality and use memory selection mechanisms to reconstruct missing queries in continual learning and continual missing scenarios, minimizing catastrophic forgetting.

2.5 Generative Adversarial and VAE Approaches

Unified GAN frameworks synthesize missing modalities from arbitrary subsets of inputs by integrating both commonality- and discrepancy-sensitive encoding, combined with dynamic feature unification modules to enable variable conditioning and robust synthesis (Zhang et al., 2023). Lightweight MVAE setups, sometimes augmented by cross-modal distribution mapping, optimize both per-modality and cross-modality reconstruction, and are often applied for federated or heterogeneous environments (Liu et al., 14 Apr 2025).

3. Mathematical Losses and Optimization Strategies

Substantial innovation centers on the formulation of objective functions to enforce semantic reconstruction, distributional alignment, and robust fusion.

• Predictive Coding and Reconstruction Losses: Typically MSE or L1, computed on the reconstructed versus ground truth features/tokens/voxels for the missing modality (Woo et al., 2022, Zhao et al., 2024).
• Distribution Alignment: Optimal transport (OT) and Wasserstein distances regularize the alignment of feature distributions between paired modalities, with learned meta-predictors for alignment dynamics (Han et al., 2022, Sun et al., 2024).
• Contrastive and Information-Theoretic Losses: Distribution-based contrastive objectives using 2-Wasserstein or InfoNCE enable cross-modal Gaussian alignment and reduce semantic variance (Sun et al., 2024).
• Uncertainty Estimation and Correlation Losses: Pearson-correlation-based objectives for both latent and output uncertainty estimation, with explicit propagation of reconstruction variance through the fusion module and into downstream loss (Nguyen et al., 18 Apr 2025).
• Adversarial and Likelihood-Based Losses: GAN-based adversarial terms ensure realism of generated modalities, combined with modality-conditional or full-modality log-likelihood maximization in diffusion and MVAE models (Zhang et al., 2023, Kebaili et al., 22 Jan 2025, Liu et al., 14 Apr 2025).
• Downstream Supervised/Contrastive Terms: Task-aligned KL divergence, supervised point-based contrastive loss for affective content, and feature distillation via margin-aware distillation all contribute to robust co-learning and transfer (Lu et al., 2023, Sun et al., 2024, Zhao et al., 2024).

4. Robustness, Generalization, and Empirical Findings

Empirical evaluation consistently demonstrates that explicit missing modality reconstruction—when architected and regularized appropriately—achieves substantial improvements in task performance, robustness, and generalization.

• Task-Specific Gains: In action recognition, ActionMAE reduced the mean accuracy gap under missing modalities from 36.0% to 9.3% (Woo et al., 2022). In multimodal sentiment/emotion, ADMC and CM-ARR improved weighted/unweighted accuracy by 6–10% and more on IEMOCAP, compared to prior methods (Zhang et al., 8 Jul 2025, Sun et al., 2024).
• Fidelity Metrics: In MRI synthesis, AMM-Diff and Unified GAN approaches yielded PSNR and SSIM improvements of +1–3 dB and +0.02–0.03 versus baselines (Kebaili et al., 22 Jan 2025, Zhang et al., 2023).
• Federated and Continual Contexts: Fed-PMG and FedRecon achieve “ideal” performance (identical to fully-paired data) with up to $97.5\%$ reduction in communication, and maintain higher accuracy and less forgetting in non-IID and continual settings (Yan et al., 2023, Liu et al., 14 Apr 2025, Zhao et al., 2024).
• Semantic and Affective Consistency: Event-to-video approaches leveraging cross-modal feature alignment (e.g., Semantic-E2VID) outperform non-semantic counterparts on SSIM/LPIPS while providing better texture and contour recovery (Wu et al., 20 Oct 2025).
• Foundation Model and Zero-Shot Robustness: Scalable latent-space diffusion models for VLMs restore missing modality features bidirectionally, with +7 pp F1 improvement on MM-IMDb and superior resilience to increasing missing rates (Dai et al., 3 Feb 2026).
• Ablation Studies: Extensive studies verify that key components—memory tokens, positional embeddings, bidirectional alignment, or uncertainty modules—are essential for resilience and semantic fidelity across modalities and tasks.

5. Applications Across Domains

The spectrum of applications includes:

• Action Recognition: Modalities such as RGB, depth, and IR are processed, dropped, and reconstructed to enable robust action classification under sensor dropout or occlusion (Woo et al., 2022).
• Affective Computing: Multimodal sentiment and emotion prediction leverages audio, text, and vision streams; cross-modal translation and latent diffusion compensate for incomplete cues (Liu et al., 2023, Zhang et al., 8 Jul 2025, Sun et al., 2024).
• Medical Imaging: MRI modalities (T1, T2, FLAIR, etc.) are reconstructed either via diffusion, GAN, or frequency-based mixing for diagnostic segmentation (Zhang et al., 2023, Kebaili et al., 22 Jan 2025, Zhao et al., 2024).
• Event-Based Vision: Semantic priors from frame-based SAM transfer, fused with event representations, yield significantly better video frame recovery (Wu et al., 20 Oct 2025).
• Remote Sensing: SAR-based optical reconstruction supports urban mapping when cloud occlusion obscures optical data (Hafner et al., 2023).
• Federated and Distributed Learning: Cross-client non-IID handling and distributed modality imputation via pseudo generation or MVAE supports learning on incomplete, privacy-preserving data (Yan et al., 2023, Liu et al., 14 Apr 2025).
• Large-Scale Foundation Models: Scalable diffusion plug-ins enable foundation VLMs (e.g., CLIP) to regain generalization and reliability in high missingness scenarios (Dai et al., 3 Feb 2026).

6. Limitations, Open Problems, and Future Directions

Although missing modality reconstruction has yielded state-of-the-art performance across a range of tasks and domains, intrinsic limitations persist:

• Dependence on Full-Modality Training: Many methods require initially complete data to compute reconstruction targets; extensions to fully unsupervised cases remain open (Lu et al., 2023, Zhao et al., 2024).
• Degradation with Multiple or Key Modality Loss: Performance degrades sharply if multiple highly-informative modalities are missing, or under extreme dropout (e.g., 99% missing) (Kebaili et al., 22 Jan 2025).
• Computational Overhead and Latency: Diffusion models, while powerful, introduce additional inference latency in comparison to direct imputation (Kebaili et al., 22 Jan 2025, Dai et al., 3 Feb 2026).
• Generalization to Arbitrary/Novel Modalities: Extension to more than 2–4 modalities and handling unseen modality combinations is an active area, as is adaptation to unaligned or unregistered sensor data (Zhao et al., 2024, Kebaili et al., 22 Jan 2025, Wu et al., 20 Oct 2025).
• Uncertainty Quantification: Explicit modeling and propagation of uncertainty is emerging as essential for reliable deployment, interpretability, and clinical or high-impact applications (Nguyen et al., 18 Apr 2025).
• Theoretical Guarantees: Formal analysis of convergence and generalizability in federated, diffusion, and autoencoding settings, especially with generated pseudo modalities, is limited (Yan et al., 2023, Liu et al., 14 Apr 2025).

A plausible implication is that future work will focus on (a) unsupervised or few-shot missing-modality learning, (b) universal and efficient architectures for high-dimensional multimodal data, (c) joint uncertainty–task optimization, and (d) real-world deployments under distribution shift and adversarial missingness.

7. Summary Table of Core Approaches and Outcomes

Approach	Reconstruction Mechanism	Robustness/Performance Highlights
ActionMAE (Woo et al., 2022)	Random modality masking + MAE	Mean accuracy drop under missing: 36%→9.3%
ADMC (Zhang et al., 8 Jul 2025)	Attention-diffusion (latent)	WA/UA +6–10% IEMOCAP vs prior; multi-missing
Fed-PMG (Yan et al., 2023)	Pseudo modality via spectrum mix	FL PSNR=35.0 dB (=Ideal), 97.5% comm. saved
Unified GAN (Zhang et al., 2023)	GAN+CDS Encoder+DFUM Fusion	PSNR/SSIM improvement +1–3 dB/+0.02 over prior
Semantic-E2VID (Wu et al., 20 Oct 2025)	Cross-modal semantic align/fusion	ECD: SSIM 0.594/LPIPS 0.208 (state-of-art)
Scalable Diffusion (Dai et al., 3 Feb 2026)	Latent-space DiT w/ gating, bidirectional flow	MM-IMDb F1-M 58.22 (+7.1 pp) zero-shot, robust to 90% missing

A clear trend is the shift from naive imputation or zero-fill toward principled, uncertainty-aware, and generalizable reconstruction pipelines that support robust downstream learning—advancing the reliability and applicability of multimodal learning in incomplete, distributed, and real-world scenarios.