Missing-Modality Compensation Module

• Missing-Modality Compensation Module is a component that generates complete multimodal representations by synthesizing or rebalancing information when some inputs are missing.
• It employs diverse mechanisms such as image synthesis, latent feature completion, token projection, and graph propagation to recover critical modality-specific details.
• The approach enhances system robustness and performance in applications like medical imaging, audio-visual emotion recognition, and sensor fusion under incomplete data conditions.

A Missing-Modality Compensation Module is a component in a multimodal system that, given an arbitrary subset of observed modalities, produces either a completed multimodal stack or a task-sufficient latent representation so that downstream inference can proceed despite absent inputs. In the recent literature, this concept appears in several forms: unified image synthesis blocks that reconstruct missing MRI sequences, latent diffusion or CVAE modules that complete feature vectors, graph- and joint-embedding mechanisms that infer missing task-related information from related samples or correlated modalities, token- or prompt-level substitutes for transformer backbones, and non-generative weighting modules that rebalance available experts instead of explicitly synthesizing data (Zhang et al., 2023, Zhang et al., 8 Jul 2025, Zhang et al., 2022, Nezakati et al., 2024, Li et al., 25 Jul 2025, Lu et al., 26 Feb 2026). Across these variants, the common objective is robustness under incomplete modality availability while preserving the gains of multimodal learning.

1. Problem setting and formal role

Missing modalities arise in heterogeneous settings for heterogeneous reasons. In multi-modal MRI, T1, T2, T1Gd, and FLAIR may be absent because of limited scanning time, image corruption, patient condition constraints, or site-specific protocols and hardware differences (Zhang et al., 2023). In multimodal emotion, intent, or reaction modeling, audio, video, and text may be missing because of sensor malfunctions, incomplete data, network issues, environmental noise, occlusions, poor lighting, or privacy restrictions (Zhang et al., 8 Jul 2025, Hu et al., 2024, Devulapally et al., 2024, Ramazanova et al., 2024). In multimodal healthcare records, entire clinical modalities may be absent for social or clinical reasons, producing block-wise rather than entry-wise missingness (Zhang et al., 2022). Remote sensing, RGB-D perception, visible-infrared ReID, and multimodal image understanding exhibit analogous failures when one sensing stream becomes unavailable (Zhang et al., 2023, Kieu et al., 20 Jan 2026, Lu et al., 26 Feb 2026).

The module’s functional contract is therefore broader than simple imputation. In the most literal formulation, it receives a multi-channel input with arbitrary missing channels plus an availability indicator, and outputs reconstructed available channels together with synthesized missing channels. A representative expression is

$\hat{Y}_i = G(X_1,\dots,X_4, AC)_i,$

where $AC=\{ac_i\}_{i=1}^{4}$ encodes modality presence, available modalities are reconstructed when $ac_i=1$, and missing modalities are synthesized when $ac_i=0$ (Zhang et al., 2023). In other families of methods, the output is not a reconstructed image but a compensated feature vector, a projected token sequence, a graph-refined latent code, or a reweighted ensemble prediction (Zhang et al., 8 Jul 2025, Nezakati et al., 2024, Zhang et al., 2022, Li et al., 25 Jul 2025).

A recurring systems-level interpretation is that the compensation module sits between modality-specific encoders and the downstream task head. Upstream, it consumes whichever modalities are present. Downstream, it presents a representation that behaves like a full-modality input, or at least minimizes the gap between complete and incomplete settings. This is explicit in medical-image synthesis (Zhang et al., 2023), tri-modal medical fusion (Wang et al., 2023), EHR modeling (Zhang et al., 2022), and unified segmentation under incomplete MRI, RGB-D, or RGB-T inputs (Zhao et al., 19 Sep 2025).

2. Major architectural realizations

Recent implementations differ mainly in what they compensate—pixels, tokens, latent features, graph states, or branch contributions—and in whether they generate missing information explicitly or compensate implicitly through fusion and optimization.

Representative system	Compensation locus	Core mechanism
"Unified Multi-Modal Image Synthesis for Missing Modality Imputation" (Zhang et al., 2023)	Image and feature space	CDS-Encoder + DFUM + multi-stream decoders + PatchGAN
"ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completion" (Zhang et al., 8 Jul 2025)	Latent feature space	Independent encoders + diffusion denoiser + masked reverse process
"Robust Facial Reactions Generation: An Emotion-Aware Framework with Modality Compensation" (Hu et al., 2024)	Unimodal embeddings	CVAE-based CMA + emotion-aware attention
"Towards Robust Multi-Modal Learning with Masked Modality Projection" (Nezakati et al., 2024)	Token space	Cross-attention projection from available tokens to missing-token estimates
"Exploring Missing Modality in Multimodal Egocentric Datasets" (Ramazanova et al., 2024)	Token input space	Repeated Missing Modality Token within MBT
"Learning with Missing Modalities in Multimodal Healthcare Data" (Zhang et al., 2022)	Latent patient graph	Task-guided similarity + GCN aggregation + adaptive imputation
"A Simple Framework for Multi-modal Learning with Missing Modality" (Li et al., 25 Jul 2025)	Expert outputs	Dynamic Mixture of Modality Experts + MoFe ranking loss
"Plug, Play, and Fortify: A Low-Cost Module for Robust Multimodal Image Understanding Models" (Lu et al., 26 Feb 2026)	Optimization dynamics	FRM-guided modality rebalancing during training

One class of modules performs direct synthesis. The unified GAN in (Zhang et al., 2023) synthesizes missing MRI modalities from any subset of available ones with a single model. Its generator comprises a Commonality- and Discrepancy-Sensitive Encoder, a Dynamic Feature Unification Module, and four modality-specific decoders, while four PatchGAN discriminators enforce local realism. This realizes compensation at both feature and image levels.

A second class performs latent completion rather than image reconstruction. ADMC trains unimodal encoders independently and freezes them, then uses an Attention-based Diffusion Network to model the joint distribution of stacked text, visual, and acoustic embeddings; missing modalities are initialized with Gaussian noise and completed by reverse diffusion while observed entries are overwritten using a modality mask at every step (Zhang et al., 8 Jul 2025). EMC uses a Conditional VAE to reconstruct missing speech from facial embeddings or missing facial embeddings from speech, and then treats compensated embeddings as normal inputs to the rest of the reaction-generation model (Hu et al., 2024). TMDC similarly compensates missing modalities in latent space after a denoising stage, using cross-attention between modality-specific and modality-common representations rather than explicit raw-data reconstruction (Zhuang et al., 13 Nov 2025).

A third class acts in token space. MMP randomly masks modalities during training and learns projection functions $f_i$ that estimate missing modality tokens from the available ones using aggregated tokens, cross-attention, and an MLP; projected tokens are then substituted directly into the multimodal backbone (Nezakati et al., 2024). MMT replaces all tokens of a missing modality with a repeated learned embedding plus positional encodings, allowing a transformer to maintain fixed sequence structure and learn a stable “missing modality” representation (Ramazanova et al., 2024).

A fourth class relies on retrieval or graph propagation. M$^3$Care compensates missing EHR modalities in latent space by discovering similar patients with a task-guided modality-adaptive similarity metric and aggregating neighbor information on a patient graph, after which adaptive imputation substitutes aggregated representations for missing modalities (Zhang et al., 2022). AM$^2$-EmoJE uses a Multimodal Joint Embedding module to align correlated modality descriptors into a shared space so available modalities can stand in for missing ones during emotion recognition (Devulapally et al., 2024).

A fifth class avoids explicit generation altogether. SimMLM treats compensation as dynamic fusion over modality-specific experts, with a gating network that assigns zero weight to missing modalities and reweights available experts adaptively (Li et al., 25 Jul 2025). MWAM instead compensates at training time by identifying modality preference in the frequency domain and down-weighting dominant modalities’ gradients or auxiliary losses so under-optimized branches remain useful when others disappear (Lu et al., 26 Feb 2026). This suggests that “compensation” can mean preserving the utility of remaining modalities rather than synthesizing absent ones.

3. Representation decomposition, fusion, and conditioning

A central design question is what information should be shared and what should remain modality-specific. Several papers converge on explicit decomposition. In multi-modal MRI synthesis, commonality corresponds to anatomy and spatial structure, while discrepancy corresponds to contrast-dependent characteristics; the CDS-Encoder therefore processes each modality with both a shared encoder $ES_C$ and a modality-specific stream $ES_i$, then fuses common and discrepant features at multiple scales (Zhang et al., 2023). In visible-infrared ReID, MRCN uses Instance Normalization to remove modality-dependent statistics, defines residuals $M_v = F_v - \widetilde{F}_v$ and $AC=\{ac_i\}_{i=1}^{4}$0, and then separates them into modality-irrelevant features for restitution and modality-relevant features for compensation; the compensated features are

$AC=\{ac_i\}_{i=1}^{4}$1

so each branch receives style information from the other modality (Zhang et al., 2023).

TMDC adopts a similar but more general split into modality-specific denoised representations $AC=\{ac_i\}_{i=1}^{4}$2 and modality-common denoised representations $AC=\{ac_i\}_{i=1}^{4}$3, then fuses them within each modality via

$AC=\{ac_i\}_{i=1}^{4}$4

and across modalities through cross-attention such as $AC=\{ac_i\}_{i=1}^{4}$5, yielding a complementary latent stream $AC=\{ac_i\}_{i=1}^{4}$6 for missing-modality prediction (Zhuang et al., 13 Nov 2025). DIS2 formalizes the same intuition in remote sensing through Distinct and Supplement branches: the Distinct branch captures modality-specific information, the Supplement branch captures compensatory information, and orthogonality plus distillation enforces non-redundancy and teacher-guided usefulness (Kieu et al., 20 Jan 2026).

Feature fusion mechanisms also encode different assumptions about missingness. DFUM is a canonical example. At each scale it fuses available modality features $AC=\{ac_i\}_{i=1}^{4}$7 using both hard integration

$AC=\{ac_i\}_{i=1}^{4}$8

and soft integration

$AC=\{ac_i\}_{i=1}^{4}$9

where $ac_i=1$0 is obtained from convolutions with kernel sizes $ac_i=1$1, $ac_i=1$2, and $ac_i=1$3; the concatenated result is then projected into a unified latent feature (Zhang et al., 2023). Hard integration emphasizes salient responses, while soft integration reduces information loss and supports spatially varying reliance on modalities.

Other architectures encode conditioning explicitly through masks. ADMC freezes observed modalities during reverse diffusion by applying

$ac_i=1$4

so only missing dimensions evolve under the denoising dynamics (Zhang et al., 8 Jul 2025). MMP uses a masked subset of modalities at each iteration and predicts missing tokens from available ones via cross-attention from learnable aggregated tokens of the missing modality to updated aggregated tokens of the present modalities (Nezakati et al., 2024). AM$ac_i=1$5-EmoJE inserts a Boolean mask vector $ac_i=1$6 directly into cross-attention keys and values so absent modalities are zeroed in the ACN, and then relies on joint embedding alignment to provide latent compensation (Devulapally et al., 2024). Missing-modality enabled tri-modal medical fusion uses architectural routing rather than explicit generation: if one modality is missing, only the surviving bi-modal module produces a 256-dimensional vector, and FRCL makes that vector approximate the full tri-modal fusion representation (Wang et al., 2023).

At a broader level, these mechanisms indicate that compensation is typically conditioned by at least one of three signals: an availability mask, a learned reliability or attention score, or a similarity structure over modalities or samples. This suggests that missing-modality handling is less a single algorithm than a family of conditional representation-alignment problems.

4. Objectives, supervision, and optimization strategies

The loss design determines what “compensation” means operationally. In unified MRI synthesis, compensation is optimized through a hybrid objective with synthesis loss on missing modalities,

$ac_i=1$7

reconstruction loss on present modalities,

$ac_i=1$8

and least-squares adversarial loss, combined as

$ac_i=1$9

with $ac_i=0$0, $ac_i=0$1, and $ac_i=0$2 (Zhang et al., 2023). Curriculum learning then exposes the model first to one missing modality, then two, then three, and finally random patterns.

Latent completion methods use objectives tailored to their generative process. ADMC minimizes a diffusion noise-prediction loss,

$ac_i=0$3

while preserving observed modalities with the mask constraint during reverse diffusion (Zhang et al., 8 Jul 2025). EMC’s CMA optimizes a CVAE objective $ac_i=0$4, where alignment is an $ac_i=0$5 loss between original and compensated embeddings and the KL term regularizes the latent distribution to a normal prior; a KL-based fusion consistency loss then aligns reaction distributions from full and compensated inputs (Hu et al., 2024). UniMRSeg distributes supervision hierarchically: Stage 1 uses $ac_i=0$6SSIM reconstruction, Stage 2 combines NT-Xent with Dice, and Stage 3 uses feature consistency

$ac_i=0$7

plus prediction consistency

$ac_i=0$8

with a frozen encoder and a trainable reverse attention adapter (Zhao et al., 19 Sep 2025).

Graph- and distillation-based methods define compensation through alignment with a privileged target. M$ac_i=0$9Care combines prediction loss with a stability regularizer on deep kernel mappings,

$f_i$0

so similarity learning remains task-guided but numerically stable (Zhang et al., 2022). DIS2 uses orthogonality loss between Distinct and Supplement features plus feature-level and logit-level distillation from the full-modality teacher, making missing-modality compensation an explicitly teacher-supervised latent approximation problem (Kieu et al., 20 Jan 2026). AdaMM does likewise in brain tumor segmentation, combining voxel-level MSE, bottleneck-level structural and adversarial distillation, and lesion-presence-guided reliability weighting (Zhu et al., 18 Sep 2025).

Non-generative methods encode a different objective. SimMLM formalizes the principle that more modalities should not increase task loss by defining the More vs. Fewer ranking loss

$f_i$1

where $f_i$2 (Li et al., 25 Jul 2025). MWAM instead computes a Frequency Ratio Metric, maps it to modality weights, and rescales gradients or auxiliary losses so dominant modalities do not suppress weaker ones during training (Lu et al., 26 Feb 2026). These objectives do not reconstruct missing data, but they directly optimize robustness under missingness.

5. Empirical evidence across domains

In medical MRI synthesis, unified compensation produces strong gains across one-to-one and many-to-one settings. On BraTS, the unified GAN reports for the T1+T2+T1Gd$f_i$3FLAIR scenario a PSNR of 31.94 and SSIM of 0.975, versus 31.06/0.970 for MM-Synthesis and 30.60/0.967 for MM-GAN. In one-to-one tasks it reports BraTS T1$f_i$4T2 PSNR 27.78 versus 27.10 and 26.94 for pGAN and Pix2Pix, and IXI PD$f_i$5T2 PSNR 30.93 versus 30.30 and 29.94. The ablation average over 14 BraTS scenarios gives DFUM 28.23/0.937 versus 27.79/0.932 for Max operation and 27.60/0.930 for HeMIS-style unification (Zhang et al., 2023).

In multimodal recognition, latent completion has comparable impact. ADMC reports on IEMOCAP average WA/UA of 68.7/70.2 for MMCR, versus 64.5/65.3 for IF-MMIN and 64.1/65.2 for MMIN, and on MIntRec 49.8/44.3 versus 45.5/39.9 and 45.3/39.0. Under missing rate 0.5 on IEMOCAP binary classification, it reports F1/ACC 81.4/82.7 versus 72.4/80.5 for TATE, 69.3/77.0 for MMIN, and 68.0/76.0 for TransM (Zhang et al., 8 Jul 2025). EMC reports an average FRCorr improvement of 57.2% over original model structures, with cases where missing speech improves appropriateness relative to the original full-modality baseline, while missing facial input leads only to minimal degradation (Hu et al., 2024). AM$f_i$6-EmoJE reports around 2–5% improvement in weighted-F1 and, more specifically, on MELD improves $f_i$7 from 0.6263 to 0.6836 with JE, $f_i$8 from 0.6196 to 0.6897, and $f_i$9 from 0.5285 to 0.6085; on IEMOCAP the corresponding gains are 0.6162 to 0.6919, 0.6343 to 0.7094, and 0.5379 to 0.6580 (Devulapally et al., 2024).

Transformer-token compensation also yields substantial robustness. MMT reduces performance loss from its original $^3$0 drop to only $^3$1 when half of the test set is modal-incomplete in egocentric video understanding (Ramazanova et al., 2024). MMP reports on NYUDv2 average mIoU 46.56 versus 44.77 for the best compared baseline, and in the RGB-missing case 41.08 versus 36.84 for TokenFusion and 36.72 for Reza et al.; on CMU-MOSI it reports average accuracy/F1 63.26/62.08 versus 61.68/60.11 for the best baseline (Nezakati et al., 2024).

Non-generative compensation remains competitive or superior. SimMLM reports on BraTS average Dice across all 15 configurations of ET 67.16, TC 80.20, WT 87.67, versus 66.05/79.14/86.71 for DMoME without MoFe and 65.56/78.58/86.69 for MoMKE; on UPMC Food-101 it reports 72.20 image-only, 87.20 text-only, and 94.99 image+text, exceeding ShaSpec and MoMKE; on avMNIST it reports 92.69 image-only, 91.61 audio-only, and 99.27 image+audio (Li et al., 25 Jul 2025). MWAM lowers performance collapse rates across segmentation, classification, detection, and fine-grained recognition; for example, on CASIA-SURF SF-MD+MWAM raises average accuracy from 92.85 to 96.06 and lowers average PCR from 5.43 to 3.61 (Lu et al., 26 Feb 2026).

Graph- and teacher-guided compensation is particularly effective in medical settings. M$^3$2Care reports on ODIR micro-AUC 0.8490 and macro-AUC 0.8245, versus approximately 0.8092 and lower for the best baselines, and on OV AUPRC 0.7549 and AUROC 0.7998, exceeding reported baseline values (Zhang et al., 2022). Missing-modality enabled tri-modal medical fusion reports full TriMF AUROC 0.914 and AUPRC 0.552, while under missing-tabular inference the full-trained model retains AUROC 0.912 versus 0.879 for a model trained only on image+text, and under missing-image inference retains 0.911 versus 0.875 for text+tabular-only training (Wang et al., 2023). In remote sensing, DIS2 outperforms prior methods under full and missing RGIR/NDSM scenarios, with particularly large gains on the “car” class and the lowest feature-distance between missing and full scenarios (Kieu et al., 20 Jan 2026). In brain tumor segmentation, AdaMM reports on BraTS 2024 FLAIR-only WT Dice 80.55 and T1Gd-only ET Dice 75.48, while maintaining lower WT HD95 than compared methods; on BraTS 2018 it reports average WT Dice 86.94 and ET Dice 63.75 (Zhu et al., 18 Sep 2025).

These results collectively indicate that compensation modules are not confined to one modality type, task, or backbone family. They are effective in synthesis, segmentation, classification, re-identification, sentiment analysis, and reaction generation, provided the compensation locus is matched to the task’s error mode.

6. Misconceptions, trade-offs, and future directions

A common misconception is that a Missing-Modality Compensation Module is synonymous with generating raw missing data. The literature does not support that equivalence. Some modules synthesize pixels or sequences (Zhang et al., 2023), but others complete latent features (Zhang et al., 8 Jul 2025), project missing tokens (Nezakati et al., 2024), inject learned missing-modality tokens (Ramazanova et al., 2024), retrieve task-related latent information from similar patients (Zhang et al., 2022), or simply rebalance expert contributions and training dynamics without generation (Li et al., 25 Jul 2025, Lu et al., 26 Feb 2026). Another misconception is that modality-invariant representations alone are sufficient. In strongly heterogeneous settings, several papers argue the opposite: modality-specific or compensatory features remain necessary because discriminative evidence is often modality-exclusive (Zhang et al., 2023, Zhang et al., 2023, Kieu et al., 20 Jan 2026, Zhuang et al., 13 Nov 2025).

The main trade-offs are architectural cost, dependence on training distribution, and the risk of over-coupling or artifact propagation. Multi-stream image synthesizers require multiple encoders and decoders and therefore substantial memory; the original unified GAN notes that 4 modality-specific encoders plus 1 common encoder and 4 decoders increase parameter count and GPU memory usage, especially when scaling beyond four modalities (Zhang et al., 2023). Generative modules may produce visually plausible but clinically insufficient outputs, and full clinical validation is often absent (Zhang et al., 2023). Diffusion and CVAE feature completers can reduce over-coupling by freezing unimodal encoders, but their success still depends on the quality of learned joint distributions and on having full-modality data during training (Zhang et al., 8 Jul 2025, Hu et al., 2024, Nezakati et al., 2024). SimMLM’s MoFe embodies an intuitive principle—more modalities should not hurt—but its own discussion notes that this assumption may fail when modalities are noisy or adversarial (Li et al., 25 Jul 2025).

Future directions in the source material cluster around four themes. The first is lightweight design: more parameter sharing across modality-specific streams, depthwise separable convolutions, bottleneck structures, or modality-conditioned backbones are explicitly proposed as ways to reduce memory and computation (Zhang et al., 2023). The second is task-aware compensation: several works suggest jointly optimizing compensation with downstream segmentation or classification so synthesized or reweighted features are judged by utility rather than by appearance alone (Zhang et al., 2023, Hu et al., 2024). The third is interpretability and uncertainty: visualizing attention maps, modality-wise importance, DFUM weights, or lesion-presence priors, and attaching uncertainty estimates to synthesized modalities, are repeatedly proposed or implied (Zhang et al., 2023, Lu et al., 26 Feb 2026, Zhu et al., 18 Sep 2025). The fourth is broader generalization: multi-center domain adaptation, mixed-modality settings such as CT/MR/PET or RGB/depth/thermal, and extension from image or feature compensation to token- and prompt-space compensation all appear as active directions (Zhang et al., 2023, Nezakati et al., 2024, Zhang et al., 19 Sep 2025).

Taken together, the literature suggests a stable core definition: a Missing-Modality Compensation Module is any learnable mechanism that transforms incomplete multimodal evidence into a representation whose geometry, semantics, or predictive behavior approximates the full-modality case closely enough for robust downstream inference. The specific implementation—synthesis, completion, token substitution, graph propagation, expert reweighting, or distillation—depends primarily on where the modality gap is most harmful in the target system.