Modality Dropout Training in Multimodal Systems

• Modality Dropout Training (MDT) is a multimodal training method that deliberately suppresses entire input modalities to improve robustness and mitigate modality over-dependence.
• It employs techniques like zeroing input channels, stochastic masking of embeddings, and learnable placeholder tokens to simulate missing data conditions.
• Empirical evidence shows that moderate MDT boosts performance in tasks such as segmentation, emotion recognition, and action recognition while ensuring graceful degradation with incomplete modalities.

Modality Dropout Training (MDT) denotes a family of multimodal training procedures in which entire modalities, modality-conditioned feature blocks, or modality-specific paths are deliberately suppressed, attenuated, or replaced during training so that a single model learns to operate under both full- and missing-modality conditions. In the cited literature, MDT appears as random zeroing of whole input channels, stochastic masking of embedding or score vectors, replacement by learnable placeholder tokens, gradual contrast attenuation of missing modalities, and learned sample-wise exclusion of irrelevant modalities (Lau et al., 2019, Blois et al., 2020, Liu et al., 7 Jan 2026). Its primary roles are regularization, mitigation of modality dominance or modality collapse, and robustness to deployment settings in which only a subset of modalities is available (Yang et al., 10 Apr 2026, Korse et al., 9 Jul 2025, Dai et al., 2024).

1. Conceptual scope and distinction from ordinary dropout

MDT differs from standard dropout, DropPath, and ordinary data augmentation in the granularity of what is removed. Standard dropout zeros individual neuron activations or feature dimensions; DropPath removes residual branches or layers; MDT removes an entire modality, or an entire modality-conditioned feature set, as a unit. In the segmentation setting with missing MRI modalities, modality dropout was defined as “randomly zeroing out entire input channels,” with each imaging modality corresponding to one input channel (Lau et al., 2019). “Input Dropout” made the same principle explicit for spatially aligned modalities such as RGB, depth, thermal, or semantic channels that are available during training but not necessarily at test time (Blois et al., 2020). In MGPC, the dropped object is not a raw channel but the whole auxiliary condition-token set built from image and text, which is replaced by learnable placeholder tokens before cross-attention (Liu et al., 7 Jan 2026).

Across the literature, MDT is motivated by a common train–test mismatch. A model trained only with all modalities present may overfit to the easiest or strongest modality and fail when that modality is absent at inference. Clinical triage models described this failure mode as modality collapse toward structured vitals, while multi-modal target speaker extraction described it as modality dominance, and AVSR described a related phenomenon as dropout-induced modality bias toward audio (Yang et al., 10 Apr 2026, Korse et al., 9 Jul 2025, Dai et al., 2024). This suggests that MDT is best understood not as a narrow regularizer but as a robustness mechanism for incomplete, unreliable, or deployment-mismatched multimodal inputs.

Representative setting	Dropped unit	Missingness representation
MGPC point cloud completion (Liu et al., 7 Jan 2026)	Joint image+text condition tokens	Learnable placeholder tokens $T_{\text{learn}}$
Pediatric triage late fusion (Yang et al., 10 Apr 2026)	5-dimensional probability block per modality	Zero-masked vectors
Emotion recognition (Qi et al., 2024)	Pooled modality embedding $e_m$	Zero vector
Thrombus segmentation (Vargas-Ibarra et al., 1 Apr 2026)	Whole MRI modality channel	Gradual scaling $g(t)+\epsilon$ to black image
Action recognition (Alfasly et al., 2022)	Audio embedding when irrelevant	Hard mask $\delta=0$

2. Granularity, masking regimes, and missingness encodings

The granularity of MDT varies substantially across domains. MGPC applies dropout to the condition token set $T_c$ before Transformer cross-attention: $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \ge p\ T_{\text{learn}}, & \text{if } z < p, \end{cases}$$ with $z \sim \mathcal{U}(0,1)$, $p$ a dropout probability, and $T_{\text{learn}}$ a learnable token bank matching the shape of $[T_i,T_t]$ (Liu et al., 7 Jan 2026). The primary point-cloud tokens are never dropped, so the design is asymmetric: geometry is always present, while image and text are jointly toggled as auxiliary conditioning.

Late-fusion clinical triage instead applies symmetric or asymmetric Bernoulli masking to entire modality probability vectors at the meta-classifier input: $e_m$0 with $e_m$1 and a shared symmetric dropout rate $e_m$2 in the main study (Yang et al., 10 Apr 2026). Emotion recognition uses the same whole-modality logic at embedding level,

$e_m$3

for speech, image, text, and video embeddings before concatenation and fusion (Qi et al., 2024).

Other papers constrain the sampling regime more explicitly. In multi-modal target speaker extraction, the joint embedding is chosen with equal probability among full multimodal, audio-only, and video-only states,

$e_m$4

so that not all modalities are dropped simultaneously (Korse et al., 9 Jul 2025). In multimodal dialogue, the stochasticity is layer-wise rather than sample-wise: each encoder layer randomly chooses a text-only stream, a text-enhanced image stream, or their average, with probabilities derived from $e_m$5 (Sun et al., 2021). In action recognition, the dropout decision is not random at all: a learned relevance score $e_m$6 and threshold $e_m$7 determine whether the audio embedding is completely dropped as irrelevant for that sample (Alfasly et al., 2022).

Missingness encodings also differ. Zero vectors remain common, but several papers replace zeros with structured substitutes. Improved multimodal fusion for disease detection uses learnable missing-modality tokens $e_m$8 and $e_m$9 instead of zeros, while GMD in thrombus segmentation attenuates a dropped modality by a schedule $g(t)+\epsilon$0 plus $g(t)+\epsilon$1 before reaching a true black image (Gu et al., 22 Sep 2025, Vargas-Ibarra et al., 1 Apr 2026). This suggests that “missing modality” is not a single representation but an architectural design choice.

3. Architectural insertion points

MDT has been inserted at nearly every stage of multimodal systems. At the raw-input level, segmentation and aligned-modality vision papers zero entire input channels before the backbone, so the rest of the architecture remains unchanged (Lau et al., 2019, Blois et al., 2020). In thrombus segmentation, GMD is applied directly to selected MRI channels before cross-attention and recurrent slice aggregation, so the attention module learns from dimmed, partially visible, and fully black susceptibility sequences (Vargas-Ibarra et al., 1 Apr 2026). In ModDrop++, whole MRI modalities are zeroed at the input, but the first convolutional layer is further conditioned on a modality code by a dynamic head (Liu et al., 2022).

At the representation level, MDT is often attached to the fusion interface. MER applies independent Bernoulli masks to pooled unimodal embeddings immediately before concatenation and the attention-based fusion network (Qi et al., 2024). DDSD applies modality dropout to score or embedding branches at the inputs of late and intermediate fusion models; missing scores are represented by $g(t)+\epsilon$2 and missing embeddings by arrays filled with $g(t)+\epsilon$3 during robustness evaluation (Krishna et al., 2023). Talking-face animation zeroes either the audio tensor or the visual tensor before their respective encoders, and the losses attached to video-only outputs are disabled when video is dropped (Abdelaziz et al., 2020).

In token-based multimodal Transformers, MDT often acts on condition tokens rather than raw modalities. MGPC constructs point tokens $g(t)+\epsilon$4, image tokens $g(t)+\epsilon$5, and a text token $g(t)+\epsilon$6, then replaces the whole condition token set by $g(t)+\epsilon$7 before Transformer cross-attention from points to conditions (Liu et al., 7 Jan 2026). The cross-attention operator itself remains unchanged; robustness emerges because the Transformer sees both real and placeholder-conditioned contexts under a fixed interface. In AVSR, dropout is even more localized: video frames are replaced by zero-padded frames only at the input of the video branch, leaving the audio branch intact, and the resulting bias shift toward audio becomes a central object of analysis (Dai et al., 2024).

Late-fusion systems provide a complementary design point. The pediatric triage model trains XGBoost and Bio_ClinicalBERT first, freezes them, then applies MDT only while training the Logistic Regression meta-classifier on the concatenated 10-dimensional probability vector $g(t)+\epsilon$8 (Yang et al., 10 Apr 2026). This demonstrates that MDT need not be built into encoders; it can operate purely at the decision-fusion layer.

4. Objectives, schedules, and auxiliary supervision

In many implementations, MDT does not alter the task loss. MGPC uses the same multiscale Hyperbolic Chamfer Distance whether image and text are present or replaced by $g(t)+\epsilon$9 (Liu et al., 7 Jan 2026). MER keeps its 6-way cross-entropy unchanged under modality masking (Qi et al., 2024). DDSD retains weighted binary cross-entropy while randomly dropping score or embedding branches (Krishna et al., 2023). Multi-modal target speaker extraction uses the same SI-SDR loss under all three training states in its single-pass MDT regime (Korse et al., 9 Jul 2025). Talking-face animation likewise keeps the speech-related losses active but zeroes the video-only losses when video is dropped (Abdelaziz et al., 2020).

Other papers augment MDT with explicit subset supervision or consistency terms. Improved modality dropout for disease detection replaces stochastic subset sampling with simultaneous modality dropout: for $\delta=0$0, the model is supervised on full multimodal, CT-only, and tabular-only predictions in each forward pass,

$\delta=0$1

while missing modalities are represented by learnable tokens $\delta=0$2 and $\delta=0$3 (Gu et al., 22 Sep 2025). ModDrop++ adds intra-subject co-training: the full-modality and missing-modality versions of the same subject share a segmentation objective and an SSIM-based feature similarity term between shallow feature maps (Liu et al., 2022). AVSR supplements dropout with a teacher–student hidden-distribution matching term in MDA-KD and later introduces MS-Adapters for the entirely missing-modality regime (Dai et al., 2024).

Scheduling also varies. Many papers use fixed probabilities throughout training; MGPC fixes $\delta=0$4, MER fixes $\delta=0$5 per run, pediatric triage evaluates symmetric rates from 10% to 60%, and DDSD tunes per-modality probabilities on validation data (Liu et al., 7 Jan 2026, Qi et al., 2024, Yang et al., 10 Apr 2026, Krishna et al., 2023). By contrast, gradual modality dropout in thrombus segmentation uses a piecewise schedule

$\delta=0$6

so “dropped” modalities are first dimmed and only later blacked out, making MDT a curriculum over missingness severity rather than just a fixed Bernoulli mask (Vargas-Ibarra et al., 1 Apr 2026).

5. Empirical regularities

Across tasks, moderate MDT usually improves robustness and often improves the main metric even when all modalities are present at test time. The pattern is visible in point cloud completion, clinical triage, multimodal emotion recognition, device-directed speech detection, and co-learning under unimodal deployment (Liu et al., 7 Jan 2026, Yang et al., 10 Apr 2026, Qi et al., 2024, Krishna et al., 2023, Magal et al., 1 Jan 2025).

Setting	No MDT	Best reported MDT
MGPC point cloud completion (Liu et al., 7 Jan 2026)	$\delta=0$7: 0.399 / 8.85 / 0.757	$\delta=0$8: 0.378 / 8.57 / 0.772
Pediatric triage, pediatric cohort (Yang et al., 10 Apr 2026)	QWK 0.331, Accuracy 0.559	QWK 0.351, Accuracy 0.571 at 30–40%
Emotion recognition (Qi et al., 2024)	WAF 89.52 at $\delta=0$9	WAF 90.15 at $T_c$0
DDSD with 30% missing modalities (Krishna et al., 2023)	11.46 FA@10% FR, 10.43 EER	10.61 FA@10% FR, 10.17 EER
bi-EFLSTM on IEMOCAP, language-only test (Magal et al., 1 Jan 2025)	27%	47% with 80% audio/visual dropout; unimodal baseline 45%

MGPC illustrates both regularization and graceful degradation. Moderate dropout improves CD-$T_c$1, CD-$T_c$2, and F-score, while inference with no image and no text still yields 0.470 / 9.10 / 0.749 after training with $T_c$3, rather than collapsing (Liu et al., 7 Jan 2026). Pediatric triage shows the same shape under distribution shift: adult-trained multimodal models already outperform unimodal baselines on pediatric zero-shot evaluation, and 30–40% symmetric modality dropout further raises QWK to 0.351 while preserving adult performance near 0.636 (Yang et al., 10 Apr 2026). In MER, $T_c$4 gives the highest test WAF, while $T_c$5 already constitutes over-regularization relative to the optimum (Qi et al., 2024).

Speech and talking-face tasks show that MDT can be particularly valuable when one modality dominates training. In DDSD, modality dropout improves false-accept performance under simulated missing-modality inference and also improves the full-modality setting (Krishna et al., 2023). In talking-face animation, viewer preference for audiovisual-driven animation relative to video-only rises from 51% to 74% after introducing modality dropout, while preference for video-only falls from 18% to 8% (Abdelaziz et al., 2020). In co-learning, aggressive audio/visual dropout converts a language-only deployment regime from negative co-learning to positive co-learning: the multimodal model’s language-only test accuracy rises from 27% to 47%, exceeding the unimodal baseline of 45% (Magal et al., 1 Jan 2025).

Medical segmentation work emphasizes robustness under missing-site or missing-sequence conditions. In multi-center thrombus segmentation, the lower-bound ISLES score with missing B0 is 0.622, whereas GMD reaches 0.701 on the unseen center at $T_c$6 (Vargas-Ibarra et al., 1 Apr 2026). In MS lesion segmentation, ModDrop++ improves over classic ModDrop and over ModDrop plus dynamic head alone across almost all missing-modality configurations on UMCL and ISBI (Liu et al., 2022). These results support the view that MDT is especially effective when deployment missingness is systematic rather than accidental.

6. Caveats, controversies, and adjacent alternatives

A central caveat is that MDT does not have a universally monotone effect. Multiple papers report that moderate dropout is beneficial, but too much dropout harms learning because the model rarely sees the informative multimodal regime. MGPC degrades at $T_c$7 and $T_c$8; pediatric triage degrades sharply at 50–60%; MER drops from 90.15 at $T_c$9 to 89.08 at $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \ge p\ T_{\text{learn}}, & \text{if } z < p, \end{cases}$$0 (Liu et al., 7 Jan 2026, Yang et al., 10 Apr 2026, Qi et al., 2024). This suggests that dropout probability is not a cosmetic hyperparameter but a principal control over the balance between robustness and multimodal utilization.

A second caveat is that MDT can induce or amplify modality bias depending on which modality is dropped. AVSR provides the clearest analysis: dropout on the supplementary video modality improves robustness to missing video but pushes the model toward audio-dominated hidden representations, worsening full-modality performance when dropout is too strong. This observation is formalized as the Modality Bias Hypothesis, and motivates MDA-KD and MS-Adapter as mechanisms for preserving multimodal performance while retaining missing-modality robustness (Dai et al., 2024). A related, but inverse, pattern appears in action recognition, where the problem is not missingness but irrelevance; here a learned relevance network drops audio only when predicted to be semantically irrelevant for the visually labeled action class (Alfasly et al., 2022).

A third issue is whether a single parameterization should cover all modality subsets. Standard MDT answers yes, but alternative work argues that this may waste capacity. “No Modality Left Behind” compares a hypernetwork that generates task-model parameters conditioned on the modality presence vector $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \ge p\ T_{\text{learn}}, & \text{if } z < p, \end{cases}$$1 against standard training, channel dropout, and imputation. At 25% completeness, the hypernetwork reaches balanced accuracy 0.46 versus 0.44 for channel dropout, and its abstract reports an absolute increase in accuracy of up to 8% over state-of-the-art dropout-based methods in the 25%-complete training regime (Fürböck et al., 14 Sep 2025). This suggests that dynamic model generation is a strong adjacent alternative when modality configurations are numerous and systematic.

Finally, several papers point toward learned MDT rather than fixed-rate MDT. The dropout-compaction framework introduces learnable retention probabilities with a sparsity-inducing prior and explicitly notes the extension from units to group-wise or modality-wise masks (Kubo et al., 2016). Within the multimodal literature itself, learnable missingness tokens, dynamic filter scaling, semantic relevance gating, and gradual schedules all move in that direction (Gu et al., 22 Sep 2025, Liu et al., 2022, Alfasly et al., 2022, Vargas-Ibarra et al., 1 Apr 2026). The cumulative implication is that MDT is no longer a single technique but a design axis: what to drop, when to drop it, how to encode absence, and whether absence is random, scheduled, supervised, or learned.