Adaptive Modality Dropout

• Adaptive modality dropout is a dynamic multimodal strategy that adjusts masking based on modality contribution, token importance, or training stage.
• It reduces modality dominance by selectively suppressing strong signals, ensuring balanced cross-modal inputs and improved robustness when data is missing.
• Adaptive techniques reveal trade-offs between fusion accuracy and robustness, with optimal dropout rates yielding improved regularization and performance.

Adaptive modality dropout is a class of multimodal training and inference strategies that suppresses, attenuates, or substitutes modality-conditioned information in order to reduce modality dominance, improve robustness to missing inputs, and preserve cross-modal complementarity. In the recent literature, the term is used inconsistently. Some works reserve “adaptive” for mechanisms whose dropout behavior depends on modality contribution, token importance, or training stage, whereas many papers using “modality dropout” actually implement fixed stochastic masking with preset probabilities and no learned control (Wang et al., 7 Mar 2026, Chen et al., 8 Dec 2025, Liu et al., 7 Jan 2026, Qi et al., 2024). The design space therefore spans static whole-modality masking, curriculum-based attenuation, performance-driven per-modality dropout, and multimodal token-level dropout.

1. Definition and terminological scope

In the strict sense, adaptive modality dropout refers to dropout policies that are not fixed globally, but vary with signals such as modality performance, modality contribution, input structure, or training time. The clearest instance in the surveyed literature is AMB-DSGDN, where a dropout probability $q_m$ is recomputed for each modality from batch-level weighted F1 estimates of modality-specific emotion-recognition performance, so that stronger modalities receive higher dropout probability (Wang et al., 7 Mar 2026). A different strict form appears in Dropout Prompt Learning, where the dropout probability of each token is derived from token significance computed from intra-modal self-attention, task-token attention, and cross-modal alignment, yielding per-sample, per-layer, per-token adaptive probabilities rather than fixed rates (Chen et al., 8 Dec 2025).

By contrast, several representative papers explicitly state that their modality-dropout mechanisms are not adaptive in this sense. MGPC uses a fixed stochastic all-or-nothing dropout of the combined auxiliary image-plus-text branch, replacing it with learnable placeholder tokens $T_{learn}$ when dropped (Liu et al., 7 Jan 2026). The MER2024-SEMI system zeros each of four modality embeddings with a single fixed probability $p_1$ before attention-based fusion (Qi et al., 2024). Earlier work such as “Input Dropout” performs sample-wise whole-modality masking at the input level with preset probabilities, primarily for the regime in which auxiliary modalities are available during training but absent at test time (Blois et al., 2020). Performance-driven talking-face animation similarly uses manually tuned audio and video dropout probabilities to force the network to exploit both streams, but does not learn or condition those probabilities on the sample (Abdelaziz et al., 2020).

A weaker usage of “adaptive” appears in curriculum-style masking. The thrombus-segmentation paper introduces gradual modality dropout, where the decision to corrupt a modality is still Bernoulli, but the severity of corruption follows an epoch-wise attenuation schedule $g_j(t)$ that moves from partial visibility to full blackout. The paper explicitly frames this as closer to curriculum-based modality masking than to a fully data-driven adaptive policy (Vargas-Ibarra et al., 1 Apr 2026).

2. Historical development and taxonomy

The literature exhibits a progression from fixed, whole-modality masking toward finer-grained and more explicitly controlled mechanisms. Early approaches in 2020 emphasized robustness and regularization through simple stochastic removal of modality streams. “Input Dropout” assumed spatially aligned modalities such as RGB with depth, segmentation, or thermal channels, and zeroed entire modality channel groups during training while always testing with RGB only (Blois et al., 2020). In audiovisual talking-face animation, batches mixed audiovisual, video-only, and audio-only conditions to prevent the visual stream from dominating the regression to speech-related facial controls (Abdelaziz et al., 2020).

Later work extended the same principle to broader multimodal settings without making the masking adaptive. In multimodal emotion recognition, each modality embedding $e_S,e_I,e_T,e_V$ could be replaced by a zero vector before attention-based fusion, and the best fixed dropout rate was $0.3$ (Qi et al., 2024). MGPC moved this pattern into a multimodal Transformer for point cloud completion, but retained a fixed stochastic decision over the combined auxiliary branch rather than learned or per-modality masking (Liu et al., 7 Jan 2026). In multi-modal target speaker extraction, modality dropout training sampled three auxiliary conditions—both present, video only, or audio only—with equal probability, again using a fixed policy rather than adaptive scheduling or reliability-aware gating (Korse et al., 9 Jul 2025).

Recent work differentiates several distinct regimes.

Regime	Representative papers	Characteristic decision rule
Static whole-modality masking	(Blois et al., 2020, Qi et al., 2024, Liu et al., 7 Jan 2026)	Fixed-probability zeroing or substitution
Curriculum-based attenuation	(Vargas-Ibarra et al., 1 Apr 2026)	Epoch-wise severity schedule
Performance-driven modality balancing	(Wang et al., 7 Mar 2026)	Batch-level modality-dependent $q_m$
Adaptive multimodal token dropout	(Chen et al., 8 Dec 2025)	Per-token importance-derived $p_j$
Missingness-aware explicit subset supervision	(Gu et al., 22 Sep 2025)	All nonempty subsets trained jointly

This taxonomy also clarifies that several mechanisms adjacent to modality dropout are not themselves dropout policies. MDA-KD analyzes dropout-induced modality bias and corrects it with hidden-layer knowledge distillation, while MS-Adapter dynamically switches to an audio-specialized pathway when video is entirely absent (Dai et al., 2024). HAM replaces dropout-robust fixed networks with a hypernetwork that generates task-model parameters conditioned on the binary modality-availability vector $\boldsymbol{\mu}$ (Fürböck et al., 14 Sep 2025).

3. Core mathematical formulations

Static whole-modality masking is typically implemented by replacing an entire modality or modality bundle with zeros or a learned substitute. In MGPC, the condition tokens used for fusion are

$$T_c = \begin{cases} [T_i, T_t], & \text{if } z \geq p,\ T_{learn}, & \text{if } z < p, \end{cases}$$

where $T_{learn}$0, $T_{learn}$1 is the modality dropout probability, and $T_{learn}$2 is a trainable empty-condition representation matched to the dimensionality of the image and text tokens (Liu et al., 7 Jan 2026). The paper is explicit that this acts on the entire auxiliary condition branch and does not selectively drop image patches, text dimensions, or feature channels.

A per-modality static version appears in MER2024-SEMI. The paper informally writes the fusion rule as

$T_{learn}$3

which the authors’ own reconstruction interprets as independently replacing each modality embedding by zeros with probability $T_{learn}$4 before concatenation and attention-based fusion (Qi et al., 2024). A closely related stochastic mixture over modality states is used in multi-modal target speaker extraction: $T_{learn}$5 with both auxiliary modalities never dropped simultaneously (Korse et al., 9 Jul 2025).

Performance-driven adaptive modality dropout is formalized most explicitly in AMB-DSGDN. For modality $T_{learn}$6, a modality performance metric $T_{learn}$7 is computed from weighted F1-like statistics, relative advantage ratios are formed as

$T_{learn}$8

and the dropout probability is then

$T_{learn}$9

with $p_1$0, $p_1$1, and $p_1$2 in the main implementation (Wang et al., 7 Mar 2026). During execution, masks are sampled as

$p_1$3

and retained features are globally rescaled by $p_1$4, where $p_1$5 depends on the aggregate dropout rate across modalities.

Adaptive token-level dropout in vision-LLMs uses a different control variable. Dropout Prompt Learning defines token significance as

$p_1$6

then maps normalized significance inversely to dropout probability: $p_1$7 Here $p_1$8 is derived from intra-modal self-attention, $p_1$9 from attention of the global task token, and $g_j(t)$0 from bridge-token-based cross-modal alignment. The method excludes $g_j(t)$1 and $g_j(t)$2 from dropout (Chen et al., 8 Dec 2025).

Curriculum-style attenuation occupies an intermediate position. Gradual modality dropout first samples a Bernoulli retain variable $g_j(t)$3, but when a modality is selected for corruption it is multiplied by a time-dependent attenuation value rather than immediately zeroed: $g_j(t)$4 with a discrete schedule

$g_j(t)$5

and an added Gaussian perturbation of mean $g_j(t)$6 and standard deviation $g_j(t)$7 (Vargas-Ibarra et al., 1 Apr 2026).

4. Architectural placement and representations of missingness

A central design decision is where dropout is inserted. In many systems it is applied after modality-specific encoding and before fusion. MGPC extracts point tokens $g_j(t)$8, image tokens $g_j(t)$9, and a text token $e_S,e_I,e_T,e_V$0; modality dropout converts the auxiliary image-plus-text condition into either $e_S,e_I,e_T,e_V$1 or $e_S,e_I,e_T,e_V$2, and the resulting $e_S,e_I,e_T,e_V$3 is consumed by cross-attention in which point tokens remain the queries and the condition tokens are the keys and values (Liu et al., 7 Jan 2026). MER2024-SEMI first computes pooled modality embeddings $e_S,e_I,e_T,e_V$4, then zero-masks some of them before concatenation and attention-based fusion (Qi et al., 2024). In multi-modal target speaker extraction, dropout is applied directly to the auxiliary embeddings before attentive combination and multiplicative fusion with the mixture representation $e_S,e_I,e_T,e_V$5 (Korse et al., 9 Jul 2025).

Other systems apply dropout at raw input level. “Input Dropout” zeroes whole modality channel groups before any feature extraction and requires only that the first convolutional layer be widened to accept the concatenated multimodal tensor (Blois et al., 2020). Gradual modality dropout in MRI segmentation likewise multiplies raw modality images by attenuation coefficients before the segmentation network, representing absent sequences as black images (Vargas-Ibarra et al., 1 Apr 2026).

The representation of missingness varies substantially. The most common encoding is an all-zero vector or black image, as in MER2024-SEMI, talking-face animation, input dropout, the thrombus-segmentation setting, and the pediatric EHR late-fusion model’s zero-masking evaluations (Qi et al., 2024, Abdelaziz et al., 2020, Blois et al., 2020, Vargas-Ibarra et al., 1 Apr 2026, Yang et al., 10 Apr 2026). Several recent methods instead use learned missingness surrogates. MGPC uses trainable placeholder tokens $e_S,e_I,e_T,e_V$6 initialized from a standard normal distribution (Liu et al., 7 Jan 2026). The medical contrastive-fusion framework replaces fixed zero placeholders with learnable modality tokens $e_S,e_I,e_T,e_V$7 and $e_S,e_I,e_T,e_V$8, so missing CT and missing tabular data receive modality-specific trainable representations (Gu et al., 22 Sep 2025). HAM moves further away from masking by conditioning a hypernetwork on the binary availability vector $e_S,e_I,e_T,e_V$9 and generating a task model specialized to the observed modality subset (Fürböck et al., 14 Sep 2025).

These design choices reflect different assumptions. Zero masking is simple and shape-preserving, but it conflates “absent modality” with a degenerate feature value. Learned tokens preserve tensor interfaces while allowing the network to learn a default missingness prior. Hypernetwork conditioning treats missingness as a change in the model itself rather than merely a perturbation of the input representation. The available evidence suggests that missingness representation becomes especially consequential when the training objective explicitly includes incomplete subsets, as in simultaneous modality dropout with learnable tokens (Gu et al., 22 Sep 2025).

5. Training objectives and empirical behavior

Most modality-dropout systems do not introduce a dedicated dropout loss; the effect enters through the forward path while the task loss remains unchanged. MGPC uses HyperCD reconstruction loss at multiple decoder scales and adds no alignment, consistency, or placeholder-token regularizer specific to dropout (Liu et al., 7 Jan 2026). MER2024-SEMI similarly uses the usual classification objective and attributes robustness solely to exposure to random missing-modality patterns during training (Qi et al., 2024). The target speaker extraction study contrasts standard training, multi-task training over all modality states, and stochastic modality dropout training, all with SI-SDR-based supervision rather than a special missing-modality penalty (Korse et al., 9 Jul 2025).

Empirically, fixed stochastic masking typically exhibits a non-monotonic dependence on dropout rate. In MGPC, the ablation over $0.3$0 showed the best performance at $0.3$1, with CD-$0.3$2 $0.3$3, CD-$0.3$4 $0.3$5, and F1 $0.3$6, compared with $0.3$7 at $0.3$8 and $0.3$9 at $q_m$0. The “Modalities Missing” inference condition, where both image and text were removed, produced CD-$q_m$1 $q_m$2, CD-$q_m$3 $q_m$4, and F1 $q_m$5, demonstrating partial but not cost-free robustness (Liu et al., 7 Jan 2026). In MER2024-SEMI, a fixed dropout rate of $q_m$6 yielded the best test accuracy of $q_m$7, compared with $q_m$8 at dropout $q_m$9 and $p_j$0 at $p_j$1, while TrainVal accuracy decreased slightly as dropout increased from $p_j$2 to $p_j$3, which the paper interprets as regularization improving generalization (Qi et al., 2024).

The same pattern appears in other domains. For performance-driven talking faces, viewers preferred audiovisual-driven animation in $p_j$4 of test sequences before modality dropout and $p_j$5 after modality dropout, while the corresponding preference for video-only animation dropped from $p_j$6 to $p_j$7 (Abdelaziz et al., 2020). In spatially aligned multimodal vision, input dropout improved RGB-only test-time performance across several tasks, including dehazing from PSNR $p_j$8 to $p_j$9 on RGB+D and from $\boldsymbol{\mu}$0 to $\boldsymbol{\mu}$1 on RGB+segmentation, as well as classification on NYU V2 from $\boldsymbol{\mu}$2 to $\boldsymbol{\mu}$3 (Blois et al., 2020). In pediatric emergency triage, symmetric late-fusion modality dropout improved zero-shot pediatric Quadratic Weighted Kappa from $\boldsymbol{\mu}$4 without dropout to $\boldsymbol{\mu}$5 at $\boldsymbol{\mu}$6–$\boldsymbol{\mu}$7 dropout, while adult in-domain QWK changed only from $\boldsymbol{\mu}$8 to $\boldsymbol{\mu}$9 (Yang et al., 10 Apr 2026).

More explicitly adaptive methods show a similar regularization profile, but their gains are targeted at modality balance rather than only missingness robustness. In AMB-DSGDN, removing modality balancing reduced IEMOCAP performance from wa-F1 $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \geq p,\ T_{learn}, & \text{if } z < p, \end{cases}$$0 and wa-ACC $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \geq p,\ T_{learn}, & \text{if } z < p, \end{cases}$$1 in the full model to wa-F1 $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \geq p,\ T_{learn}, & \text{if } z < p, \end{cases}$$2 and wa-ACC $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \geq p,\ T_{learn}, & \text{if } z < p, \end{cases}$$3; on MELD the gains were smaller, from $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \geq p,\ T_{learn}, & \text{if } z < p, \end{cases}$$4 to $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \geq p,\ T_{learn}, & \text{if } z < p, \end{cases}$$5 (Wang et al., 7 Mar 2026). The thrombus-segmentation study reports that gradual dropout was especially helpful under unseen-center missing-modality conditions: in the out-domain B0-missing lesion-segmentation experiment, gradual dropout with $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \geq p,\ T_{learn}, & \text{if } z < p, \end{cases}$$6 achieved unseen ISLES Dice $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \geq p,\ T_{learn}, & \text{if } z < p, \end{cases}$$7, compared with $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \geq p,\ T_{learn}, & \text{if } z < p, \end{cases}$$8 for MultiUnet and $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \geq p,\ T_{learn}, & \text{if } z < p, \end{cases}$$9 for ModDrop++ at the same setting (Vargas-Ibarra et al., 1 Apr 2026). In vision-language prompt learning, adaptive multimodal token dropout achieved an average base-to-novel harmonic mean of $T_{learn}$00, outperforming KgCoOp at $T_{learn}$01, PSRC at $T_{learn}$02, TAP at $T_{learn}$03, and TAC at $T_{learn}$04; uniform token dropout baselines such as $T_{learn}$05 dropout and DropBlock were worse than the adaptive scheme (Chen et al., 8 Dec 2025).

A recurrent empirical theme is that robustness is not free. Excessive dropout can underuse complementary signals, destabilize actual fusion, or induce a shift toward a dominant unimodal solution. The AVSR study makes this explicit through the Modality Bias Hypothesis: increasing video dropout improves robustness to missing video, but can drive the multimodal model toward an overly audio-dominant decision rule whose CER curve approaches that of audio-only ASR. MDA-KD is proposed precisely to counter this dropout-induced modality bias, while MS-Adapter handles the fully missing-video case through dynamic inference-time switching (Dai et al., 2024).

6. Misconceptions, limitations, and adjacent alternatives

A common misconception is that any “modality dropout” is adaptive. Several papers explicitly warn against this interpretation. MGPC’s mechanism is random, fixed-probability, and sample-independent except for the stochastic draw; it has no learned gating variable, no confidence score, no per-modality probability, no epoch schedule, and no dynamic token masking strategy (Liu et al., 7 Jan 2026). MER2024-SEMI likewise uses a single static dropout rate and does not condition masking on input content, modality quality, or confidence (Qi et al., 2024). The pediatric triage study uses symmetric and asymmetric fixed rates selected offline rather than learned policies (Yang et al., 10 Apr 2026). These methods are important baselines, but they are not adaptive in the strong sense.

A second misconception is that robustness to missing modalities automatically implies balanced multimodal use. The AVSR results show that naive dropout can improve missing-video robustness by making the model behave more like audio-only ASR, thereby sacrificing multimodal synergy under complete input (Dai et al., 2024). The target speaker extraction study reaches a related conclusion from a different angle: standard training and even explicit multi-task enumeration can remain susceptible to modality dominance, whereas stochastic modality dropout training yields more balanced AoTSE and VoTSE behavior across normalization choices (Korse et al., 9 Jul 2025). This suggests that robustness should be evaluated together with modality balance and full-modality performance, not only by the slope of degradation under missingness.

A third limitation concerns granularity. Whole-modality dropout does not cover corrupted, contradictory, or weakly aligned modalities unless those cases are simulated directly. Input Dropout assumes strong spatial alignment and experiments only with one additional modality paired with RGB (Blois et al., 2020). Gradual modality dropout simulates missing sequences by black images and intensity attenuation, which matches scanner-driven absence but does not directly model misregistration or more complex artifact patterns (Vargas-Ibarra et al., 1 Apr 2026). AMB-DSGDN adapts probabilities only at batch level rather than utterance level, despite motivating fluctuations in modality usefulness across dialogue stages (Wang et al., 7 Mar 2026). DroPLe is highly adaptive, but only at token level within a CLIP-style prompt-learning setup and not for full-modality absence (Chen et al., 8 Dec 2025).

These limitations motivate adjacent alternatives beyond dropout itself. The medical contrastive-fusion framework combines simultaneous supervision of all nonempty modality subsets with learnable missing tokens $T_{learn}$06, avoiding random under-supervision while remaining missingness-aware (Gu et al., 22 Sep 2025). HAM removes the fixed-model compromise entirely by generating a task model specialized to the observed modality subset $T_{learn}$07, and reports an absolute balanced-accuracy increase of up to $T_{learn}$08 at $T_{learn}$09 completeness relative to complete-case training (Fürböck et al., 14 Sep 2025). A plausible implication is that future adaptive modality dropout methods will increasingly integrate three ingredients already visible in the current literature: explicit modeling of modality contribution, learned representations of missingness, and dynamic routing or model specialization when entire modalities are absent.