Interleaved Modality Dropout

• Interleaved Modality Dropout is a training strategy where entire modality streams are randomly dropped to force models to handle incomplete inputs.
• It employs various masking techniques—including sample-level, layer-wise, and window-level dropout—to simulate real-world missing modality conditions.
• Empirical studies demonstrate that moderate dropout rates improve robustness and generalization, while excessive masking can induce modality bias.

Searching arXiv for papers on modality dropout and interleaved multimodal training. Interleaved modality dropout, as an Editor’s term, denotes multimodal training procedures in which modality availability is varied across optimization steps, samples, layers, windows, or stages, so that a model repeatedly encounters both full and incomplete modality sets. In the literature assembled around this idea, the operative mechanisms range from whole-modality zeroing at the input or fusion interface, to layer-wise switching between modality-specific streams, to explicit supervision over all non-empty modality subsets. The term itself is not uniformly standardized: for example, MGPC describes “modality dropout” without using “interleaved,” while Orthus studies interleaved image–text sequences but does not implement modality dropout (Liu et al., 7 Jan 2026, Kou et al., 2024).

1. Terminology and conceptual scope

Within the cited literature, “modality dropout” consistently refers to dropping entire modality streams rather than individual neurons or features. MGPC defines a per-sample Bernoulli choice between real auxiliary condition tokens and a learnable placeholder token set $T_{learn}$, so the model alternates between a multimodal regime and a point-only regime during training (Liu et al., 7 Jan 2026). In audiovisual talking-face generation, batches contain audio-only, video-only, and audiovisual examples, again by dropping whole input streams rather than hidden units (Abdelaziz et al., 2020). In medical segmentation, ModDrop and related schemes randomly remove whole MRI channels so that a single model is exposed to many missing-modality configurations during training (Lau et al., 2019, Liu et al., 2022).

This suggests that “interleaving” is best understood operationally rather than terminologically. In some systems, interleaving occurs across training iterations because different samples encounter different modality subsets. In other systems, interleaving occurs across network depth: the non-hierarchical multimodal dialogue model switches, at each encoder layer, among a text-only stream, a text-enhanced visual stream, or their average (Sun et al., 2021). Orthus is relevant chiefly as a contrast case: it interleaves image and text within a single autoregressive sequence, but the paper explicitly states that it does not use “modality dropout,” “modality masking,” or any equivalent random removal scheme (Kou et al., 2024).

2. Canonical masking patterns

The literature contains several distinct masking patterns that fall under this broader notion.

Pattern	Mechanism	Representative papers
Sample-level all-or-none auxiliary masking	Replace the full auxiliary condition with learnable placeholders for a sample	MGPC (Liu et al., 7 Jan 2026)
Independent per-modality masking	Zero each modality embedding independently before fusion	MER2024 emotion recognition (Qi et al., 2024)
Layer-wise interleaving	Choose text-only, image-enhanced, or averaged features per encoder layer	Multimodal dialogue generation (Sun et al., 2021)
Window-level sequence masking	Zero an entire visual window while keeping audio active	ABAW expression recognition (Yu et al., 9 Mar 2026)
Input-channel masking	Zero full input modalities treated as extra channels	Spatially aligned vision tasks (Blois et al., 2020)
Exhaustive subset supervision	Optimize full and unimodal subsets simultaneously for each sample	Clinical fusion with learnable tokens (Gu et al., 22 Sep 2025)

MGPC provides the clearest example of structured sample-level masking. Its condition tokens are defined by

$$T_c = \begin{cases} [T_i, T_t], & \text{if } z \ge p,\ T_{learn}, & \text{if } z < p, \end{cases} \qquad z \sim \mathcal{U}(0,1),$$

so image and text are either both present or both replaced by learnable empty-condition tokens. The paper is explicit that this is not per-layer interleaving; all $N=8$ fusion blocks see the same $T_c$ in a given forward pass (Liu et al., 7 Jan 2026).

By contrast, the multimodal dialogue model implements genuine layer-wise interleaving. At encoder layer $l$, the fused feature is

$$\begin{aligned} M_l^c &= \mathbb{I}\!\left(U^l < \frac{p_{\text{net}}}{2}\right) T_l^c + \mathbb{I}\!\left(U^l > 1-\frac{p_{\text{net}}}{2}\right) I_l^c \ &\quad + \frac{1}{2}\mathbb{I}\!\left(\frac{p_{\text{net}}}{2}\le U^l \le 1-\frac{p_{\text{net}}}{2}\right)(T_l^c+I_l^c), \end{aligned}$$

with $U^l \sim \mathrm{Uniform}(0,1)$. That construction makes some layers text-only, some image-conditioned, and some balanced multimodal fusion layers (Sun et al., 2021).

Medical segmentation work broadens the same idea to many-modality settings. One line samples the number of dropped modalities from a truncated geometric distribution and then chooses which channels to zero, producing a stochastic sequence of active modality subsets over training (Lau et al., 2019). Another line conditions a dynamic head on a binary modality code and combines random modality dropping with intra-subject co-training, so the network is trained across all $2^K-1$ non-empty configurations while adapting its first-layer filters to the active subset (Liu et al., 2022).

3. Architectural insertion points

A central design variable is where the dropout acts. The simplest form is input-level masking. “Input Dropout for Spatially Aligned Modalities” concatenates modalities channel-wise and zeroes entire input modalities at training time while always zeroing the auxiliary modality at test time; only the first convolutional layer must be expanded to accept the added channels (Blois et al., 2020). The talking-face model likewise applies modality dropout at the raw input level by setting the audio spectrogram or grayscale face crop to zero for a training block (Abdelaziz et al., 2020).

Other systems place modality dropout at a higher-level fusion interface. MGPC applies it after the frozen RGB and text encoders, but before transformer cross-attention, so point tokens remain untouched and only the condition-token bank is replaced (Liu et al., 7 Jan 2026). The MER2024 system performs pooling first, obtains modality embeddings $e_S,e_I,e_T,e_V$, and then replaces some of those embeddings with zero vectors before concatenation and attention-based fusion (Qi et al., 2024). The pediatric emergency triage model goes even later: XGBoost and Bio_ClinicalBERT each produce 5-way probability vectors, and modality dropout is applied only to those probability vectors before a logistic-regression meta-classifier (Yang et al., 10 Apr 2026).

A distinct insertion point is the fusion block itself. The ABAW audiovisual expression framework zeros the entire visual feature window at the multimodal network input, then relies on “safe cross-attention” so that audio-to-visual attention degenerates to an identity transform when vision is absent; this avoids invalid attention behavior on fully masked visual sequences (Yu et al., 9 Mar 2026). In multi-modal target speaker extraction, modality dropout is applied to the audio-enrolment and video embeddings before attentive combination, yielding a single extraction network that must operate under AV, audio-only, or video-only auxiliary conditions (Korse et al., 9 Jul 2025).

The most explicit missingness-aware design appears in the clinical disease framework with “improved modality dropout.” There, missing modalities are not represented by zeros alone. Instead, the fusion MLP receives learnable modality tokens $E_c$ and $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \ge p,\ T_{learn}, & \text{if } z < p, \end{cases} \qquad z \sim \mathcal{U}(0,1),$$0 in place of missing CT or tabular embeddings, so the model sees $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \ge p,\ T_{learn}, & \text{if } z < p, \end{cases} \qquad z \sim \mathcal{U}(0,1),$$1, $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \ge p,\ T_{learn}, & \text{if } z < p, \end{cases} \qquad z \sim \mathcal{U}(0,1),$$2, and $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \ge p,\ T_{learn}, & \text{if } z < p, \end{cases} \qquad z \sim \mathcal{U}(0,1),$$3 as three distinct semantic states (Gu et al., 22 Sep 2025).

4. Optimization formulations

Most modality-dropout systems keep the task loss unchanged and rely on stochastic masking to create an expectation over missing-modality configurations. In MGPC, no special loss is attached to the dropout mechanism; the model is simply supervised by geometric reconstruction losses while being exposed to both $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \ge p,\ T_{learn}, & \text{if } z < p, \end{cases} \qquad z \sim \mathcal{U}(0,1),$$4 and $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \ge p,\ T_{learn}, & \text{if } z < p, \end{cases} \qquad z \sim \mathcal{U}(0,1),$$5 (Liu et al., 7 Jan 2026). The multimodal dialogue model uses standard sequence negative log-likelihood, with modality dropout only altering the fused encoder representation passed to the decoder (Sun et al., 2021). The ABAW framework uses focal loss, and the pediatric triage system trains only the late-fusion logistic regressor under stochastic modality masking, not the base unimodal models (Yu et al., 9 Mar 2026, Yang et al., 10 Apr 2026).

A more explicit formulation appears in the two-modality clinical fusion framework. Instead of sampling a single subset per example, it supervises all non-empty subsets simultaneously: $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \ge p,\ T_{learn}, & \text{if } z < p, \end{cases} \qquad z \sim \mathcal{U}(0,1),$$6 with $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \ge p,\ T_{learn}, & \text{if } z < p, \end{cases} \qquad z \sim \mathcal{U}(0,1),$$7. This replaces stochastic subset sampling by a deterministic “full + unimodal” objective for every sample (Gu et al., 22 Sep 2025). The same paper then augments unimodal contrastive alignment with fused multimodal representations through

$$T_c = \begin{cases} [T_i, T_t], & \text{if } z \ge p,\ T_{learn}, & \text{if } z < p, \end{cases} \qquad z \sim \mathcal{U}(0,1),$$8

so fused and unimodal embeddings are jointly aligned during pretraining (Gu et al., 22 Sep 2025).

The target-speaker-extraction literature provides a second important variant. Multi-task training evaluates AV, audio-only, and video-only passes explicitly and averages their losses, whereas Modality Dropout Training samples one of those configurations per training sample with probability $$T_c = \begin{cases} [T_i, T_t], & \text{if } z \ge p,\ T_{learn}, & \text{if } z < p, \end{cases} \qquad z \sim \mathcal{U}(0,1),$$9 and uses a single SI-SDR loss. That stochastic formulation was reported to be more robust than standard training and more stable than explicit multi-task averaging across normalization choices (Korse et al., 9 Jul 2025).

5. Empirical operating regimes

Across otherwise unrelated domains, the same empirical pattern recurs: moderate modality dropout improves robustness and often improves generalization, while excessively aggressive masking degrades performance.

Paper	Dropout setting	Reported best setting
MGPC	Probability $N=8$0 of replacing $N=8$1 by $N=8$2	$N=8$3 (Liu et al., 7 Jan 2026)
MER2024 emotion recognition	Per-modality embedding dropout rate $N=8$4	$N=8$5 (Qi et al., 2024)
Multimodal dialogue generation	Layer-wise modality dropout rate $N=8$6	$N=8$7 (Sun et al., 2021)
ABAW expression recognition	Visual-window dropout probability $N=8$8	$N=8$9 (Yu et al., 9 Mar 2026)
Pediatric ED triage	Symmetric modality dropout on stacked probability vectors	$T_c$0–$T_c$1 (Yang et al., 10 Apr 2026)

MGPC reports the clearest ablation of the all-or-none auxiliary mask: $T_c$2 yields the best combination of CD-$T_c$3, CD-$T_c$4, and F1, whereas $T_c$5 effectively reduces the system to a single-modal model with learned dummy tokens and performs substantially worse (Liu et al., 7 Jan 2026). The MER2024 solution shows the same non-monotonic behavior: a modality dropout rate of $T_c$6 achieves the best test WAF, while $T_c$7 degrades performance (Qi et al., 2024). In multimodal dialogue, $T_c$8 gives the best BLEU and NIST scores; very large rates again hurt generation quality (Sun et al., 2021). The pediatric triage study similarly reports that $T_c$9–$l$0 symmetric modality dropout maximizes pediatric Quadratic Weighted Kappa, while $l$1–$l$2 causes clear degradation (Yang et al., 10 Apr 2026).

The robustness benefits are substantial when modalities disappear at inference. In MGPC, a model trained with modality dropout can replace missing image and text by $l$3 at test time without changing architecture, and its “Modalities Missing” performance remains competitive with, or better than, a separately trained single-modal baseline (Liu et al., 7 Jan 2026). In the ABAW framework, simulating visual loss during training improves validation F1 from $l$4 to $l$5 at $l$6, supporting the claim that the network learns to rely on audio when visual cues are absent (Yu et al., 9 Mar 2026). In late-fusion EHR triage, dropout was motivated explicitly as a mitigation for “modality collapse,” where the ensemble over-relies on tabular vitals; the best pediatric zero-shot gains appear precisely in the out-of-distribution demographic setting (Yang et al., 10 Apr 2026).

At the same time, modality dropout can induce its own failure mode: modality bias. In audio-visual speech recognition, heavier video dropout improves robustness to missing video but progressively shifts the latent distribution and outputs toward those of a pure ASR system. The paper formalizes this with the Modality Bias Hypothesis, arguing that dropout can increase the proportion of audio-specific decisive features while reducing genuinely cross-modal information (Dai et al., 2024). Closely related concerns appear in multi-modal target speaker extraction, where standard and multi-task training are described as susceptible to modality dominance, while modality dropout training is reported to mitigate that dominance regardless of normalization choice (Korse et al., 9 Jul 2025).

6. Limits, distinctions, and research directions

The literature makes two distinctions especially clear. First, interleaving modalities in sequence format is not the same as interleaving modality dropout. Orthus uses a single autoregressive sequence with [BOI], image patches, [EOI], and text spans, but explicitly does not implement modality dropout or random modality masking (Kou et al., 2024). Second, robustness to missing modalities is not guaranteed merely by seeing corrupted inputs; the AVSR work shows that naive dropout can over-regularize the weaker modality and push the model toward unimodal behavior unless additional mechanisms preserve multimodal structure (Dai et al., 2024).

Several concrete extensions are already suggested inside the cited work. MGPC explicitly notes that its per-sample Bernoulli mechanism could be generalized to per-block interleaving, per-modality masking instead of all-or-nothing image+text replacement, stage-wise interleaving with the progressive decoder, or a curriculum over epochs (Liu et al., 7 Jan 2026). The multimodal dialogue model already realizes one such extension through depth-wise interleaving, and the clinical disease framework realizes another by replacing zero placeholders with learnable missing-modality tokens and supervising all non-empty subsets simultaneously (Sun et al., 2021, Gu et al., 22 Sep 2025). Input Dropout, finally, points to a straightforward scaling path for spatially aligned modalities: extend the input-channel mask from one auxiliary modality to many, although the paper states that it only experimented with adding a single additional modality to RGB (Blois et al., 2020).

A plausible synthesis is that “interleaved modality dropout” is less a single algorithm than a design family organized by three choices: the masking granularity, the architectural insertion point, and the optimization schedule. The surveyed papers cover sample-level Bernoulli masking, independent per-modality zeroing, layer-wise alternation, whole-window masking, exhaustive subset supervision, and missingness-aware learnable tokens. Taken together, they show that the central problem is not merely how to hide modalities, but how to do so without collapsing fusion into a disguised unimodal model.