---
title: Interleaved Modality Dropout
url: https://www.emergentmind.com/topics/interleaved-modality-dropout
type: topic
---

# Interleaved Modality Dropout

Searching arXiv for recent papers on modality dropout and interleaved multimodal training.
Interleaved modality dropout, as an *Editor’s term*, denotes multimodal training procedures in which modality availability is varied across optimization steps, samples, layers, windows, or stages, so that a model repeatedly encounters both full and incomplete modality sets. In the literature assembled around this idea, the operative mechanisms range from whole-modality zeroing at the input or fusion interface, to layer-wise switching between modality-specific streams, to explicit supervision over all non-empty modality subsets. The term itself is not uniformly standardized: for example, MGPC describes “modality dropout” without using “interleaved,” while Orthus studies interleaved image–text sequences but does not implement modality dropout [2601.03660], [2412.00127].

## 1. Terminology and conceptual scope

Within the cited literature, “modality dropout” consistently refers to dropping entire modality streams rather than individual neurons or features. MGPC defines a per-sample Bernoulli choice between real auxiliary condition tokens and a learnable placeholder token set \(T_{learn}\), so the model alternates between a multimodal regime and a point-only regime during training [2601.03660]. In audiovisual talking-face generation, batches contain audio-only, video-only, and audiovisual examples, again by dropping whole input streams rather than hidden units [2005.13616]. In medical segmentation, ModDrop and related schemes randomly remove whole MRI channels so that a single model is exposed to many missing-modality configurations during training [1908.06683], [2203.04959].

This suggests that “interleaving” is best understood operationally rather than terminologically. In some systems, interleaving occurs across training iterations because different samples encounter different modality subsets. In other systems, interleaving occurs across network depth: the non-hierarchical multimodal dialogue model switches, at each encoder layer, among a text-only stream, a text-enhanced visual stream, or their average [2110.09702]. Orthus is relevant chiefly as a contrast case: it interleaves image and text within a single autoregressive sequence, but the paper explicitly states that it does not use “modality dropout,” “modality masking,” or any equivalent random removal scheme [2412.00127].

## 2. Canonical masking patterns

The literature contains several distinct masking patterns that fall under this broader notion.

| Pattern | Mechanism | Representative papers |
|---|---|---|
| Sample-level all-or-none auxiliary masking | Replace the full auxiliary condition with learnable placeholders for a sample | MGPC [2601.03660] |
| Independent per-modality masking | Zero each modality embedding independently before fusion | MER2024 emotion recognition [2409.07078] |
| Layer-wise interleaving | Choose text-only, image-enhanced, or averaged features per encoder layer | Multimodal dialogue generation [2110.09702] |
| Window-level sequence masking | Zero an entire visual window while keeping audio active | ABAW expression recognition [2603.08034] |
| Input-channel masking | Zero full input modalities treated as extra channels | Spatially aligned vision tasks [2002.02852] |
| Exhaustive subset supervision | Optimize full and unimodal subsets simultaneously for each sample | Clinical fusion with learnable tokens [2509.18284] |

MGPC provides the clearest example of structured sample-level masking. Its condition tokens are defined by
\[
T_c =
\begin{cases}
[T_i, T_t], & \text{if } z \ge p,\\
T_{learn}, & \text{if } z < p,
\end{cases}
\qquad z \sim \mathcal{U}(0,1),
\]
so image and text are either both present or both replaced by learnable empty-condition tokens. The paper is explicit that this is not per-layer interleaving; all \(N=8\) fusion blocks see the same \(T_c\) in a given forward pass [2601.03660].

By contrast, the multimodal dialogue model implements genuine layer-wise interleaving. At encoder layer \(l\), the fused feature is
\[
\begin{aligned}
M_l^c &= \mathbb{I}\!\left(U^l < \frac{p_{\text{net}}}{2}\right) T_l^c
+ \mathbb{I}\!\left(U^l > 1-\frac{p_{\text{net}}}{2}\right) I_l^c \\
&\quad + \frac{1}{2}\mathbb{I}\!\left(\frac{p_{\text{net}}}{2}\le U^l \le 1-\frac{p_{\text{net}}}{2}\right)(T_l^c+I_l^c),
\end{aligned}
\]
with \(U^l \sim \mathrm{Uniform}(0,1)\). That construction makes some layers text-only, some image-conditioned, and some balanced multimodal fusion layers [2110.09702].

Medical segmentation work broadens the same idea to many-modality settings. One line samples the number of dropped modalities from a truncated geometric distribution and then chooses which channels to zero, producing a stochastic sequence of active modality subsets over training [1908.06683]. Another line conditions a dynamic head on a binary modality code and combines random modality dropping with intra-subject co-training, so the network is trained across all \(2^K-1\) non-empty configurations while adapting its first-layer filters to the active subset [2203.04959].

## 3. Architectural insertion points

A central design variable is where the dropout acts. The simplest form is input-level masking. “Input Dropout for Spatially Aligned Modalities” concatenates modalities channel-wise and zeroes entire input modalities at training time while always zeroing the auxiliary modality at test time; only the first convolutional layer must be expanded to accept the added channels [2002.02852]. The talking-face model likewise applies modality dropout at the raw input level by setting the audio spectrogram or grayscale face crop to zero for a training block [2005.13616].

Other systems place modality dropout at a higher-level fusion interface. MGPC applies it after the frozen RGB and text encoders, but before transformer cross-attention, so point tokens remain untouched and only the condition-token bank is replaced [2601.03660]. The MER2024 system performs pooling first, obtains modality embeddings \(e_S,e_I,e_T,e_V\), and then replaces some of those embeddings with zero vectors before concatenation and attention-based fusion [2409.07078]. The pediatric emergency triage model goes even later: XGBoost and Bio_ClinicalBERT each produce 5-way probability vectors, and modality dropout is applied only to those probability vectors before a logistic-regression meta-classifier [2604.09905].

A distinct insertion point is the fusion block itself. The ABAW audiovisual expression framework zeros the entire visual feature window at the multimodal network input, then relies on “safe cross-attention” so that audio-to-visual attention degenerates to an identity transform when vision is absent; this avoids invalid attention behavior on fully masked visual sequences [2603.08034]. In multi-modal target speaker extraction, modality dropout is applied to the audio-enrolment and video embeddings before attentive combination, yielding a single extraction network that must operate under AV, audio-only, or video-only auxiliary conditions [2507.06566].

The most explicit missingness-aware design appears in the clinical disease framework with “improved modality dropout.” There, missing modalities are not represented by zeros alone. Instead, the fusion MLP receives learnable modality tokens \(E_c\) and \(E_t\) in place of missing CT or tabular embeddings, so the model sees \((h_c,h_t)\), \((h_c,E_t)\), and \((E_c,h_t)\) as three distinct semantic states [2509.18284].

## 4. Optimization formulations

Most modality-dropout systems keep the task loss unchanged and rely on stochastic masking to create an expectation over missing-modality configurations. In MGPC, no special loss is attached to the dropout mechanism; the model is simply supervised by geometric reconstruction losses while being exposed to both \([T_i,T_t]\) and \(T_{learn}\) [2601.03660]. The multimodal dialogue model uses standard sequence negative log-likelihood, with modality dropout only altering the fused encoder representation passed to the decoder [2110.09702]. The ABAW framework uses focal loss, and the pediatric triage system trains only the late-fusion logistic regressor under stochastic modality masking, not the base unimodal models [2603.08034], [2604.09905].

A more explicit formulation appears in the two-modality clinical fusion framework. Instead of sampling a single subset per example, it supervises all non-empty subsets simultaneously:
\[
\mathcal{L}^{smd}
=
-\log p(y^i\mid x_c^i,x_t^i,\theta)
-\lambda \sum_{j\in M}\log p(y^i\mid x_j^i,\theta),
\]
with \(M=\{c,t\}\). This replaces stochastic subset sampling by a deterministic “full + unimodal” objective for every sample [2509.18284]. The same paper then augments unimodal contrastive alignment with fused multimodal representations through
\[
\hat{\mathcal{L}}^{con}
=
\mathcal{L}^{con}_{c,t}
+
\mathcal{L}^{con}_{c,f}
+
\mathcal{L}^{con}_{t,f},
\]
so fused and unimodal embeddings are jointly aligned during pretraining [2509.18284].

The target-speaker-extraction literature provides a second important variant. Multi-task training evaluates AV, audio-only, and video-only passes explicitly and averages their losses, whereas Modality Dropout Training samples one of those configurations per training sample with probability \(1/3\) and uses a single SI-SDR loss. That stochastic formulation was reported to be more robust than standard training and more stable than explicit multi-task averaging across normalization choices [2507.06566].

## 5. Empirical operating regimes

Across otherwise unrelated domains, the same empirical pattern recurs: moderate modality dropout improves robustness and often improves generalization, while excessively aggressive masking degrades performance.

| Paper | Dropout setting | Reported best setting |
|---|---|---|
| MGPC | Probability \(p\) of replacing \([T_i,T_t]\) by \(T_{learn}\) | \(p=0.5\) [2601.03660] |
| MER2024 emotion recognition | Per-modality embedding dropout rate \(p_1\) | \(p_1=0.3\) [2409.07078] |
| Multimodal dialogue generation | Layer-wise modality dropout rate \(p_{\text{net}}\) | \(p_{\text{net}}=0.4\) [2110.09702] |
| ABAW expression recognition | Visual-window dropout probability \(p\) | \(p=0.10\) [2603.08034] |
| Pediatric ED triage | Symmetric modality dropout on stacked probability vectors | \(30\%\)–\(40\%\) [2604.09905] |

MGPC reports the clearest ablation of the all-or-none auxiliary mask: \(p=0.5\) yields the best combination of CD-\(\ell_2\), CD-\(\ell_1\), and F1, whereas \(p=1.0\) effectively reduces the system to a single-modal model with learned dummy tokens and performs substantially worse [2601.03660]. The MER2024 solution shows the same non-monotonic behavior: a modality dropout rate of \(0.3\) achieves the best test WAF, while \(0.5\) degrades performance [2409.07078]. In multimodal dialogue, \(p_{\text{net}}=0.4\) gives the best BLEU and NIST scores; very large rates again hurt generation quality [2110.09702]. The pediatric triage study similarly reports that \(30\%\)–\(40\%\) symmetric modality dropout maximizes pediatric Quadratic Weighted Kappa, while \(50\%\)–\(60\%\) causes clear degradation [2604.09905].

The robustness benefits are substantial when modalities disappear at inference. In MGPC, a model trained with modality dropout can replace missing image and text by \(T_{learn}\) at test time without changing architecture, and its “Modalities Missing” performance remains competitive with, or better than, a separately trained single-modal baseline [2601.03660]. In the ABAW framework, simulating visual loss during training improves validation F1 from \(0.4764\) to \(0.5029\) at \(p=0.10\), supporting the claim that the network learns to rely on audio when visual cues are absent [2603.08034]. In late-fusion EHR triage, dropout was motivated explicitly as a mitigation for “modality collapse,” where the ensemble over-relies on tabular vitals; the best pediatric zero-shot gains appear precisely in the out-of-distribution demographic setting [2604.09905].

At the same time, modality dropout can induce its own failure mode: modality bias. In audio-visual speech recognition, heavier video dropout improves robustness to missing video but progressively shifts the latent distribution and outputs toward those of a pure ASR system. The paper formalizes this with the Modality Bias Hypothesis, arguing that dropout can increase the proportion of audio-specific decisive features while reducing genuinely cross-modal information [2403.04245]. Closely related concerns appear in multi-modal target speaker extraction, where standard and multi-task training are described as susceptible to modality dominance, while modality dropout training is reported to mitigate that dominance regardless of normalization choice [2507.06566].

## 6. Limits, distinctions, and research directions

The literature makes two distinctions especially clear. First, interleaving modalities in sequence format is not the same as interleaving modality dropout. Orthus uses a single autoregressive sequence with `[BOI]`, image patches, `[EOI]`, and text spans, but explicitly does not implement modality dropout or random modality masking [2412.00127]. Second, robustness to missing modalities is not guaranteed merely by seeing corrupted inputs; the AVSR work shows that naive dropout can over-regularize the weaker modality and push the model toward unimodal behavior unless additional mechanisms preserve multimodal structure [2403.04245].

Several concrete extensions are already suggested inside the cited work. MGPC explicitly notes that its per-sample Bernoulli mechanism could be generalized to per-block interleaving, per-modality masking instead of all-or-nothing image+text replacement, stage-wise interleaving with the progressive decoder, or a curriculum over epochs [2601.03660]. The multimodal dialogue model already realizes one such extension through depth-wise interleaving, and the clinical disease framework realizes another by replacing zero placeholders with learnable missing-modality tokens and supervising all non-empty subsets simultaneously [2110.09702], [2509.18284]. Input Dropout, finally, points to a straightforward scaling path for spatially aligned modalities: extend the input-channel mask from one auxiliary modality to many, although the paper states that it only experimented with adding a single additional modality to RGB [2002.02852].

A plausible synthesis is that “interleaved modality dropout” is less a single algorithm than a design family organized by three choices: the masking granularity, the architectural insertion point, and the optimization schedule. The surveyed papers cover sample-level Bernoulli masking, independent per-modality zeroing, layer-wise alternation, whole-window masking, exhaustive subset supervision, and missingness-aware learnable tokens. Taken together, they show that the central problem is not merely how to hide modalities, but how to do so without collapsing fusion into a disguised unimodal model.

Source: https://www.emergentmind.com/topics/interleaved-modality-dropout