Evaluate finer-grained and distributed layer-dropout schemes

Investigate distributed-training and gradient-accumulation schemes that combine the weight-loading advantages of per-batch layer dropout with the accuracy advantages of per-sequence dropout, and compare them with per-token and per-neuron dropout.

Background

The paper finds that sampling layer-dropout masks independently for each sequence yields lower loss than using one mask for the entire batch, while per-batch dropout can avoid loading a layer’s weights during a training step. The authors suggest that distributed training or gradient accumulation might combine these benefits by assigning different dropped batches across devices or accumulation steps.

Whether these approaches provide the proposed computational and accuracy advantages is unresolved, as is whether even finer-grained sparsity at the token or neuron level can produce comparable benefits.

References

We leave exploring such approaches, as well as comparing with even finer-grain dropout such as per-token or per-neuron, for future work.

— Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference  (2609.05275 - Elhoushi et al., 4 Sep 2026) in Section 4, Subsection “Tensor Granularity”

We leave other schedules such as applying dropout to mid-training, SFT, or continual pre-training for future work.

— Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference  (2609.05275 - Elhoushi et al., 4 Sep 2026) in Section 5, Subsection “Dropout Schedule”

Whether other granularities, such as attention-head or neuron-level dropout, can induce similar structural resilience remains an open question.

— Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference  (2609.05275 - Elhoushi et al., 4 Sep 2026) in Section 10, “Limitations”