Compare attention-FFN sub-layer dropout configurations

Determine why sub-layer dropout underperforms whole-layer dropout in transformer language models and compare additional configurations that apply dropout only to attention blocks or only to feed-forward network blocks.

Background

The paper compares two structural granularities for transformer dropout: independently dropping the attention and feed-forward sub-blocks within each transformer layer, versus dropping the complete transformer layer with a shared mask. Whole-layer dropout produces better accuracy in the reported experiments, but the mechanism responsible for the difference is not established.

The authors also identify attention-only and FFN-only dropout as untested configurations. These questions are unresolved because the empirical comparison establishes which configuration performs better without explaining the observed performance gap or determining whether one sub-block is principally responsible.

References

We leave investigating the reason sub-layer dropout underperforms layer dropout, and also leave investigating other configurations such as applying dropout only on attention or only on FFN, for future work.

— Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference  (2609.05275 - Elhoushi et al., 4 Sep 2026) in Section 4, Subsection “Model Granularity”

Because fault-hardened models show signs of learning to operate within good codes, we conjecture that they may be formally fault-tolerant, retaining finite capacity in the limit of infinite $N$,

— Fault-tolerant foundation models  (2610.10311 - McCourt et al., 7 Oct 2026) in Main text, paragraph introducing Eq. (6), and subsection “Towards formally fault-tolerant AI”