Mean Mode Screaming (MMS) in Ultra-Deep DiTs

• MMS is an abrupt trigger event in ultra-deep DiTs that forces the network into a silent, mean-dominated collapse state with nearly identical token representations.
• It arises via an exact gradient decomposition into mean-coherent and centered terms, where coherent accumulation accentuates mean-mode influences while suppressing token-specific diversity.
• The instability is mitigated by MV-Split Residuals, which selectively damp mean-mode updates while preserving centered learning to stabilize training at extreme model depths.

Mean Mode Screaming (MMS) is the name given to an abrupt trigger event in ultra-deep Diffusion Transformers (DiTs), especially deep Post-Norm residual chains, that drives the network into a silent, mean-dominated collapse regime in token space (Lu, 7 May 2026). In that regime, token representations become nearly identical, centered token variation is suppressed, and the model effectively predicts from a trivial shared token mean rather than from token-specific structure. The phenomenon is presented not as a generic exploding-gradient failure, but as a subspace-selective instability in which residual writers undergo a mean-coherent backward shock, deep residual branches open rapidly, and the attention pathway subsequently loses the ability to restore diversity because of structural suppression in the Softmax Jacobian (Lu, 7 May 2026).

1. Definition and collapse regime

The paper distinguishes between the collapse state itself and the event that triggers entry into it. The collapse state is a silent, mean-dominated regime with token homogenization. MMS is the abrupt trigger event that pushes the network into that state. This distinction is central because the underlying training run can appear stable for thousands of steps before a few updates precipitate collapse; in zero-writer runs, the loss can then jump back near initialization level and fail to recover (Lu, 7 May 2026).

The organizing decomposition is

$X = JX + PX \equiv \mu(X) + c(X),$

for a sequence $X\in\mathbb{R}^{T\times D}$, with

$$J \triangleq \frac{1}{T}\mathbf{1}\mathbf{1}^\top,\qquad P \triangleq I-J.$$

Here $\mu(X)=JX$ is the token-wise mean component and $c(X)=PX$ is the centered variation across tokens. Collapse corresponds to dominance of $\mu(X)$ together with shrinkage of $c(X)$. The paper tracks this through the energy ratio

$\rho_T = \frac{\|\mu(X)\|_F}{\|c(X)\|_F+\epsilon}.$

Two visible symptoms are emphasized. First, token homogenization is measured by token cosine similarity approaching $1$. Second, centered variation is suppressed, so the network stops maintaining token-specific structure. This suggests that the salient pathology is not simply magnitude blow-up in the residual stream, but a redistribution of representational energy toward the invariant token-mean subspace.

The paper locates this phenomenon in very deep DiTs, especially Post-Norm residual chains. In the main diagnostic regime, attention output projection $W_O$ and FFN output projection $X\in\mathbb{R}^{T\times D}$0 are zero-initialized residual writers, so early training begins with nearly closed residual branches. MMS is then identified with the point at which writer gradients become dominated by a mean-coherent mode, opening those branches rapidly and injecting predominantly mean-mode content into the trunk (Lu, 7 May 2026).

2. Gradient decomposition and the trigger mechanism

The mechanistic core of MMS is an exact decomposition of residual-writer gradients into mean-coherent and centered terms. For a token-wise linear map $X\in\mathbb{R}^{T\times D}$1,

$X\in\mathbb{R}^{T\times D}$2

where $X\in\mathbb{R}^{T\times D}$3 is the input for token $X\in\mathbb{R}^{T\times D}$4 and $X\in\mathbb{R}^{T\times D}$5 is the backward adjoint. Writing

$X\in\mathbb{R}^{T\times D}$6

and

$X\in\mathbb{R}^{T\times D}$7

one obtains the exact identity

$X\in\mathbb{R}^{T\times D}$8

The paper defines

$X\in\mathbb{R}^{T\times D}$9

The mean-coherent term is rank-1, with norm

$$J \triangleq \frac{1}{T}\mathbf{1}\mathbf{1}^\top,\qquad P \triangleq I-J.$$0

and therefore scales coherently like $$J \triangleq \frac{1}{T}\mathbf{1}\mathbf{1}^\top,\qquad P \triangleq I-J.$$1 when token means cease to cancel. By contrast, the centered term is built from token deviations and tends to cancel diffusively when token-level signs vary. The paper identifies MMS precisely with the transition from that cancellation regime to coherent mean-mode accumulation (Lu, 7 May 2026).

This transition is quantified by the alignment-amplification law

$$J \triangleq \frac{1}{T}\mathbf{1}\mathbf{1}^\top,\qquad P \triangleq I-J.$$2

with

$$J \triangleq \frac{1}{T}\mathbf{1}\mathbf{1}^\top,\qquad P \triangleq I-J.$$3

If activations and adjoints are heterogeneous across tokens, $$J \triangleq \frac{1}{T}\mathbf{1}\mathbf{1}^\top,\qquad P \triangleq I-J.$$4, and gradient growth is roughly diffusive. If they align across tokens, $$J \triangleq \frac{1}{T}\mathbf{1}\mathbf{1}^\top,\qquad P \triangleq I-J.$$5, and accumulation approaches the coherent $$J \triangleq \frac{1}{T}\mathbf{1}\mathbf{1}^\top,\qquad P \triangleq I-J.$$6 regime. In the paper’s interpretation, MMS is the moment at which this alignment becomes sufficiently strong that writer updates cease to average out and instead reinforce the mean mode.

The residual writers are the structurally sensitive locus because they are the direct interface back into the trunk. In the studied DiT, the relevant writers are $$J \triangleq \frac{1}{T}\mathbf{1}\mathbf{1}^\top,\qquad P \triangleq I-J.$$7 in attention and $$J \triangleq \frac{1}{T}\mathbf{1}\mathbf{1}^\top,\qquad P \triangleq I-J.$$8 in the FFN, where

$$J \triangleq \frac{1}{T}\mathbf{1}\mathbf{1}^\top,\qquad P \triangleq I-J.$$9

A mean-coherent gradient spike on these parameters rapidly opens deep residual branches, after which the forward states become increasingly mean-dominated (Lu, 7 May 2026).

3. Attention geometry and Softmax-null-space lock-in

The paper’s second major claim is that the attention pathway cannot readily recover once values homogenize, because attention-logit gradients are structurally suppressed by the null space of the Softmax Jacobian. This is presented as the mechanism that makes the post-MMS state self-reinforcing rather than transient (Lu, 7 May 2026).

For a single attention row $\mu(X)=JX$0, if all values are equal,

$\mu(X)=JX$1

then

$\mu(X)=JX$2

The derivative with respect to attention weights is then independent of $\mu(X)=JX$3: $\mu(X)=JX$4 The Softmax Jacobian is

$\mu(X)=JX$5

with null-space property

$\mu(X)=JX$6

Therefore

$\mu(X)=JX$7

The exact zero requires perfectly constant values, but approximate homogenization still removes the constant component and strongly suppresses Q/K learning. The paper therefore describes a locked configuration: writer gradients remain nonzero because they bypass the Softmax Jacobian,

$\mu(X)=JX$8

yet Q/K gradients collapse, so attention can no longer restore token-specific routing. This produces the characteristic sequence emphasized in the paper: mean-coherent gradient spike, rapid branch opening, forward mean domination, token homogenization, and then Q/K gradient suppression (Lu, 7 May 2026).

A related structural asymmetry is that row-stochastic attention preserves the pure token mean. For row-stochastic $\mu(X)=JX$9, $c(X)=PX$0, so

$c(X)=PX$1

By contrast, the centered component evolves through

$c(X)=PX$2

and

$c(X)=PX$3

The paper defines

$c(X)=PX$4

When $c(X)=PX$5, attention is contractive on the centered subspace. This implies an intrinsic imbalance: mean is preserved, centered structure may decay, and the network relies on residual branches to replenish token-varying information. If those residual updates become mean-dominated, the drift toward a pure-mean state becomes structurally favored.

4. MV-Split Residuals

To target the unstable mode directly, the paper proposes Mean-Variance Split (MV-Split) Residuals as a replacement for standard Post-Norm residual addition (Lu, 7 May 2026). The standard merge is

$c(X)=PX$6

where $c(X)=PX$7. MV-Split instead uses

$c(X)=PX$8

followed by

$c(X)=PX$9

The forward decomposition is

$\mu(X)$0

$\mu(X)$1

Thus the centered update is separately gained, while the mean is not carried purely residually; instead it is leakily replaced. The paper describes this as a separately gained centered residual update plus a leaky trunk-mean replacement. For $\mu(X)$2, each mean feature acts as a leaky integrator.

Backward propagation splits analogously. If

$\mu(X)$3

then

$\mu(X)$4

Small $\mu(X)$5 damps the unstable mean-coherent mode, while $\mu(X)$6 can remain larger so that centered feature learning is not equally suppressed. This is the paper’s principal contrast with LayerScale and ReZero, both of which gate the residual branch isotropically in token space and therefore shrink mean and centered components together.

Three differences are emphasized. First, MV-Split contracts the carried trunk mean, whereas LayerScale leaves the trunk mean unchanged: $\mu(X)$7 Second, MV-Split is anisotropic in token subspace: $\mu(X)$8 Third, it allows the centered path to retain a larger gain while the mean path remains damped. The paper’s claim is therefore not that deep DiTs require less residual magnitude in general, but that they require selective damping of coherent mean-mode writing (Lu, 7 May 2026).

In the multimodal implementation, image and text means are handled separately through

$\mu(X)$9

Initialization values reported in the paper are $c(X)$0, $c(X)$1 for the 400-layer MV-Split run, and $c(X)$2, $c(X)$3 for the 1000-layer MV-Split run. The gains are unconstrained learnable vectors, though empirically $c(X)$4 remains in $c(X)$5. The implementation also uses non-affine RMSNorm and QK-Norm (Lu, 7 May 2026).

5. Empirical evidence in 400-layer and 1000-layer DiTs

The main controlled study uses a 400-layer single-stream multimodal DiT with Post-Norm residual chain, no AdaLN or extra modulation pathways, image and text tokens concatenated, $c(X)$6, and 5.45B parameters. Training uses ImageNet-2012 latents from a frozen FLUX.2 VAE, a frozen Qwen3-0.6B text encoder, AdamW, batch size 1024, and gradient clipping 1.0 (Lu, 7 May 2026).

In this setting, the unstabilized baseline trains stably for a period and then abruptly diverges. At the divergence event, the global gradient norm spikes; the spike is concentrated in $c(X)$7 rather than $c(X)$8; Q/K gradients drop by roughly four orders of magnitude; residual branch opening accelerates; $c(X)$9 rises sharply; and deep token cosine similarity approaches $\rho_T = \frac{\|\mu(X)\|_F}{\|c(X)\|_F+\epsilon}.$0. The paper treats this joint signature as the full empirical manifestation of MMS.

At a representative spike step $\rho_T = \frac{\|\mu(X)\|_F}{\|c(X)\|_F+\epsilon}.$1, the alignment law is measured on image tokens with $\rho_T = \frac{\|\mu(X)\|_F}{\|c(X)\|_F+\epsilon}.$2 for both $\rho_T = \frac{\|\mu(X)\|_F}{\|c(X)\|_F+\epsilon}.$3 and $\rho_T = \frac{\|\mu(X)\|_F}{\|c(X)\|_F+\epsilon}.$4. Active layers lie close to the coherence saturation envelope, and the largest active layers reach

$\rho_T = \frac{\|\mu(X)\|_F}{\|c(X)\|_F+\epsilon}.$5

corresponding to about a 13× writer-gradient norm amplification relative to the independent-token baseline. This is presented as direct support for the claim that MMS coincides with disappearance of signed cancellation across tokens (Lu, 7 May 2026).

The comparison among stable alternatives is between the baseline, LayerScale, and MV-Split. The baseline diverges; lowering the learning rate only delays failure. LayerScale remains stable, but suppresses both mean and centered updates. MV-Split prevents collapse while retaining much of the baseline’s early learning trajectory. At 50k steps in the matched 400-layer comparison, the reported FID/IS values are

• LayerScale: $\rho_T = \frac{\|\mu(X)\|_F}{\|c(X)\|_F+\epsilon}.$6,
• MV-Split: $\rho_T = \frac{\|\mu(X)\|_F}{\|c(X)\|_F+\epsilon}.$7.

The paper further notes that MV-Split is already substantially ahead of LayerScale by 20k–30k steps and trains in a higher but bounded gradient band, which is offered as evidence that it is not equivalent to stronger global shrinkage. A writer-gradient decomposition ablation reinforces this point: LayerScale bounds $\rho_T = \frac{\|\mu(X)\|_F}{\|c(X)\|_F+\epsilon}.$8 but also compresses $\rho_T = \frac{\|\mu(X)\|_F}{\|c(X)\|_F+\epsilon}.$9, whereas MV-Split bounds $1$0 while keeping $1$1 in a higher stable band (Lu, 7 May 2026).

A separate 1000-layer MV-Split DiT is used as a scale-validation run. It has 13.64B parameters, uses the same residual design, remains stable over the full 100k-step horizon, and reports:

• 20k: $1$2,
• 30k: $1$3,
• 40k: $1$4,
• 50k: $1$5.

The paper is explicit that this is not a controlled superiority claim over a 1000-layer baseline, but a boundary-scale validation that the architecture remains trainable at extreme depth.

An important negative control concerns partial protection. Applying MV-Split only to the attention output branch in a 1000-layer model still fails, with the spike relocating to the unprotected FFN writer $1$6, which takes about 93% of top-$1$7 squared gradient mass at failure. This indicates that the failure pathway is not attention-exclusive and that both residual writer families need the split (Lu, 7 May 2026).

6. Diagnostics, interpretation, and limitations

The paper develops a diagnostic stack intended to expose MMS before ordinary scalar training monitors do. The most direct detector is the writer-gradient decomposition

$1$8

with magnitudes

$1$9

Additional diagnostics are the mean/centered energy ratio

$W_O$0

token cosine similarity

$W_O$1

attention contraction on the centered subspace

$W_O$2

the residual branch update ratio

$W_O$3

and centered replenishment metrics such as VarGain,

$W_O$4

and centered retention,

$W_O$5

Strong drops in $W_O$6 and $W_O$7 are treated as evidence of Softmax-null-space lock-in (Lu, 7 May 2026).

The broader significance proposed by the paper is that very deep transformer failures may need to be analyzed in subspace-selective rather than purely norm-based terms. This suggests that stable training at extreme depth depends on controlling the relative dynamics of mean and centered token components, not merely on damping residual branches globally.

The paper is also careful about scope. It does not claim that MMS explains every deep-transformer instability. The identified pathway is specific to ultra-deep Post-Norm Softmax-attention DiTs. The alignment law explains when coherent accumulation occurs but does not predict the exact future onset time $W_O$8. The Softmax-null-space argument depends on row-stochastic attention and does not directly transfer to non-attention sequence mixers. At the same time, the writer-gradient decomposition is more general, which the paper takes as a reason to regard analogous mean-dominated failures outside standard attention as plausible but unestablished (Lu, 7 May 2026).

Finally, the token mean is not treated as intrinsically pathological. In the studied noise-agnostic DiT, the token mean acts as an implicit timestep carrier with near-perfect linear decodability of diffusion time $W_O$9. This is why MV-Split gain-limits the mean rather than projecting it out. A plausible implication is that the paper’s intervention is best understood not as mean suppression per se, but as regulation of a mode that is both useful and structurally capable of destabilizing ultra-deep residual dynamics.