Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers

Abstract: Scaling Diffusion Transformers (DiTs) to hundreds of layers introduces a structural vulnerability: networks can enter a silent, mean-dominated collapse state that homogenizes token representations and suppresses centered variation. Through mechanistic auditing, we isolate the trigger event of this collapse as Mean Mode Screaming (MMS). MMS can occur even when training appears stable, with a mean-coherent backward shock on residual writers that opens deep residual branches and drives the network into a mean-dominated state. We show this behavior is driven by an exact decomposition of these gradients into mean-coherent and centered components, compounded by the structural suppression of attention-logit gradients through the null space of the Softmax Jacobian once values homogenize. To address this, we propose Mean-Variance Split (MV-Split) Residuals, which combine a separately gained centered residual update with a leaky trunk-mean replacement. On a 400-layer single-stream DiT, MV-Split prevents the divergent collapse that crashes the un-stabilized baseline; it tracks close to the baseline's pre-crash trajectory while remaining substantially better than token-isotropic gating methods such as LayerScale across the full schedule. Finally, we present a 1000-layer DiT as a scale-validation run at boundary scales, establishing that the architecture remains stably trainable at extreme depth.

• The paper presents MV-Split Residuals to counter Mean Mode Screaming, enabling stable training of ultra-deep diffusion transformers.
• It decomposes residual gradients into mean and centered components to precisely control training dynamics, even beyond 400 layers.
• Experimental results on ImageNet show improved FID and Inception Scores, validating the approach for high-quality generative performance.

Mean Mode Screaming and MV-Split Residuals in Ultra-Deep Diffusion Transformers

Problem Statement: Depth Failure Modes in Diffusion Transformers

The empirical scaling laws for generative models indicate that Transformer depth is a primary lever for improving generative performance, yet scaling Diffusion Transformers (DiTs) to ultra-deep regimes ($\gg 400$ layers) introduces new structural vulnerabilities. Standard Post-Norm residual architectures suffer from abrupt failures characterized by a sharp transition where token representations become homogenized, suppressing spatially centered variation and entering a mean-dominated regime. This failure, termed Mean Mode Screaming (MMS), is not directly attributable to exploding or vanishing gradients nor to forward saturation or residual norm divergence. The hallmark of MMS is a backward event—a coherent spike in the mean component of the writer gradient—followed by residual branch opening and logit-gradient extinction in the attention pathway.

Figure 1: Architecture schematic and training diagnostics; MV-Split design stabilizes depth and prevents mean-mode collapse compared to the un-stabilized baseline.

Mechanistic Analysis: Geometric Vulnerabilities and Gradient Dynamics

The root cause of MMS resides in the geometric asymmetry induced by row-stochastic attention in deep Transformers. For any input $X$, tokens are decomposed into sequence mean ($\mu(X)$) and centered variation ($c(X)$). Row-stochastic attention strictly preserves pure-mean states ($A\,\mu(X) = \mu(X)$) while potentially contracting the centered subspace ($c(AX) = PAPX$, $\|PAP\|_2 < 1$). As depth increases, attention fails to replenish the centered subspace, and residual updates become increasingly mean-dominated.

Empirically, MMS manifests as a gradient spike concentrated in the mean-coherent component at the residual writer interface, while Q/K gradients collapse, locking the network into a trivial prediction baseline. This spike is mechanistically explainable via an exact additive decomposition of the writer gradient into a mean-coherent rank-1 component and a diffusive centered component:

$$\nabla_W \mathcal{L} = T\,\bar{\delta}\bar{y}^\top + \sum_{t=1}^T \tilde{\delta}_t \tilde{y}_t^\top$$

When representations become aligned, the mean component amplifies ($\mathcal{O}(T)$ scaling), triggering MMS.

Figure 2: Empirical trajectory of divergence—gradient spike, Q/K collapse, and token homogenization coinciding with MMS.

Additionally, once tokens collapse to the mean direction, the attention Softmax Jacobian suppresses logit gradients through its null space, extinguishing Q/K learning and preventing recovery, as proven via chain rule analysis.

MV-Split Residuals: Anisotropic Stabilizers for Deep Transformers

Token-isotropic residual gain methods like LayerScale and ReZero damp both mean and centered paths, slowing convergence and failing to selectively suppress the dangerous mean-coherent mode. MV-Split Residuals are introduced to overcome this limitation: the residual branch output is projected into mean and centered subspaces, each updated with independent gains ($\alpha$ for mean, $X$0 for centered), and merged with the trunk via leaky mean replacement and centered addition:

$X$1

This intervention contracts the carried trunk mean while allowing the centered feature-learning path to remain efficiently trainable, structurally avoiding MMS and unlocking convergence at extreme depth.

Experimental Protocols and Numerical Outcomes

A controlled comparative evaluation is executed on ImageNet-2012 latents, contrasting MV-Split to LayerScale and un-stabilized baselines. MV-Split eliminates collapse events in 400-layer configurations, maintains higher bounded post-clipping gradient norms, and exhibits superior FID-50K and Inception Score (IS) relative to LayerScale. Early learning speed in MV-Split rivals that of the unstable baselines and persists past 20k–50k steps, with the advantage maintained throughout training.

A scale-validation run at 1000 layers demonstrates stable training and strong generative performance, with the same residual control design remaining effective. Key numerical results include:

• 400-layer MV-Split: achieves FID-50K = 2.60, IS = 185.5 at 50k steps (best stable result among compared methods).
• 1000-layer MV-Split: achieves FID-50K = 2.77, IS = 217.3 at 50k steps; remains stably trainable without collapse.

  Figure 3: High-fidelity text-to-image generation samples from the 1000-layer MV-Split DiT, demonstrating stable scaling and semantic breadth.

MV-Split controls the stability frontier, marking a significant shift from isotropic residual gating paradigms.

Figure 4: Full-horizon training loss curves confirming sustained stable convergence in 400- and 1000-layer MV-Split runs.

Analysis: Gradient Mode Decomposition and Alignment Law

MV-Split's mode-selective gain is empirically validated by decomposing residual-writer gradients and confirming successful suppression of the mean-coherent component while maintaining robust centered updates. The alignment-amplification law predicts when token-wise gradients enter coherent accumulation—gradient norm amplifies in proportion to pairwise token/adjoint alignment, consistent with MMS onset.

Figure 5: Step-level gradient trace on spike—top-K parameter family norm dominance aligns with mean-coherent gradient amplification.

MV-Split in Context: Architectural Implications and Extensions

MV-Split operates orthogonally to previously reported normalization, gating, and residual scaling strategies. It provides a mechanistic fix for the mean-dominated collapse state in sequence models with Post-Norm residuals, demonstrating that stable depth scaling in diffusion backbones is contingent not just upon global norm control but upon mode-selective residual interface design.

MV-Split preserves structurally useful directions such as the token mean, which acts as an implicit global timestep carrier in noise-agnostic backbones, preserving global scene context and temporal coordinate decodability.

Figure 6: Linear probes confirm that the token-mean subspace reliably encodes the continuous timestep throughout depth.

Practical and Theoretical Implications

Practically, MV-Split enables much deeper DiTs, unlocking larger-scale generative models with reliable convergence and improved sample quality. Its efficacy at 1000 layers (and potentially beyond) expands the design space for spatiotemporal and multimodal diffusion models. Theoretically, the mechanism illustrates the importance of subspace-level control in architectures dominated by transformer-style residual merges. The writer-gradient decomposition and alignment-amplification law provide avenues for predictive training dynamics and design of resilient deep architectures, with MV-Split offering a blueprint for stabilizers orthogonal to global norm-based methods.

Future Outlook

Future directions include extending MV-Split to architectures beyond standard Softmax attention—such as state-space sequence mixers (e.g., Mamba)—and exploring its efficacy in extreme-context video and 3D generation tasks. Further theoretical development may yield predictive models for the onset time of MMS in un-stabilized designs, advancing understanding of deep training dynamics. High-performance implementation strategies (e.g., Triton fusion for efficient normalization and merge ops) are vital for practical deployment at extreme depth.

Conclusion

Mean Mode Screaming identifies a specific mean-dominated failure channel in ultra-deep Diffusion Transformers, driven by gradient-mode amplification and attention-induced geometric vulnerabilities. MV-Split Residuals, which anisotropically decouple and gain-limit the mean and centered residual updates, eliminate collapse and enable stable training at 1000+ layers. This architectural innovation redefines the stability-constrained quality frontier, enabling robust scaling of generative diffusion models and offering theoretical, practical, and engineering insights into the mechanics of ultra-deep sequence modeling (2605.06169).