Conditions under which normalization layers substitute for volume control

Determine conditions under which normalization layers in neural networks effectively substitute for volume control in distance-based log-sum-exp objectives, and characterize when they fail to do so, with respect to preventing collapse in implicit expectation-maximization dynamics.

Background

Normalization layers (e.g., layer normalization) and related regularizers are often used to stabilize training, but it is unclear when they provide the same role as explicit volume penalties that prevent metric collapse in mixture-like learning.

Clarifying the circumstances in which normalization layers can reliably stand in for volume control, and when they cannot, would connect the implicit EM framework to practical stability concerns and guide objective/architecture choices.

References

Several directions remain open. Understanding when normalization layers substitute for volume control, and when they do not, would connect the implicit EM framework to practical stability concerns.

— Gradient Descent as Implicit EM in Distance-Based Neural Models  (2512.24780 - Oursland, 31 Dec 2025) in Discussion, Open Directions (Section 7, Open Directions)

We attempted to mitigate this degradation through regularization, but were unable to obtain stable training.

— Emergent Multi-View Geometry Through Self-Distillation  (2609.39227 - Nordström et al., 30 Sep 2026) in Appendix, Section “Things That Did Not Work” (Section sec.supp:not-work)