Explain directed scale drift in undercomplete ReLU autoencoders

Investigate the geometric basis of the decoder-heavy scale bias in undercomplete ReLU autoencoders with neuron-wise positive scaling symmetries.

Background

The paper presents preliminary numerical evidence that undercomplete ReLU autoencoders exhibit the same qualitative decoder-heavy scale drift as undercomplete linear autoencoders. ReLU networks retain neuron-wise positive scaling symmetries, but the richer nonlinear geometry prevents the direct transfer of the linear PCA-manifold argument. A concrete unresolved problem is therefore to determine which geometric properties of the decoder manifold and residual decomposition produce the observed directional bias, and under what conditions the phenomenon holds.

References

We leave an in-depth investigation of the geometric basis for this bias to future work.

— Broken scale symmetries in undercomplete linear autoencoders  (2610.03640 - Pashakhanloo et al., 2 Oct 2026) in Section 5, “Generalization to homogeneous nonlinear autoencoders”