Explain and remedy numerical instabilities from unconstrained TTN optimization

Determine the origins of the numerical instabilities arising from unconstrained optimization of tree tensor networks and establish whether those instabilities can be remedied.

Background

The experiments indicate that unconstrained ADAM can produce tree tensor network parameters with exploding norms, especially for deeper networks. Subsequent orthogonalization may then become numerically unstable, causing downstream procedures such as model compression to fail or produce substantially degraded results.

The paper suggests possible explanations and remedies, including the exploding norm of the root tensor, weight decay, other regularization methods, and alternative downstream algorithms that avoid orthogonalization. However, the authors explicitly state that the source of the instability and the effectiveness of such remedies remain unresolved.