Existence of better Kronecker factorizations for the softmax KL Hessian

Determine whether Kronecker factorizations substantially better than the separable class-feature factorization used by SoftWater exist for approximating the smoothed softmax KL Hessian.

Background

SoftWater replaces the full class-feature curvature matrix with a separable Kronecker approximation formed from the expected softmax curvature and the hidden-state covariance. The authors report that this proxy underestimates the true distortion by at most 10% in their experiments and then investigate a Frobenius-optimal diagonal-Kronecker factorization using alternating power iteration. The SoftWater factors are close to the resulting fixed point from the outset, but the existence of materially better factorizations is not established in general.