Softmax-saturation behavior of Sign GF under general embeddings

Establish whether the element-wise sign operation in Sign Gradient Flow weakens the effect of softmax saturation for general orthonormal token embeddings, analogously to the spectral orthogonalization in Spectral Gradient Flow.

Background

The paper analyzes Sign Gradient Flow for selected combinations of one-hot and Hadamard embeddings and finds that, for those constructions, the learning times depend only logarithmically on the inverse target error. This suggests that the element-wise sign operation may mitigate the slowdown caused by softmax saturation. However, Sign Gradient Flow is embedding-dependent, and the paper does not establish whether the same mitigation occurs for arbitrary orthonormal embeddings. The unresolved issue concerns whether logarithmic dependence on target accuracy and weakened saturation are general properties or artifacts of the specific embedding constructions studied.