Finer-Grained Magnitude Granularities

Characterize the trade-off between communication overhead and gradient-scale fidelity for finer-grained magnitude representations, including per-row and per-head magnitude scalars, in compressed sign synchronization.

Background

Compressed sign synchronization uses one magnitude scalar per parameter tensor as a compromise between retaining scale information and minimizing communication overhead. Finer-grained alternatives, such as per-row or per-head scalars, could represent gradient scale more accurately but would increase the communication payload. The paper does not analyze this trade-off and leaves a broader study of magnitude granularities unresolved.