Determine the cause of MXFP4 P/V training divergence

Determine whether MXFP4 probability/value training instability is caused by forward quantization, the saved MXFP4 value payload, or the use of MXFP4 representations in backward, and establish whether MXFP4 is fundamentally unsuitable for training.

Background

The paper reports that both tested MXFP4 P/V training trajectories diverged, whereas corresponding FP8 P/V trajectories remained non-divergent. This behavior persisted across two learned-projection formats, making MXFP4 P/V the common factor in the observed failures.

However, the experiments do not isolate which component is responsible: quantization in the forward pass, the saved value representation, or the use of MXFP4 values during backward. The authors therefore leave unresolved both the specific mechanism of divergence and whether MXFP4 is intrinsically unsuitable for training.