Implementation of the attention reduction-order analysis

Implement the theoretical reduction-order analysis for attention kernels as a concrete bitwise-equivalence and descriptor mechanism.

Background

The paper extends its reduction-order framework theoretically to attention, whose recurrence introduces rescaling and multiple nested reductions in addition to matrix multiplications. It identifies how block boundaries, causal masking, row-sum trees, probability narrowing, and final normalization affect numerical results.

The implementation is expressly unresolved: the paper states that it provides a theoretical description while leaving implementation open. This is distinct from the broader discussion item concerning the incomplete treatment of attention and GEMM fusions.

References

Readers may refer to Appendix~\ref{app:attention} for the reduction order of a complex kernel such as flash attention, where this work offers a theoretical description and leaves the implementation open.