Classification of deeper singular mixed-prefix softmax fibers

Classify the effective fibers of deeper softmax attention stacks whose prefixes contain singular mixtures of skew-symmetric and nonskew-symmetric score matrices, extending the available classifications beyond depth two and beyond the treated regular cases.

Background

The paper studies effective identifiability of serial normalized attention, representing each layer by a score matrix and the network by a combined value map. Under suitable analytic normalizer assumptions, the authors recover the effective scores generically and characterize certain exceptional fibers caused by consensus collapse.

For softmax with three or more tokens, multidirectional probes can recover information after nonzero skew prefixes, and the paper obtains a complete depth-two classification. However, the analysis does not cover deeper stacks in which singular prefixes combine skew and nonskew components. Such prefixes may mix different contrast-propagation mechanisms and can create more complicated observability and collapse behavior.