Cause of the Classification-Head Anisotropy Exception

Determine whether the lack of anisotropy growth observed in the final classification head of networks trained on standard tasks is caused by the softmax bottleneck phenomenon.

Background

The paper reports that, unlike the other layers, the final classification head does not exhibit the roughly exponential growth of anisotropy observed across architectures and tasks. The authors propose, but do not establish, that this exception may result from the softmax bottleneck phenomenon. Determining whether the softmax bottleneck is the causal explanation would clarify why anisotropy-induced plasticity loss develops differently in the classification head and could help distinguish architectural effects from the general spectral-collapse mechanism studied in the paper.

References

We find that the trend of all layers except the final classification head experiencing a roughly exponential growth of anisotropy occurs across architectures and tasks; we hypothesize that the lack of anisotropy in the classification head may be due to the softmax bottleneck phenomenon \citep{yang2018breaking, NEURIPS2018_9dcb88e0}.

— SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learning  (2608.18319 - Kastner et al., 18 Aug 2026) in Section 3, “Anisotropy-Induced Plasticity Loss,” paragraph following Figure 1