Neural Collapse in Ordinal Regression

• Neural collapse in ordinal regression is a geometric phenomenon where within-class features collapse to their mean and align with a single classifier direction.
• The use of fixed thresholds in cumulative link models drives a one-dimensional arrangement of class means, effectively reducing intra-class variance.
• Empirical studies confirm that this structured collapse accelerates convergence, improves fairness among classes, and enhances model interpretability.

Neural collapse is a geometric phenomenon observed in the final phase of deep network training, where last-layer features and classifier weights exhibit highly structured arrangements. This phenomenon, first characterized in standard multiclass classification, is marked by within-class feature concentration and symmetry of class means, most classically forming a simplex equiangular tight frame. Recent research has generalized neural collapse to more complex prediction paradigms such as ordinal regression. In the context of cumulative link models for ordinal regression, a variant known as Ordinal Neural Collapse (ONC) has been rigorously defined and analyzed, demonstrating that neural collapse is not specific to classification but reflects a universal inductive bias promoted by end-to-end deep learning under regularized risk minimization (Ma et al., 6 Jun 2025).

1. Ordinal Neural Collapse in Cumulative Link Models

In deep ordinal regression tasks, the cumulative link model (CLM) serves as the standard framework: prediction is performed via a single classifier vector and a set of monotonic thresholds partitioning the real line of latent scores into ordered classes. The unconstrained feature model (UFM)—in which each penultimate feature vector is optimized independently subject to the global objective—extends the classical analysis of neural collapse to this setting.

The hallmark properties of Ordinal Neural Collapse are:

• ONC1 (Within-Class Mean Collapse):

All optimal features belonging to the same class are identical,

$$\bm{h}_{q, i}^* = \bm{h}_{q}^*, \quad \forall i \in \{1,\ldots, n_q\}.$$

This extinguishes intra-class variance at optimality.

• ONC2 (Collapse to One-Dimensional Subspace):

All class means are strictly parallel to the classifier vector,

$\bm{h}_q^* \parallel \bm{w}^*,$

implying the class means lie in a one-dimensional affine subspace determined by the classifier.

• ONC3 (Ordered Latent Variables and Threshold Midpoints):

The inner products (logits) corresponding to the class means are strictly ordered by class,

$$z_1^* \leq z_2^* \leq \cdots \leq z_Q^*, \quad \text{where } z_q^* = (\bm{w}^*)^\top\bm{h}_q^*,$$

and, in the zero-regularization limit for symmetric link functions (e.g., logistic or probit), these logits precisely align with the midpoints of their adjacent thresholds,

$z_q^* = \frac{b_q + b_{q-1}}{2}.$

This places the geometric locus of class means at the critical points where the class-conditional posterior transitions.

The above properties constitute the ONC analogue to the classical "collapse" metrics (NC1–NC4) in multiclass classification.

2. Analytical Derivation via the Unconstrained Feature Model

The theoretical underpinning of ONC in ordinal regression is established by minimizing the CLM-based loss using the unconstrained feature model: $$\min_{\bm{w}, H}\left[ \frac{1}{N} \sum_{q=1}^{Q} \sum_{i=1}^{n_q} L(\bm{w}^\top \bm{h}_{q,i}, b_{q-1}, b_q) + \frac{\lambda_w}{2}\|\bm{w}\|_2^2 + \frac{\lambda_h}{2N} \sum_{q=1}^{Q} \sum_{i=1}^{n_q} \|\bm{h}_{q,i}\|_2^2 \right]$$ where $L(z,a,b) = -\log[g(b-z) - g(a-z)]$ and $g$ is the link function.

Under mild regularity conditions on the link function (log-concavity of the derivative), optimality conditions yield:

• ONC1:

Convexity with respect to the $\bm{h}_{q,i}$ ensures all features within a class are driven to their within-class mean.

• ONC2:

The regularizer and optimality induce the alignment of class means with the classifier.

• ONC3:

The stationary point equations (Equations of State) for the class logits $(z_q^*)$ and classifier norm $\|\bm{w}^*\|$ are coupled:

$$\frac{g'(b_q-z_q^*) - g'(b_{q-1}-z_q^*)}{g(b_q - z_q^*) - g(b_{q-1} - z_q^*)} + \lambda_h \frac{z_q^*}{(w^*)^2} = 0$$

$$\lambda_w w^* - \frac{\lambda_h}{(w^*)^3} \sum_{q=1}^Q \alpha_q (z_q^*)^2 = 0,\quad \alpha_q = n_q/N$$

In the limit $\lambda_h, \lambda_w \to 0$, these yield the midpoint rule for logits:

$z_q^* = \frac{b_q + b_{q-1}}{2}.$

Fixed thresholds are a critical assumption in these derivations; if the thresholds are not fixed, ONC3 admits more flexibility, and the geometric collapse is relaxed, though the core alignment remains robust under common practical regimes.

3. Empirical Evidence and Robustness of ONC

Extensive experiments across five public ordinal regression datasets using deep neural networks confirm the analytical predictions:

• ONC1:

Quantitative analysis demonstrates near-zero intra-class feature variance at convergence.

• ONC2:

Principal component analysis reveals that class means and classifier vector effectively span the same one-dimensional subspace, evidenced by near-perfect scalar alignment.

• ONC3:

When thresholds are fixed, the empirical logits for each class center align with the theoretical midpoints. When thresholds are learned, the phenomenon persists to a lesser but still statistically significant degree.

These empirical results hold across choices of link function (logit or probit) and thresholding strategy, supporting the universality of ONC in ordinal regression models trained end-to-end with deep architectures.

4. Practical Implications: Threshold Design and Model Interpretability

The emergence of ONC under fixed thresholds suggests direct interventions for model design:

• Fixed Thresholds:

Theoretical ONC is guaranteed only under fixed thresholds. Fixing thresholds empirically accelerates convergence and yields more interpretable latent geometries, with monotonic and equispaced class margins. This is recommended for practitioners seeking robustness, particularly in the presence of class imbalance.

• Class Imbalance:

Fixed-threshold models allocate latent space more fairly among classes, potentially mitigating bias in minority classes, in contrast to flexible thresholding which can concentrate class means in narrow subregions.

• Interpretability:

The geometric arrangement dictated by ONC enables clearer post-hoc model analysis and aligns naturally with human-intuitive scoring/scaling of outcomes in ordinal prediction scenarios.

5. Comparison: ONC vs. Classical Neural Collapse

NC in Classification	ONC in Ordinal Regression
NC1: Within-class collapse	ONC1: Features collapse to class mean
NC2: Simplex ETF	ONC2: Class means align along classifier axis
NC3: Classifier aligns with means	ONC2: Same (alignment along axis)
NC4: Nearest class center rule	ONC3: Latent variables strictly ordered; midpoints for fixed thresholds

The key structural difference is that, in classification, class means arrange as a regular simplex (ETF), achieving maximal angular separation, while in ordinal regression, ONC induces all class means to collapse onto the classifier direction, differing only by norm, and the discriminative geometry is fully specified by the latent orderings defined by the thresholds.

6. Implications for Representation Learning Theory

The ONC phenomenon extends the general principle that deep representation learning—when supplied with sufficient regularization—drives penultimate-layer representations to geometrically simple, highly constrained configurations matching the symmetries and invariances of the task. The appearance of ONC in ordinal regression supports the hypothesis that neural collapse-like regimes are generic end states in a wide range of supervised deep learning tasks, beyond classical classification, and underscores the effectiveness of analytical tools such as the UFM for characterizing these phenomena.

This synthesis reinforces that, under cumulative link models and UFM, deep architectures not only optimize empirical risk but spontaneously discover highly interpretable and structurally minimal internal feature geometries optimally suited to ordered decision tasks (Ma et al., 6 Jun 2025).