---
title: Spherical Loss Functions
url: https://www.emergentmind.com/topics/spherical-family-of-loss-functions
type: topic
---

# Spherical Loss Functions

The spherical family of loss functions comprises a distinct class of objective functions—primarily for multi-class classification and high-dimensional estimation—characterized by rotational invariance and efficient computability. In classification contexts, a loss is spherical if it depends only on three statistics of the model’s output vector: the sum of entries, the sum of squared entries, and the entry corresponding to the true class. These properties enable exact, output-size-independent training algorithms, and have motivated a spectrum of losses—such as Taylor-softmax, spherical softmax, and the Z-loss—each offering targeted invariance or computational advantages [1604.08859, 1606.08061, 1511.05042]. In estimation and shrinkage, spherical (orthogonally invariant) losses are similarly defined by their dependence on squared Euclidean distances, leading to minimax results and tractable dominant estimators [2102.13083].

## 1. Formal Definition and Characterization

A multi-class loss $L(o, c)$ is in the spherical family if it can be written as a function of three scalar summaries of the prediction vector $o \in \mathbb{R}^D$ (pre-activations):

\[
L(o, c) = F\left(\sum_{k=1}^D o_k,\ \sum_{k=1}^D o_k^2,\ o_c \right)
\]

for some $F:\mathbb{R}^3\to\mathbb{R}$, where $c$ is the true class index. This dependency is exhaustive: no other combination of the output vector’s coordinates is admitted [1604.08859, 1511.05042]. Examples subsumed in this form include:

- **Mean-squared error (MSE)** to one-hot labels: $L_{\mathrm{MSE}}(o,c)=\frac{1}{2}(\sum_k o_k^2 -2o_c+1)$
- **Taylor-softmax NLL**: $L_T(o,c) = -\log t_c(o)$ for $t_i(o) = (1+o_i+\frac{1}{2}o_i^2)/\sum_k (1+o_k+\frac{1}{2}o_k^2)$

In estimation, spherical loss functions rely on orthogonal invariance: risk and loss depend only on squared Euclidean distances, leading to invariance under simultaneous rotation of all parameter and estimator vectors [2102.13083]. Canonical forms include both “balanced” and “convex-combined” loss families:

\[
L_{\omega,\rho}(\delta, \theta) = \omega\rho(\|\delta - \delta_0(X)\|^2) + (1-\omega)\rho(\|\delta - \theta\|^2) 
\]
\[
L_{\omega,\ell}(\delta, \theta) = \ell( \omega\|\delta - \delta_0(X)\|^2 + (1-\omega)\|\delta - \theta\|^2 )
\]

with $\rho$, $\ell$ concave and increasing, and $\delta_0$ a target estimator [2102.13083].

## 2. Computational Advantages and Algorithms

The principal operational advantage of the spherical family in classification is that all gradients and weight updates can be computed without explicit construction or storage of high-dimensional output layers. For a neural net with last hidden state $h\in\mathbb{R}^d$ and output $o = Wh$, the standard $O(Dd)$ cost (with $D$ classes) is replaced by $O(d^2)$ via *factored output* representations [1606.08061]. The procedure exploits the fact that the loss and its gradient depend only on:

- $s = \sum_k o_k$ ($O(d)$ to compute via maintained summary)
- $q = \sum_k o_k^2$ ($O(d^2)$ using Gram matrix)
- $o_c$, requiring access only to the true class row of $W$, if needed.

These are achieved by factorizing $W=VU$, maintaining Gram matrix $Q=W^\top W$, mean vector $\bar{w}=W^\top 1_D$, and using rank-one/rank-two updates after each step. This enables exact and scalable learning in extreme classification regimes [1606.08061, 1604.08859].

## 3. Spherical Loss Variants: Taxonomy and Properties

Multiple losses belong to the spherical family, each aligning with distinct invariance or empirical objectives:

| Loss                      | Definition/Formula                                                      | Key Properties                                      |
|---------------------------|--------------------------------------------------------------------------|-----------------------------------------------------|
| MSE                       | $L_{\rm MSE} = \frac{1}{2}(\sum o_k^2 - 2o_c + 1)$                       | Spherical, convex                                   |
| Taylor-softmax            | $L_T(o,c) = -\log t_c(o)$ as above                                       | Spherical, smooth, underperforms for large $D$      |
| Spherical softmax         | $p_i = o_i^2 / \sum_j o_j^2$, $L = -\log p_c$                            | Spherical, scale-invariant, needs stabilization     |
| Z-loss                    | $L_Z(z_c) = (1/a)\log(1+\exp(a(b-z_c)))$ with $z_k = (o_k-\mu)/\sigma$   | Spherical, shift+scale invariant, tunable via $a,b$ |

All spherical losses support $O(d^2)$ updates and benefit from implicit orthogonality and competitive score dynamics among classes [1604.08859, 1511.05042]. The Z-loss, in particular, introduces shift- and scale-invariance by operating on Z-normalized outputs $z_c$, and contains softplus nonlinearity and adjustable parameters for matching specific ranking losses (e.g., top-$k$ error rates) [1604.08859].

## 4. Empirical Performance and Evaluation

Empirical studies confirm that spherical-family losses are competitive, particularly in large-output settings or when alternative task metrics (such as top-$k$ accuracy) are prioritized. On Penn Tree Bank ($D=10^4$), the Z-loss, after tuning, matched or outperformed softmax NLL in top-5 and top-20 error rates [1604.08859]:

| Loss        | Top-1  | Top-5 | Top-10 | Top-20 |
|-------------|--------|-------|--------|--------|
| Softmax     | 30.5%  | 14.9% | 10.8%  | 7.8%   |
| Z-loss      | 30.7%  | 13.8% | 10.0%  | 7.1%   |

For very large $D$ (e.g., One Billion Word, $D\approx8\times10^5$), Z-loss enabled exact gradient-based training up to 40$\times$ faster than naive softmax and $\sim4\times$ faster than hierarchical softmax, at small cost in top-$k$ accuracy [1604.08859]. On low-dimensional outputs (e.g., MNIST, CIFAR-10), spherical losses (especially log-Taylor-softmax) are often competitive or slightly superior to softmax [1511.05042], but for very large vocabularies, the standard softmax typically achieves the best perplexity and ranking metrics.

## 5. Spherical Family in Shrinkage Estimation

In estimation and statistical shrinkage, spherical loss functions are defined by orthogonal invariance—loss and risk depend only on squared distances—enabling extensions of Baranchik-type minimax shrinkage estimators to broad settings. Specifically, for any spherically symmetric data distribution in $\mathbb{R}^d$ (density $f(\|x-\theta\|^2)$), and for losses depending on $\|\delta - \delta_0\|^2$ and $\|\delta - \theta\|^2$, explicit forms for minimax shrinkage estimators can be constructed that uniformly lower risk compared to the naive estimator $X$ [2102.13083]. These results generalize classical James-Steinian and Brandwein-Strawderman risk bounds to more general concave penalty functions.

## 6. Spherical Family for Embedding Geometries

The spherical family also extends to losses enforcing spherical geometry in embeddings, such as those optimized over the unit hypersphere in metric learning and retrieval applications. Spherical softmax classifiers, as well as margin-based angular variants (e.g., ArcFace, CosFace, SphereFace), compute class probabilities as

\[
p(y \mid z) = \frac{\exp(\beta \cos \theta_y)}{\sum_{j=1}^Y \exp(\beta \cos \theta_j)}
\]

where $\|z\|_2 = 1$, $\|w_j\|_2 = 1$, and $\theta_j = \arccos(w_j^\top z)$. Probabilistic variants such as the von Mises–Fisher (vMF) loss operate by modeling both embeddings and weights as stochastic samples from vMF distributions, with built-in uncertainty through concentration parameter $\kappa$ [2103.15718]. Empirical comparisons indicate that spherical losses can consistently improve both accuracy and calibration across fixed-set and retrieval tasks, especially when normalization and angular discrimination are desired [2103.15718].

## 7. Practical Considerations and Implementation

Practical exploitation of the spherical family requires adherence to the restriction that the loss depend only on the designated sufficient statistics. Implementation of O($d^2$) routines is then straightforward via matrix factorizations and cached summary statistics. For the Z-loss, insertion into modern deep learning frameworks (e.g., PyTorch) requires only modifications to output normalization and activation, with the option to activate the factored weight update for full $D$-independent efficiency [1604.08859]. Tuning of loss hyperparameters (e.g., $a, b$ for the Z-loss) is best performed using the actual evaluation metric. These techniques are highly recommended for tasks involving extreme-classification, large-vocabulary language modeling, or settings requiring specific invariance properties or top-$k$ performance. For low-output settings, log-Taylor-softmax may be favored for its stability and accuracy [1511.05042]; for very large $D$, Z-loss yields optimal computational scalability with minimal test loss degradation [1604.08859, 1606.08061].

Source: https://www.emergentmind.com/topics/spherical-family-of-loss-functions