Z-Loss: Scale & Shift Invariant Loss

• Z-loss is a classification loss function that normalizes pre-activation vectors to achieve both shift and scale invariance, improving robustness in multi-class settings.
• It belongs to the spherical family, offering computational efficiency with per-example cost independent of the number of output classes, making it ideal for extreme classification.
• The formulation uses a softplus function with two hyperparameters, enabling tuning to approximate ranking-based metrics like top-k error rates.

The Z-loss is a classification loss function designed to address computational and statistical limitations of log-softmax for large-scale multi-class neural networks. It achieves efficiency by being a member of the spherical family, thereby enabling training with complexity independent of the number of output classes $D$. In addition, the Z-loss exhibits both shift and scale invariance, aligning it more closely with rank-based evaluation metrics such as top-$k$ error rates. Its formulation incorporates a normalization of the pre-activation vector followed by a softplus nonlinearity, parameterized by two hyperparameters.

1. Formal Definition

Given a pre-activation vector $o \in \mathbb{R}^D$ and a target index $c$, the Z-loss is defined using the statistics:

• $\mu = \frac{1}{D} \sum_k o_k$
• $\sigma^2 = \frac{1}{D} \sum_k o_k^2 - \mu^2$
• $z_k = \frac{o_k - \mu}{\sigma}$

Introducing hyperparameters $a > 0$ and $b \in \mathbb{R}$, the Z-loss for the correct class $c$ is

$k$0

The loss takes the form of a softplus function applied to the Z-normalized target score, controlled by $k$1 (sharpness) and $k$2 (location).

2. Mathematical Invariances

The Z-loss is uniquely invariant under both affine shift and scaling of the pre-activation vector:

• Shift invariance: Adding a constant $k$3 to all $k$4 shifts $k$5 equally, leaving $k$6 and thus $k$7 unchanged.
• Scale invariance: Multiplying all $k$8 by a nonzero $k$9 scales both $o \in \mathbb{R}^D$0 and $o \in \mathbb{R}^D$1, so $o \in \mathbb{R}^D$2 rescales by $o \in \mathbb{R}^D$3, which is compensated by tuning $o \in \mathbb{R}^D$4.

By contrast, the log-softmax $o \in \mathbb{R}^D$5 is only shift-invariant, not scale-invariant. The additional invariance of Z-loss reflects the property of rank-based metrics, which depend solely on the sorted order of $o \in \mathbb{R}^D$6.

3. Spherical Family Membership and Computational Efficiency

A loss $o \in \mathbb{R}^D$7 is in the spherical family if $o \in \mathbb{R}^D$8. The Z-loss satisfies this property, as it relies exclusively on $o \in \mathbb{R}^D$9, $c$0, and $c$1. As demonstrated by Vincent et al. (2015), spherical losses allow the use of low-rank factorizations for the output weight matrix $c$2 and the maintenance of summary statistics, resulting in per-example computation cost of $c$3 — irrespective of the output size $c$4. This is a significant advantage over log-softmax, for which naive gradient computation requires $c$5 complexity.

4. Comparison to Log-Softmax and Alternative Losses

Computational Complexity and Stability

Loss function	Per-example complexity	Shift-invariant	Scale-invariant
Log-softmax	$c$6	Yes	No
Hierarchical Softmax	$c$7	Yes	No
Z-loss	$c$8	Yes	Yes

Numerical stability is enhanced in the Z-loss by Z-normalization, which bounds $c$9 in $\mu = \frac{1}{D} \sum_k o_k$0. The softplus nonlinearity ensures gradients vanish for large $\mu = \frac{1}{D} \sum_k o_k$1, which prevents the explosion of pre-activation magnitudes. In contrast, the gradient of log-softmax, $\mu = \frac{1}{D} \sum_k o_k$2, never vanishes, and $\mu = \frac{1}{D} \sum_k o_k$3 can become unbounded.

Theoretical Distinctions

The two hyperparameters $\mu = \frac{1}{D} \sum_k o_k$4 of Z-loss provide additional flexibility for surrogate loss shaping, enabling closer approximation to various ranking losses relevant for top-$\mu = \frac{1}{D} \sum_k o_k$5 metrics. Log-softmax does not offer such flexibility, as its "temperature" can be absorbed by rescaling $\mu = \frac{1}{D} \sum_k o_k$6.

5. Empirical Evaluation

Penn Tree Bank (D=10,000)

Z-loss was evaluated against MSE, Taylor-softmax, cross-entropy-sigmoid, and log-softmax using top-$\mu = \frac{1}{D} \sum_k o_k$7 error rates and Mean Reciprocal Rank. Z-loss produced the best error rates for $\mu = \frac{1}{D} \sum_k o_k$8, with competitive performance for $\mu = \frac{1}{D} \sum_k o_k$9. Cross-entropy-sigmoid outperformed log-softmax for top-1, while MSE and Taylor-softmax underperformed for large $\sigma^2 = \frac{1}{D} \sum_k o_k^2 - \mu^2$0. The parameter $\sigma^2 = \frac{1}{D} \sum_k o_k^2 - \mu^2$1 was found critical; higher values improved high-$\sigma^2 = \frac{1}{D} \sum_k o_k^2 - \mu^2$2 metrics. Z-normalization alone brought only modest improvements.

One Billion Word Dataset (D=793,471)

For a network with 793,471 outputs, the Z-loss enabled an epoch processing time of 2.81 hours (whole model), compared to 4.56 days for naïve softmax and 12.23 hours for hierarchical softmax. Final test top-1/top-20 error rates and total training time:

Loss	Architecture	Top-1	Top-20	Training time
Constant	—	95.44%	65.58%	—
Softmax	net1	—	—	≈ 40 days
H-softmax	net1	71.0%	35.73%	4.08 days
Z-loss	net1	72.13%	36.43%	0.97 days
Z-loss	net2	70.77%	38.29%	3.14 days

The Z-loss maintained superior training efficiency, with performance competitive to hierarchical softmax and substantially faster convergence.

6. Hyperparameterization, Adaptation, and Practical Usage

The hyperparameters $\sigma^2 = \frac{1}{D} \sum_k o_k^2 - \mu^2$3 and $\sigma^2 = \frac{1}{D} \sum_k o_k^2 - \mu^2$4 in

$\sigma^2 = \frac{1}{D} \sum_k o_k^2 - \mu^2$5

are central to tailoring Z-loss for particular ranking objectives (e.g., different $\sigma^2 = \frac{1}{D} \sum_k o_k^2 - \mu^2$6 in top-$\sigma^2 = \frac{1}{D} \sum_k o_k^2 - \mu^2$7). Empirical sweeps over $\sigma^2 = \frac{1}{D} \sum_k o_k^2 - \mu^2$8 and $\sigma^2 = \frac{1}{D} \sum_k o_k^2 - \mu^2$9 are recommended, typically executed in initial training epochs. Z-loss’s computational advantage is maximized for large $z_k = \frac{o_k - \mu}{\sigma}$0, making it well-suited for extreme classification and large-scale language modeling.

The combination of shift and scale invariance, coupled with bounded gradients, offers improved numerical properties and mitigates the risk of feature explosion.

7. Extensions and Future Directions

It has been observed that applying Z-normalization alone (removing the softplus) to other losses introduces tunable hyperparameters, but the combined Z-loss configuration outperforms these alternatives in empirical evaluations. A suggested direction is the dynamic adaptation of $z_k = \frac{o_k - \mu}{\sigma}$1 during training to maintain optimal alignment of the surrogate loss with the evolving task metric.

The Z-loss thus exemplifies a simple yet computationally efficient loss function that leverages spherical family structure and affine invariance, supporting state-of-the-art training regimes for classification problems with very large output spaces (Brébisson et al., 2016).