Scale-and-Shift-Invariant Loss (SSIL)

• Scale-and-Shift-Invariant Loss (SSIL) is a class of loss functions that remains invariant to both additive and multiplicative transformations, aligning with rank-based evaluation metrics.
• SSIL, exemplified by the Z-loss, employs Z-normalized scores and tunable hyperparameters to ensure smooth optimization with bounded gradients.
• Its spherical loss framework reduces computational complexity, making it practical for training with extremely large output spaces compared to conventional log-softmax.

A Scale-and-Shift-Invariant Loss (SSIL) is a class of loss functions in multi-class classification that remain unchanged when their input vectors are subjected to affine transformations—specifically, rescaling and shifting. The canonical example is the Z-loss, which establishes a loss landscape invariant to both additive and multiplicative changes of pre-activation outputs. This invariance aligns SSILs with the nature of rank-based evaluation metrics while delivering computational advantages in extreme classification scenarios with very large output spaces (Brébisson et al., 2016).

1. Mathematical Definition and Properties

Given a neural network pre-activation vector $o = [o_1, \ldots, o_D] \in \mathbb{R}^D$ and a target class $c \in \{1, \ldots, D\}$, define the mean and standard deviation:

• $\mu = \frac{1}{D} \sum_{k=1}^D o_k$
• $\sigma^2 = \frac{1}{D} \sum_{k=1}^D o_k^2 - \mu^2$

The Z-normalized score for coordinate $k$ is given by:

• $z_k = \frac{o_k - \mu}{\sigma}$

With tunable hyperparameters $a > 0$ (scale) and $b \in \mathbb{R}$ (shift), the Z-loss for the target $c$ is:

$$L_Z(o, c) = \frac{1}{a} \log \left[1 + \exp\left(a \left[b - z_c\right]\right)\right]$$

This formulation depends solely on $c \in \{1, \ldots, D\}$0, but as $c \in \{1, \ldots, D\}$1 is a function of all $c \in \{1, \ldots, D\}$2, gradients are distributed across all output coordinates. The softplus nonlinearity ensures smoothness and bounded gradients.

2. Shift and Scale Invariance

The Z-loss $c \in \{1, \ldots, D\}$3 is invariant under any affine transformation $c \in \{1, \ldots, D\}$4 for $c \in \{1, \ldots, D\}$5, $c \in \{1, \ldots, D\}$6:

$c \in \{1, \ldots, D\}$7

This is a direct consequence of the properties of mean and standard deviation under affine transforms: both shift and scale changes in $c \in \{1, \ldots, D\}$8 result in identical $c \in \{1, \ldots, D\}$9. As such, $\mu = \frac{1}{D} \sum_{k=1}^D o_k$0 is strictly a function of the relative rank and normalized deviation of the target output.

By contrast, the log-softmax loss is only shift-invariant, while hierarchical softmax is invariant to neither shift nor scale on pre-activations.

3. Spherical Loss Family and Computational Efficiency

SSILs like the Z-loss belong to the spherical loss family—those expressible as functions of $\mu = \frac{1}{D} \sum_{k=1}^D o_k$1, $\mu = \frac{1}{D} \sum_{k=1}^D o_k$2, and $\mu = \frac{1}{D} \sum_{k=1}^D o_k$3. Because the Z-loss is ultimately

$\mu = \frac{1}{D} \sum_{k=1}^D o_k$4

for some $\mu = \frac{1}{D} \sum_{k=1}^D o_k$5, optimization may be performed using factored output-layer representations, $\mu = \frac{1}{D} \sum_{k=1}^D o_k$6 ($\mu = \frac{1}{D} \sum_{k=1}^D o_k$7, $\mu = \frac{1}{D} \sum_{k=1}^D o_k$8), with updates in $\mu = \frac{1}{D} \sum_{k=1}^D o_k$9 per example, independent of the number of classes $\sigma^2 = \frac{1}{D} \sum_{k=1}^D o_k^2 - \mu^2$0.

This is not possible for standard log-softmax, which has $\sigma^2 = \frac{1}{D} \sum_{k=1}^D o_k^2 - \mu^2$1 update cost due to the need to process all output dimensions. Hierarchical softmax reduces complexity to $\sigma^2 = \frac{1}{D} \sum_{k=1}^D o_k^2 - \mu^2$2 (for balanced trees), but this still grows with $\sigma^2 = \frac{1}{D} \sum_{k=1}^D o_k^2 - \mu^2$3. The independence from $\sigma^2 = \frac{1}{D} \sum_{k=1}^D o_k^2 - \mu^2$4 in the spherical setting allows practical training for output spaces with hundreds of thousands to millions of categories (Brébisson et al., 2016).

4. Theoretical and Empirical Comparison with Competing Losses

Invariances and Dynamics:

• Z-loss offers invariance to both additive and multiplicative transformations, aligning with the invariance properties of rank-based metrics.
• Gradients of Z-loss sum to zero, implementing competitive learning among classes; the gradients are bounded, enabling existence of fixed points and improved numerical stability relative to log-softmax.
• Log-softmax, while competitive (gradients sum to zero), lacks scale-invariance and suffers from the possibility of ever-growing output magnitudes.

Complexity Comparison Table

Loss Type	Invariances	Update Cost
Z-loss	Shift & Scale	$\sigma^2 = \frac{1}{D} \sum_{k=1}^D o_k^2 - \mu^2$5
Log-softmax	Shift only	$\sigma^2 = \frac{1}{D} \sum_{k=1}^D o_k^2 - \mu^2$6
Hierarchical SM	Neither full	$\sigma^2 = \frac{1}{D} \sum_{k=1}^D o_k^2 - \mu^2$7 or $\sigma^2 = \frac{1}{D} \sum_{k=1}^D o_k^2 - \mu^2$8

Empirical Metrics:

On Penn Treebank ($\sigma^2 = \frac{1}{D} \sum_{k=1}^D o_k^2 - \mu^2$9) and One Billion Word ($k$0), Z-loss (with tuned $k$1) achieves lower top-$k$2 error than log-softmax, Taylor-softmax, or sigmoid-based cross-entropy on top-{\small $k$3} error rates. Wall-clock convergence for Z-loss with factored (spherical) updates is competitive with hierarchical softmax and outpaces log-softmax by orders of magnitude. Numerical stability is enhanced due to bounded updates; fixed points ensure outputs do not diverge, reducing the risk of overflow or underflow seen with log-softmax training.

5. Large-Scale Experimental Outcomes

On the One Billion Word dataset ($k$4), key results are reported for two network architectures (net1, net2) and compared to hierarchical softmax and standard softmax.

Training Time Table

Loss	CPU (whole model)	GPU (output only)
Softmax	78.5 days	4.44 days
H-softmax	—	10.88 hours
Z-loss	7.50 days	1.24 hours

Top-k Error Rate Table

Loss	Arch.	Top-1 Err (%)	Top-20 Err (%)	Train Time
Constant baseline	—	95.44	65.58	—
Softmax (naive)	net1	—	—	~40 days
H-softmax	net1	71.00	35.73	4.08 days
Z-loss	net1	72.13	36.43	0.97 days
Z-loss	net2	70.77	38.29	3.14 days

A salient observation is that while hierarchical softmax achieves slightly lower top-$k$5 errors on net1, Z-loss permits much faster training; when training a larger net2 network for the same duration as hierarchical softmax on net1, Z-loss surpasses hierarchical’s top-1 error.

6. Hyperparameter Tuning and Adaptation

Z-loss exposes two distinct, tunable hyperparameters: $k$6 (softplus sharpness) and $k$7 (decision boundary). These parameters are not absorbable by mere scaling of the weights, in contrast to a log-softmax.

• Varying $k$8 (with fixed $k$9) shifts the top-$z_k = \frac{o_k - \mu}{\sigma}$0 error minimum on task metrics, while $z_k = \frac{o_k - \mu}{\sigma}$1 controls the cut point.
• These hyperparameters thus allow explicit alignment of the surrogate training criterion with the top-$z_k = \frac{o_k - \mu}{\sigma}$2 or mean reciprocal rank (MRR) task loss targeted at deployment time.
• It is possible, in principle, to adapt $z_k = \frac{o_k - \mu}{\sigma}$3 dynamically by gradient descent or hypergradient methods on a held-out validation estimate.
• Z-normalization can be similarly applied to other losses (e.g., log-softmax), generating a broader family of SSILs exhibiting analogous invariances.

7. Summary and Implications

The Z-loss forms a paradigm in SSILs combining theoretical invariance properties, computational efficiency via the spherical loss framework, and empirically validated speed and stability advantages in large-scale classification. Its two hyperparameters provide essential adaptability to varied task losses, particularly rank-based metrics. A plausible implication is that SSILs like Z-loss enable practical scaling of neural classification architectures to vocabularies or label sets with millions of entries, attaining rapid convergence and stability without loss in accuracy under fixed resource constraints (Brébisson et al., 2016).