Log-Softmax: Definition, Properties, and Applications

• Log-softmax is the componentwise logarithm of the softmax transformation, producing non-positive values that sum to 0 and exponentiate to a categorical distribution.
• It is widely applied in loss functions for classification, including in categorical negative log-likelihood, cross-entropy, and probability computations, due to numerical stability and precision in handling large logit values.
• Robotics, natural language processing, and machine learning for optimizing probabilities and improving cross-entropy loss values.

Log-softmax is the componentwise logarithm of the softmax transformation. For a logit vector $\mathbf{o}\in\mathbb{R}^{D}$, it is defined by

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$

The resulting coordinates are log probabilities: they are nonpositive and exponentiate to a categorical distribution whose components sum to one. Log-softmax is the standard representation used in categorical negative log-likelihood, cross-entropy, likelihood computation, and many probabilistic neural-network objectives. It is mathematically equivalent to softmax followed by a logarithm, but its direct log-sum-exp formulation is substantially preferable for numerical computation.

1. Definition and probabilistic interpretation

Let $\mathbf{o}=W\mathbf{h}\in\mathbb{R}^{D}$ be the output of a neural network, where $D$ is the number of classes. Softmax maps unrestricted real-valued scores to the categorical posterior

$$p(y=k\mid x) = \operatorname{softmax}(\mathbf{o})_k = \frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)}.$$

Every component is positive and the components sum to one. Log-softmax is the vector of corresponding logarithms:

$$\ell_k(\mathbf{o}) = \operatorname{logsoftmax}(\mathbf{o})_k = o_k-\operatorname{LSE}(\mathbf{o}),$$

where

$$\operatorname{LSE}(\mathbf{o}) = \log\sum_{j=1}^{D}\exp(o_j).$$

For a target class $c$, the categorical negative log-likelihood is

$$L(\mathbf{o},c) = -\ell_c(\mathbf{o}) = -o_c+\log\sum_{j=1}^{D}\exp(o_j).$$

This is the single-example softmax cross-entropy. For a dataset, maximum likelihood minimizes

$$\mathcal{L}(\theta) = -\sum_n\log p_\theta(y_n\mid x_n).$$

Thus softmax is the probability-valued map, log-softmax is the log-probability-valued map, and cross-entropy or negative log-likelihood is the loss formed by negating the log-softmax coordinate associated with the observed class (Brébisson et al., 2015).

The transformation is invariant to common translation:

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$0

Only relative logit differences affect the probability distribution. The log-sum-exp potential itself is translation-equivariant:

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$1

Pairwise differences of log-softmax coordinates eliminate the common normalization:

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$2

For linear class scores $$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$3, this becomes

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$4

Consequently, pairwise log-odds boundaries are affine, even though the probabilities and individual log-softmax values are nonlinear normalized functions (Ahmed, 2018).

2. Differential and convex-analytic properties

Writing $$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$5 with inverse temperature $$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$6,

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$7

With the scaled log-sum-exp potential

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$8

the softmax is its gradient:

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$9

Since $\mathbf{o}=W\mathbf{h}\in\mathbb{R}^{D}$0 is convex, softmax is a monotone map. Its Jacobian and the Hessian of the potential are

$\mathbf{o}=W\mathbf{h}\in\mathbb{R}^{D}$1

For any $\mathbf{o}=W\mathbf{h}\in\mathbb{R}^{D}$2,

$\mathbf{o}=W\mathbf{h}\in\mathbb{R}^{D}$3

The quadratic form is $\mathbf{o}=W\mathbf{h}\in\mathbb{R}^{D}$4 times a variance under the categorical distribution $\mathbf{o}=W\mathbf{h}\in\mathbb{R}^{D}$5.

The log-softmax Jacobian is

$\mathbf{o}=W\mathbf{h}\in\mathbb{R}^{D}$6

For the untempered case $\mathbf{o}=W\mathbf{h}\in\mathbb{R}^{D}$7,

$\mathbf{o}=W\mathbf{h}\in\mathbb{R}^{D}$8

For a fixed class $\mathbf{o}=W\mathbf{h}\in\mathbb{R}^{D}$9, $D$0 is concave in $D$1 because it is affine minus a convex log-sum-exp function. Its gradient is

$D$2

The Jacobian of the vector-valued log-softmax map is generally nonsymmetric. Therefore, monotonicity and co-coercivity results proved for the softmax map do not automatically apply to the vector log-softmax map (Gao et al., 2017).

The softmax map is $D$3-Lipschitz under the Euclidean norm:

$D$4

It is also $D$5-co-coercive:

$D$6

These properties follow from the gradient structure of log-sum-exp and the Baillon–Haddad theorem. Lower inverse temperature produces a smoother map; higher inverse temperature increases sensitivity to logit changes.

The entropy-regularized characterization is

$D$7

whose unique maximizer is $D$8. Softmax is therefore the entropy-regularized counterpart of argmax, with temperature $D$9 controlling the trade-off between concentration and entropy. As $$p(y=k\mid x) = \operatorname{softmax}(\mathbf{o})_k = \frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)}.$$0, the distribution approaches the uniform distribution; as $$p(y=k\mid x) = \operatorname{softmax}(\mathbf{o})_k = \frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)}.$$1, it concentrates on maximal logits, subject to tie behavior.

3. Numerical evaluation

The identity

$$p(y=k\mid x) = \operatorname{softmax}(\mathbf{o})_k = \frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)}.$$2

should be evaluated with a maximum shift. Let

$$p(y=k\mid x) = \operatorname{softmax}(\mathbf{o})_k = \frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)}.$$3

Then

$$p(y=k\mid x) = \operatorname{softmax}(\mathbf{o})_k = \frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)}.$$4

and

$$p(y=k\mid x) = \operatorname{softmax}(\mathbf{o})_k = \frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)}.$$5

All shifted exponent arguments are nonpositive, so exponential overflow is avoided. Underflow of terms far below the maximum is generally harmless because their contribution to the normalization is negligible (Blanchard et al., 2019).

When one shifted term is exactly zero, a more accurate log-sum-exp implementation uses

$$p(y=k\mid x) = \operatorname{softmax}(\mathbf{o})_k = \frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)}.$$6

where $$p(y=k\mid x) = \operatorname{softmax}(\mathbf{o})_k = \frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)}.$$7. The use of log1p preserves small contributions that might be lost when forming $$p(y=k\mid x) = \operatorname{softmax}(\mathbf{o})_k = \frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)}.$$8 explicitly.

A fused log_softmax operation is generally preferable to computing softmax, followed by a logarithm. Explicit exponentiation and normalization can underflow a small probability to zero, producing $$p(y=k\mid x) = \operatorname{softmax}(\mathbf{o})_k = \frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)}.$$9 after the logarithm even when the corresponding log probability is representable. For finite inputs with at least one finite maximum, the shifted formulation avoids exponential overflow.

Relative error is not always an appropriate accuracy measure for log-softmax. The largest log-softmax coordinate may be close to zero, particularly when one class dominates. A small absolute error can then correspond to a large relative error. The final subtraction $$\ell_k(\mathbf{o}) = \operatorname{logsoftmax}(\mathbf{o})_k = o_k-\operatorname{LSE}(\mathbf{o}),$$0 can also exhibit cancellation for a unique maximal logit. Absolute error is therefore more informative for coordinates near zero.

A recent analysis identifies two additional floating-point effects in common implementations. Softmax Collapse occurs when the winning-class probability rounds to one, causing the computed gradient $$\ell_k(\mathbf{o}) = \operatorname{logsoftmax}(\mathbf{o})_k = o_k-\operatorname{LSE}(\mathbf{o}),$$1 to round to zero while non-target gradients remain nonzero. Softmax Cancellation occurs in distillation when separately rounded teacher and student probabilities are subtracted, producing a nonzero gradient even when teacher and student logits coincide. The proposed remedies compute small differences through softplus, log1p, and expm1, and enforce the exact zero-sum gradient identity rather than obtaining the target gradient by subtracting one from a rounded probability (Barron et al., 22 Oct 2026).

For ordinary cross-entropy with target distribution $$\ell_k(\mathbf{o}) = \operatorname{logsoftmax}(\mathbf{o})_k = o_k-\operatorname{LSE}(\mathbf{o}),$$2 and model distribution $$\ell_k(\mathbf{o}) = \operatorname{logsoftmax}(\mathbf{o})_k = o_k-\operatorname{LSE}(\mathbf{o}),$$3,

$$\ell_k(\mathbf{o}) = \operatorname{logsoftmax}(\mathbf{o})_k = o_k-\operatorname{LSE}(\mathbf{o}),$$4

For a hard target $$\ell_k(\mathbf{o}) = \operatorname{logsoftmax}(\mathbf{o})_k = o_k-\operatorname{LSE}(\mathbf{o}),$$5,

$$\ell_k(\mathbf{o}) = \operatorname{logsoftmax}(\mathbf{o})_k = o_k-\operatorname{LSE}(\mathbf{o}),$$6

In high-confidence regimes, an alternative expression preserves the small target gradient directly:

$$\ell_k(\mathbf{o}) = \operatorname{logsoftmax}(\mathbf{o})_k = o_k-\operatorname{LSE}(\mathbf{o}),$$7

where

$$\ell_k(\mathbf{o}) = \operatorname{logsoftmax}(\mathbf{o})_k = o_k-\operatorname{LSE}(\mathbf{o}),$$8

This avoids losing the target gradient when $$\ell_k(\mathbf{o}) = \operatorname{logsoftmax}(\mathbf{o})_k = o_k-\operatorname{LSE}(\mathbf{o}),$$9 rounds to one.

4. Representational capacity and alternatives

For a linear-logit LLM with hidden representations $$\operatorname{LSE}(\mathbf{o}) = \log\sum_{j=1}^{D}\exp(o_j).$$0 and output embeddings $$\operatorname{LSE}(\mathbf{o}) = \log\sum_{j=1}^{D}\exp(o_j).$$1,

$$\operatorname{LSE}(\mathbf{o}) = \log\sum_{j=1}^{D}\exp(o_j).$$2

The matrix of log probabilities is

$$\operatorname{LSE}(\mathbf{o}) = \log\sum_{j=1}^{D}\exp(o_j).$$3

where

$$\operatorname{LSE}(\mathbf{o}) = \log\sum_{j=1}^{D}\exp(o_j).$$4

The first term has rank at most $$\operatorname{LSE}(\mathbf{o}) = \log\sum_{j=1}^{D}\exp(o_j).$$5, and the second has rank at most one. Therefore,

$$\operatorname{LSE}(\mathbf{o}) = \log\sum_{j=1}^{D}\exp(o_j).$$6

This is the softmax bottleneck: the conditional log-probability matrix generated by a linear-softmax model is confined to a low-dimensional linear space. The limitation concerns log probabilities, not necessarily the ordinary-rank of the probability matrix obtained after exponentiation (Ganea et al., 2019).

The bottleneck can be understood directly from

$$\operatorname{LSE}(\mathbf{o}) = \log\sum_{j=1}^{D}\exp(o_j).$$7

The exponential in softmax is canceled by the logarithm, leaving the original low-rank logits plus one shared normalization direction. If the true log-probability matrix has rank greater than $$\operatorname{LSE}(\mathbf{o}) = \log\sum_{j=1}^{D}\exp(o_j).$$8, exact representation is impossible within the linear-softmax parameterization.

Several alternatives modify the score function or normalization:

• Sigsoftmax uses

$$\operatorname{LSE}(\mathbf{o}) = \log\sum_{j=1}^{D}\exp(o_j).$$9

where $c$0 is the sigmoid. Its log unnormalized score contains

$c$1

introducing nonlinear structure that can raise the rank of the log-output matrix. It preserves categorical normalization and monotonicity (Kanai et al., 2018).

• Learnable monotonic transformations replace logits $c$2 by $c$3 before softmax:

$c$4

The resulting log probabilities are

$c$5

Piecewise Linear Increasing Functions provide an efficient parameterization of $c$6. The identity function recovers ordinary softmax, while sigsoftmax is a special case with

$c$7

These transformations preserve within-context logit ordering while avoiding the linear-logit rank bound (Ganea et al., 2019).

• Mixtures of softmaxes combine several softmax distributions. The logarithm of a mixture does not reduce to one low-rank bilinear logit matrix plus a rank-one normalization term, allowing a higher-rank family of log distributions. Mixture-of-sigsoftmax extends this principle with sigsoftmax components (Kanai et al., 2018).
• Spherical normalizers replace the exponential by positive functions such as

$c$8

A quadratic normalizer depends on the score sum $c$9 and squared norm $$L(\mathbf{o},c) = -\ell_c(\mathbf{o}) = -o_c+\log\sum_{j=1}^{D}\exp(o_j).$$0, enabling exact output-weight updates with computational cost $$L(\mathbf{o},c) = -\ell_c(\mathbf{o}) = -o_c+\log\sum_{j=1}^{D}\exp(o_j).$$1 rather than $$L(\mathbf{o},c) = -\ell_c(\mathbf{o}) = -o_c+\log\sum_{j=1}^{D}\exp(o_j).$$2 under the spherical-loss algorithm. Log-Spherical Softmax uses $$L(\mathbf{o},c) = -\ell_c(\mathbf{o}) = -o_c+\log\sum_{j=1}^{D}\exp(o_j).$$3, while Log-Taylor Softmax uses

$$L(\mathbf{o},c) = -\ell_c(\mathbf{o}) = -o_c+\log\sum_{j=1}^{D}\exp(o_j).$$4

The latter is strictly positive and requires no $$L(\mathbf{o},c) = -\ell_c(\mathbf{o}) = -o_c+\log\sum_{j=1}^{D}\exp(o_j).$$5 (Brébisson et al., 2015).

The empirical behavior is task-dependent. Log-Taylor Softmax outperformed ordinary log-softmax on the reported MNIST and CIFAR-10 experiments, while ordinary log-softmax performed better on CIFAR-100 and language-modeling tasks. On Penn Treebank and One Billion Word, log-softmax had substantially better perplexity and embedding quality. Sigsoftmax and related monotonic transformations improved language-model perplexity on reported Penn Treebank and WikiText-2 experiments, while mixture models achieved further improvements at higher computational cost (Kanai et al., 2018).

5. Temperature, robustness, and generalized log losses

Ordinary log-softmax uses the ordinary exponential and logarithm. A generalized construction replaces them with tempered functions. The tempered logarithm is

$$L(\mathbf{o},c) = -\ell_c(\mathbf{o}) = -o_c+\log\sum_{j=1}^{D}\exp(o_j).$$6

with $$L(\mathbf{o},c) = -\ell_c(\mathbf{o}) = -o_c+\log\sum_{j=1}^{D}\exp(o_j).$$7 as $$L(\mathbf{o},c) = -\ell_c(\mathbf{o}) = -o_c+\log\sum_{j=1}^{D}\exp(o_j).$$8. For $$L(\mathbf{o},c) = -\ell_c(\mathbf{o}) = -o_c+\log\sum_{j=1}^{D}\exp(o_j).$$9, it is bounded below:

$$\mathcal{L}(\theta) = -\sum_n\log p_\theta(y_n\mid x_n).$$0

Its inverse is the tempered exponential

$$\mathcal{L}(\theta) = -\sum_n\log p_\theta(y_n\mid x_n).$$1

Tempered softmax uses an implicitly determined normalization scalar:

$$\mathcal{L}(\theta) = -\sum_n\log p_\theta(y_n\mid x_n).$$2

where

$$\mathcal{L}(\theta) = -\sum_n\log p_\theta(y_n\mid x_n).$$3

At $$\mathcal{L}(\theta) = -\sum_n\log p_\theta(y_n\mid x_n).$$4, this reduces to ordinary softmax. For $$\mathcal{L}(\theta) = -\sum_n\log p_\theta(y_n\mid x_n).$$5, the output has heavier negative tails and assigns more probability mass to low-scoring classes.

The bi-tempered logistic loss combines a $$\mathcal{L}(\theta) = -\sum_n\log p_\theta(y_n\mid x_n).$$6-tempered Bregman divergence with a $$\mathcal{L}(\theta) = -\sum_n\log p_\theta(y_n\mid x_n).$$7-tempered probability map:

$$\mathcal{L}(\theta) = -\sum_n\log p_\theta(y_n\mid x_n).$$8

The emphasized robust regime is

$$\mathcal{L}(\theta) = -\sum_n\log p_\theta(y_n\mid x_n).$$9

The loss temperature $$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$00 bounds the influence of examples assigned very small probability, while $$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$01 produces heavy-tailed probabilities and reduces sensitivity to small-margin label noise. The mismatch makes the last-layer objective generally non-convex. Matched temperatures $$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$02 produce a convex matching loss, and $$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$03 recovers ordinary KL divergence, cross-entropy, and log-softmax (Amid et al., 2019).

For a one-hot target of class $$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$04, the complete bi-tempered loss is

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$05

The second, model-dependent term is essential; the loss is not merely a tempered negative log-likelihood. Its logit gradient contains escort-probability factors and reduces to $$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$06 only in the ordinary case.

6. Applications, geometry, and limitations

Classification and class geometry

For linear softmax scores, dominant regions are convex intersections of half-spaces:

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$07

These are convex log-odds polytopes. Desired affine class boundaries can be encoded through linear difference constraints

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$08

Data-free synthesis therefore reduces to solving a linear system of class-parameter differences. Feasibility requires loop-summation constraints: prescribed pairwise normals and offsets must telescope to zero around every closed class loop. Convex separability alone is not sufficient for exact realization by an $$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$09-class linear softmax (Ahmed, 2018).

Game theory and reinforcement learning

In entropy-regularized strategy selection, a softmax policy

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$10

is the gradient of the scaled log-sum-exp potential. In discounted game dynamics,

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$11

equilibria satisfy

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$12

This is a logit equilibrium. Convexity of log-sum-exp and co-coercivity of softmax support convergence arguments under the stated anti-monotonicity and fixed-point assumptions. These results concern the probability-valued softmax map, not the vector-valued log-softmax map (Gao et al., 2017).

Robustness verification

Because

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$13

convex bounds on log-sum-exp yield bounds on log-softmax. An upper bound on LSE gives a lower bound:

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$14

The softmax itself admits convex lower and concave upper bounds suitable for convex relaxations of neural networks and transformer attention. These bounds can be constructed through exponential-reciprocal or log-sum-exp decompositions (Wei et al., 2023).

Attribution

For a target class $$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$15,

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$16

Therefore,

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$17

Probability gradients include an additional factor $$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$18 and can vanish when the target probability is close to zero or one. Log-softmax gradients retain the class-competition structure without the low-probability attenuation. For one-hot cross-entropy,

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$19

Logits, probabilities, and log-softmax thus provide different attribution semantics: raw logits measure absolute target-score sensitivity, whereas probabilities and log probabilities measure relative class-discriminative sensitivity. Preliminary attribution experiments reported similar probability and log-softmax maps in several settings, but no universal superiority was established (Lerma et al., 2023).

Mixture-of-experts

Softmax gating assigns covariate-dependent mixture weights

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$20

The gate’s log probabilities are log-softmax values:

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$21

In multinomial-logistic mixture-of-experts models, the same covariates can appear in both the gate and expert regressions. When expert slopes vanish, derivatives of the gating and expert components can become linearly dependent, producing local weak-identifiability and slower parameter-estimation rates. A bounded transformation $$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$22 in the gate can restore derivative independence under the stated conditions (Nguyen et al., 2023).

7. Distinctions, limitations, and implementation principles

Log-softmax should not be conflated with several related objects:

• Softmax is a normalized exponential probability map.
• Log-softmax is the logarithm of that probability map.
• Cross-entropy or negative log-likelihood is a loss formed from target-weighted negative log-softmax values.
• Log-odds are pairwise differences of log-softmax coordinates.
• Log-sum-exp is the shared normalization potential.

The exponential is not uniquely forced by the requirement that outputs be positive and sum to one. Other positive normalizers can define categorical distributions, including quadratic spherical normalizers and learned monotonic transformations. Nevertheless, ordinary log-softmax remains a strong default because it combines a coherent categorical likelihood, simple gradients, translation invariance, broad software support, and stable log-sum-exp evaluation. Its principal limitations include computational dependence on output dimension, the linear-logit softmax bottleneck, sharp exponential competition, and floating-point failure modes in extreme confidence or distillation regimes.

A reliable implementation should compute

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$23

preferably through a fused operation. It should avoid explicit softmax followed by logarithm, use log1p where a small quantity is added to one, use softplus for expressions such as $$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$24, and use expm1 when differences of nearly equal exponentials are required. In low-precision or high-confidence settings, preserving the zero-sum cross-entropy gradient is as important as preventing overflow.

The central structural identity is

$$\operatorname{logsoftmax}(\mathbf{o})_k = \log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)} = o_k-\log\sum_{j=1}^{D}\exp(o_j).$$25

It explains the probabilistic normalization, translation invariance, gradient structure, convex-analytic properties of softmax, affine pairwise log-odds, the low-rank softmax bottleneck, the construction of verification bounds, and the numerical algorithms required for reliable computation.