---
title: 'Log-Softmax: Definition, Properties, and Applications'
url: https://www.emergentmind.com/topics/log-softmax
type: topic
---

# Log-Softmax: Definition, Properties, and Applications

Log-softmax is the componentwise logarithm of the softmax transformation. For a logit vector $\mathbf{o}\in\mathbb{R}^{D}$, it is defined by
$$
\operatorname{logsoftmax}(\mathbf{o})_k
=
\log\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)}
=
o_k-\log\sum_{j=1}^{D}\exp(o_j).
$$
The resulting coordinates are log probabilities: they are nonpositive and exponentiate to a categorical distribution whose components sum to one. Log-softmax is the standard representation used in categorical negative log-likelihood, cross-entropy, likelihood computation, and many probabilistic neural-network objectives. It is mathematically equivalent to softmax followed by a logarithm, but its direct log-sum-exp formulation is substantially preferable for numerical computation.

## 1. Definition and probabilistic interpretation

Let $\mathbf{o}=W\mathbf{h}\in\mathbb{R}^{D}$ be the output of a neural network, where $D$ is the number of classes. Softmax maps unrestricted real-valued scores to the categorical posterior
$$
p(y=k\mid x)
=
\operatorname{softmax}(\mathbf{o})_k
=
\frac{\exp(o_k)}{\sum_{j=1}^{D}\exp(o_j)}.
$$
Every component is positive and the components sum to one. Log-softmax is the vector of corresponding logarithms:
$$
\ell_k(\mathbf{o})
=
\operatorname{logsoftmax}(\mathbf{o})_k
=
o_k-\operatorname{LSE}(\mathbf{o}),
$$
where
$$
\operatorname{LSE}(\mathbf{o})
=
\log\sum_{j=1}^{D}\exp(o_j).
$$

For a target class $c$, the categorical negative log-likelihood is
$$
L(\mathbf{o},c)
=
-\ell_c(\mathbf{o})
=
-o_c+\log\sum_{j=1}^{D}\exp(o_j).
$$
This is the single-example softmax cross-entropy. For a dataset, maximum likelihood minimizes
$$
\mathcal{L}(\theta)
=
-\sum_n\log p_\theta(y_n\mid x_n).
$$
Thus softmax is the probability-valued map, log-softmax is the log-probability-valued map, and cross-entropy or negative log-likelihood is the loss formed by negating the log-softmax coordinate associated with the observed class [1511.05042].

The transformation is invariant to common translation:
$$
\operatorname{softmax}(\mathbf{o}+a\mathbf{1})
=
\operatorname{softmax}(\mathbf{o}),
\qquad
\operatorname{logsoftmax}(\mathbf{o}+a\mathbf{1})
=
\operatorname{logsoftmax}(\mathbf{o}).
$$
Only relative logit differences affect the probability distribution. The log-sum-exp potential itself is translation-equivariant:
$$
\operatorname{LSE}(\mathbf{o}+a\mathbf{1})
=
\operatorname{LSE}(\mathbf{o})+a.
$$

Pairwise differences of log-softmax coordinates eliminate the common normalization:
$$
\ell_j(\mathbf{o})-\ell_i(\mathbf{o})
=
\log\frac{p_j}{p_i}
=
o_j-o_i.
$$
For linear class scores $a_i(u)=w_i^\top u+b_i$, this becomes
$$
\log\frac{p_j(u)}{p_i(u)}
=
(w_j-w_i)^\top u+(b_j-b_i).
$$
Consequently, pairwise log-odds boundaries are affine, even though the probabilities and individual log-softmax values are nonlinear normalized functions [1806.00728].

## 2. Differential and convex-analytic properties

Writing $s=\operatorname{softmax}_\beta(z)$ with inverse temperature $\beta>0$,
$$
s_i
=
\frac{\exp(\beta z_i)}{\sum_j\exp(\beta z_j)},
\qquad
\ell_i
=
\beta z_i-\log\sum_j\exp(\beta z_j).
$$
With the scaled log-sum-exp potential
$$
\operatorname{LSE}_\beta(z)
=
\frac{1}{\beta}\log\sum_j\exp(\beta z_j),
$$
the softmax is its gradient:
$$
\nabla\operatorname{LSE}_\beta(z)
=
\operatorname{softmax}_\beta(z).
$$
Since $\operatorname{LSE}_\beta$ is convex, softmax is a monotone map. Its Jacobian and the Hessian of the potential are
$$
J_s(z)
=
\nabla^2\operatorname{LSE}_\beta(z)
=
\beta\left(\operatorname{diag}(s)-ss^\top\right).
$$
For any $v$,
$$
v^\top\nabla^2\operatorname{LSE}_\beta(z)v
=
\beta\left(
\sum_i s_i v_i^2-
\left(\sum_i s_i v_i\right)^2
\right)\geq 0.
$$
The quadratic form is $\beta$ times a variance under the categorical distribution $s$.

The log-softmax Jacobian is
$$
\frac{\partial\ell_i}{\partial z_j}
=
\beta(\delta_{ij}-s_j),
\qquad
J_\ell(z)
=
\beta(I-\mathbf{1}s^\top).
$$
For the untempered case $\beta=1$,
$$
\frac{\partial\ell_i}{\partial z_j}
=
\delta_{ij}-s_j.
$$
For a fixed class $i$, $\ell_i$ is concave in $z$ because it is affine minus a convex log-sum-exp function. Its gradient is
$$
\nabla\ell_i(z)=\beta(e_i-s).
$$
The Jacobian of the vector-valued log-softmax map is generally nonsymmetric. Therefore, monotonicity and co-coercivity results proved for the softmax map do not automatically apply to the vector log-softmax map [1704.00805].

The softmax map is $\beta$-Lipschitz under the Euclidean norm:
$$
\|s(z)-s(z')\|_2
\leq
\beta\|z-z'\|_2.
$$
It is also $1/\beta$-co-coercive:
$$
(s(z)-s(z'))^\top(z-z')
\geq
\frac{1}{\beta}\|s(z)-s(z')\|_2^2.
$$
These properties follow from the gradient structure of log-sum-exp and the Baillon–Haddad theorem. Lower inverse temperature produces a smoother map; higher inverse temperature increases sensitivity to logit changes.

The entropy-regularized characterization is
$$
\max_{x\in\Delta^{n-1}}
\left\{
x^\top z-\frac{1}{\beta}\sum_i x_i\log x_i
\right\},
$$
whose unique maximizer is $\operatorname{softmax}_\beta(z)$. Softmax is therefore the entropy-regularized counterpart of argmax, with temperature $1/\beta$ controlling the trade-off between concentration and entropy. As $\beta\to0^+$, the distribution approaches the uniform distribution; as $\beta\to\infty$, it concentrates on maximal logits, subject to tie behavior.

## 3. Numerical evaluation

The identity
$$
\operatorname{logsoftmax}(x)_i
=
x_i-\operatorname{LSE}(x)
$$
should be evaluated with a maximum shift. Let
$$
m=\max_j x_j.
$$
Then
$$
\operatorname{LSE}(x)
=
m+\log\sum_j\exp(x_j-m),
$$
and
$$
\operatorname{logsoftmax}(x)_i
=
x_i-m-\log\sum_j\exp(x_j-m).
$$
All shifted exponent arguments are nonpositive, so exponential overflow is avoided. Underflow of terms far below the maximum is generally harmless because their contribution to the normalization is negligible [1909.03469].

When one shifted term is exactly zero, a more accurate log-sum-exp implementation uses
$$
\operatorname{LSE}(x)
=
m+\log1p\left(\sum_{j\neq k}\exp(x_j-m)\right),
$$
where $x_k=m$. The use of `log1p` preserves small contributions that might be lost when forming $1+z$ explicitly.

A fused `log_softmax` operation is generally preferable to computing softmax, followed by a logarithm. Explicit exponentiation and normalization can underflow a small probability to zero, producing $-\infty$ after the logarithm even when the corresponding log probability is representable. For finite inputs with at least one finite maximum, the shifted formulation avoids exponential overflow.

Relative error is not always an appropriate accuracy measure for log-softmax. The largest log-softmax coordinate may be close to zero, particularly when one class dominates. A small absolute error can then correspond to a large relative error. The final subtraction $x_i-\operatorname{LSE}(x)$ can also exhibit cancellation for a unique maximal logit. Absolute error is therefore more informative for coordinates near zero.

A recent analysis identifies two additional floating-point effects in common implementations. **Softmax Collapse** occurs when the winning-class probability rounds to one, causing the computed gradient $q_t-1$ to round to zero while non-target gradients remain nonzero. **Softmax Cancellation** occurs in distillation when separately rounded teacher and student probabilities are subtracted, producing a nonzero gradient even when teacher and student logits coincide. The proposed remedies compute small differences through `softplus`, `log1p`, and `expm1`, and enforce the exact zero-sum gradient identity rather than obtaining the target gradient by subtracting one from a rounded probability [2610.31642].

For ordinary cross-entropy with target distribution $p$ and model distribution $q$,
$$
\nabla_z L=q-p,
\qquad
\sum_k\frac{\partial L}{\partial z_k}=0.
$$
For a hard target $t$,
$$
g_t=q_t-1,
\qquad
g_j=q_j\quad(j\neq t).
$$
In high-confidence regimes, an alternative expression preserves the small target gradient directly:
$$
g_t
=
-\frac{\exp(-m)}{1+\exp(-m)},
$$
where
$$
m=z_t-\log\sum_{j\neq t}\exp(z_j).
$$
This avoids losing the target gradient when $1+\exp(-m)$ rounds to one.

## 4. Representational capacity and alternatives

For a linear-logit language model with hidden representations $h_j\in\mathbb{R}^{d}$ and output embeddings $w_i\in\mathbb{R}^{d}$,
$$
z_{ij}=w_i^\top h_j.
$$
The matrix of log probabilities is
$$
A_{Q_\Theta}
=
W_\Theta H_\Theta^\top
-
\mathbf{1}_M(\log Z)^\top,
$$
where
$$
\log Z_j
=
\log\sum_{s=1}^{M}\exp(w_s^\top h_j).
$$
The first term has rank at most $d$, and the second has rank at most one. Therefore,
$$
\operatorname{rank}(A_{Q_\Theta})
\leq d+1.
$$
This is the **softmax bottleneck**: the conditional log-probability matrix generated by a linear-softmax model is confined to a low-dimensional linear space. The limitation concerns log probabilities, not necessarily the ordinary-rank of the probability matrix obtained after exponentiation [1902.08077].

The bottleneck can be understood directly from
$$
\log\operatorname{softmax}(z)
=
z-\operatorname{LSE}(z)\mathbf{1}.
$$
The exponential in softmax is canceled by the logarithm, leaving the original low-rank logits plus one shared normalization direction. If the true log-probability matrix has rank greater than $d+1$, exact representation is impossible within the linear-softmax parameterization.

Several alternatives modify the score function or normalization:

- **Sigsoftmax** uses
  $$
  f(z)_i
  =
  \frac{\exp(z_i)\sigma(z_i)}
  {\sum_m\exp(z_m)\sigma(z_m)},
  $$
  where $\sigma$ is the sigmoid. Its log unnormalized score contains
  $$
  2z_i-\log(1+\exp(z_i)),
  $$
  introducing nonlinear structure that can raise the rank of the log-output matrix. It preserves categorical normalization and monotonicity [1805.10829].

- **Learnable monotonic transformations** replace logits $z_i$ by $f(z_i)$ before softmax:
  $$
  Q(x_i)
  =
  \frac{\exp(f(w_i^\top h))}
  {\sum_s\exp(f(w_s^\top h))}.
  $$
  The resulting log probabilities are
  $$
  f(w_i^\top h)
  -
  \log\sum_s\exp(f(w_s^\top h)).
  $$
  Piecewise Linear Increasing Functions provide an efficient parameterization of $f$. The identity function recovers ordinary softmax, while sigsoftmax is a special case with
  $$
  f(x)=2x-\log(1+\exp x).
  $$
  These transformations preserve within-context logit ordering while avoiding the linear-logit rank bound [1902.08077].

- **Mixtures of softmaxes** combine several softmax distributions. The logarithm of a mixture does not reduce to one low-rank bilinear logit matrix plus a rank-one normalization term, allowing a higher-rank family of log distributions. Mixture-of-sigsoftmax extends this principle with sigsoftmax components [1805.10829].

- **Spherical normalizers** replace the exponential by positive functions such as
  $$
  g(o)=a_1+a_2o+a_3o^2.
  $$
  A quadratic normalizer depends on the score sum $s=\sum_i o_i$ and squared norm $q=\sum_i o_i^2$, enabling exact output-weight updates with computational cost $O(d^2)$ rather than $O(dD)$ under the spherical-loss algorithm. Log-Spherical Softmax uses $o_i^2+\epsilon$, while Log-Taylor Softmax uses
  $$
  1+o_i+\frac12o_i^2.
  $$
  The latter is strictly positive and requires no $\epsilon$ [1511.05042].

The empirical behavior is task-dependent. Log-Taylor Softmax outperformed ordinary log-softmax on the reported MNIST and CIFAR-10 experiments, while ordinary log-softmax performed better on CIFAR-100 and language-modeling tasks. On Penn Treebank and One Billion Word, log-softmax had substantially better perplexity and embedding quality. Sigsoftmax and related monotonic transformations improved language-model perplexity on reported Penn Treebank and WikiText-2 experiments, while mixture models achieved further improvements at higher computational cost [1805.10829].

## 5. Temperature, robustness, and generalized log losses

Ordinary log-softmax uses the ordinary exponential and logarithm. A generalized construction replaces them with tempered functions. The tempered logarithm is
$$
\log_t(x)
=
\frac{x^{1-t}-1}{1-t},
$$
with $\log_t(x)\to\log(x)$ as $t\to1$. For $0\leq t<1$, it is bounded below:
$$
\log_t(x)\geq-\frac{1}{1-t}.
$$
Its inverse is the tempered exponential
$$
\exp_t(x)
=
[1+(1-t)x]_+^{1/(1-t)}.
$$

Tempered softmax uses an implicitly determined normalization scalar:
$$
\hat y_i
=
\exp_{t_2}(a_i-\lambda_{t_2}(a)),
$$
where
$$
\sum_i\exp_{t_2}(a_i-\lambda_{t_2}(a))=1.
$$
At $t_2=1$, this reduces to ordinary softmax. For $t_2>1$, the output has heavier negative tails and assigns more probability mass to low-scoring classes.

The bi-tempered logistic loss combines a $t_1$-tempered Bregman divergence with a $t_2$-tempered probability map:
$$
L_{t_1,t_2}(a\mid y)
=
\Delta_{F_{t_1}}
\left(
y,
\exp_{t_2}(a-\lambda_{t_2}(a)\mathbf{1})
\right).
$$
The emphasized robust regime is
$$
0\leq t_1<1<t_2.
$$
The loss temperature $t_1<1$ bounds the influence of examples assigned very small probability, while $t_2>1$ produces heavy-tailed probabilities and reduces sensitivity to small-margin label noise. The mismatch makes the last-layer objective generally non-convex. Matched temperatures $t_1=t_2$ produce a convex matching loss, and $t_1=t_2=1$ recovers ordinary KL divergence, cross-entropy, and log-softmax [1906.03361].

For a one-hot target of class $c$, the complete bi-tempered loss is
$$
L_{t_1,t_2}
=
-\log_{t_1}\hat y_c
-
\frac{1}{2-t_1}
\left(
1-\sum_i\hat y_i^{2-t_1}
\right).
$$
The second, model-dependent term is essential; the loss is not merely a tempered negative log-likelihood. Its logit gradient contains escort-probability factors and reduces to $\hat y_i-y_i$ only in the ordinary case.

## 6. Applications, geometry, and limitations

### Classification and class geometry

For linear softmax scores, dominant regions are convex intersections of half-spaces:
$$
\mathcal C_j
=
\bigcap_{i\neq j}
\left\{
u:
(w_j-w_i)^\top u+(b_j-b_i)\geq0
\right\}.
$$
These are convex log-odds polytopes. Desired affine class boundaries can be encoded through linear difference constraints
$$
w_j-w_i=n_{ji},
\qquad
b_j-b_i=c_{ji}.
$$
Data-free synthesis therefore reduces to solving a linear system of class-parameter differences. Feasibility requires loop-summation constraints: prescribed pairwise normals and offsets must telescope to zero around every closed class loop. Convex separability alone is not sufficient for exact realization by an $m$-class linear softmax [1806.00728].

### Game theory and reinforcement learning

In entropy-regularized strategy selection, a softmax policy
$$
x=\operatorname{softmax}_\beta(z)
$$
is the gradient of the scaled log-sum-exp potential. In discounted game dynamics,
$$
\dot z=u-z,
\qquad
u=U(x),
$$
equilibria satisfy
$$
x^*=\operatorname{softmax}_\beta(U(x^*)).
$$
This is a logit equilibrium. Convexity of log-sum-exp and co-coercivity of softmax support convergence arguments under the stated anti-monotonicity and fixed-point assumptions. These results concern the probability-valued softmax map, not the vector-valued log-softmax map [1704.00805].

### Robustness verification

Because
$$
\operatorname{logsoftmax}_i(x)
=
x_i-\operatorname{LSE}(x),
$$
convex bounds on log-sum-exp yield bounds on log-softmax. An upper bound on LSE gives a lower bound:
$$
\operatorname{LSE}(x)\leq \overline{\operatorname{LSE}}(x)
\quad\Longrightarrow\quad
\operatorname{logsoftmax}_i(x)
\geq
x_i-\overline{\operatorname{LSE}}(x).
$$
The softmax itself admits convex lower and concave upper bounds suitable for convex relaxations of neural networks and transformer attention. These bounds can be constructed through exponential-reciprocal or log-sum-exp decompositions [2303.01713].

### Attribution

For a target class $c$,
$$
\frac{\partial p_c}{\partial z_j}
=
p_c(\delta_{cj}-p_j),
\qquad
\frac{\partial\ell_c}{\partial z_j}
=
\delta_{cj}-p_j.
$$
Therefore,
$$
\nabla_x p_c
=
p_c\nabla_x\ell_c.
$$
Probability gradients include an additional factor $p_c$ and can vanish when the target probability is close to zero or one. Log-softmax gradients retain the class-competition structure without the low-probability attenuation. For one-hot cross-entropy,
$$
\nabla_x\ell_c
=
-\nabla_x\mathcal{L}.
$$
Logits, probabilities, and log-softmax thus provide different attribution semantics: raw logits measure absolute target-score sensitivity, whereas probabilities and log probabilities measure relative class-discriminative sensitivity. Preliminary attribution experiments reported similar probability and log-softmax maps in several settings, but no universal superiority was established [2306.13197].

### Mixture-of-experts

Softmax gating assigns covariate-dependent mixture weights
$$
\pi_i(X)
=
\frac{\exp(\eta_i(X))}
{\sum_j\exp(\eta_j(X))},
\qquad
\eta_i(X)=\beta_{1i}^\top X+\beta_{0i}.
$$
The gate’s log probabilities are log-softmax values:
$$
\log\pi_i(X)
=
\eta_i(X)-\operatorname{LSE}(\eta(X)).
$$
In multinomial-logistic mixture-of-experts models, the same covariates can appear in both the gate and expert regressions. When expert slopes vanish, derivatives of the gating and expert components can become linearly dependent, producing local weak-identifiability and slower parameter-estimation rates. A bounded transformation $M(X)$ in the gate can restore derivative independence under the stated conditions [2310.14188].

## 7. Distinctions, limitations, and implementation principles

Log-softmax should not be conflated with several related objects:

- **Softmax** is a normalized exponential probability map.
- **Log-softmax** is the logarithm of that probability map.
- **Cross-entropy** or negative log-likelihood is a loss formed from target-weighted negative log-softmax values.
- **Log-odds** are pairwise differences of log-softmax coordinates.
- **Log-sum-exp** is the shared normalization potential.

The exponential is not uniquely forced by the requirement that outputs be positive and sum to one. Other positive normalizers can define categorical distributions, including quadratic spherical normalizers and learned monotonic transformations. Nevertheless, ordinary log-softmax remains a strong default because it combines a coherent categorical likelihood, simple gradients, translation invariance, broad software support, and stable log-sum-exp evaluation. Its principal limitations include computational dependence on output dimension, the linear-logit softmax bottleneck, sharp exponential competition, and floating-point failure modes in extreme confidence or distillation regimes.

A reliable implementation should compute
$$
\operatorname{logsoftmax}(x)_i
=
x_i-m-\log\sum_j\exp(x_j-m),
\qquad
m=\max_jx_j,
$$
preferably through a fused operation. It should avoid explicit softmax followed by logarithm, use `log1p` where a small quantity is added to one, use `softplus` for expressions such as $\log(1+\exp(-m))$, and use `expm1` when differences of nearly equal exponentials are required. In low-precision or high-confidence settings, preserving the zero-sum cross-entropy gradient is as important as preventing overflow.

The central structural identity is
$$
\operatorname{logsoftmax}(z)
=
z-\operatorname{LSE}(z)\mathbf{1}.
$$
It explains the probabilistic normalization, translation invariance, gradient structure, convex-analytic properties of softmax, affine pairwise log-odds, the low-rank softmax bottleneck, the construction of verification bounds, and the numerical algorithms required for reliable computation.

Source: https://www.emergentmind.com/topics/log-softmax