---
title: Softmax Truncation Techniques
url: https://www.emergentmind.com/topics/softmax-truncation
type: topic
---

# Softmax Truncation Techniques

Searching arXiv for recent and directly relevant papers on softmax truncation and related softmax approximations.
Softmax truncation denotes a family of operations that reduce, restrict, or bypass some component of the softmax transformation, but the term is not used uniformly across the literature. In different arXiv works, it refers to omitting softmax entirely at inference and selecting the maximum logit with a comparator [2201.04562], truncating the exponential by low-order Taylor series or piecewise interpolation on a bounded domain [2501.13379], truncating the support of a discrete random variable before applying a Gumbel-Softmax relaxation [2003.01847], restricting normalization to a subset of classes during training or large-vocabulary optimization [2112.12433, 1907.10747, 2501.08563, 2508.03175], and modifying the output distribution so that it is closer to a one-hot vector [2508.02387]. The common thread is computational or statistical simplification of softmax, but the algorithmic meaning depends on whether the target is inference, optimization, approximation of \(e^x\), or robustness.

## 1. Terminological scope and major formulations

Across the cited literature, “softmax truncation” is best understood as a cluster of non-equivalent techniques rather than a single canonical operator. Some variants preserve the top-1 decision exactly, some approximate probabilities numerically, some change the training objective, and some only define a differentiable surrogate.

| Usage | Mechanism | Representative papers |
|---|---|---|
| Inference truncation | Replace softmax by max-logit selection | [2201.04562] |
| Function truncation | Approximate \(e^x\) by Taylor or LUT interpolation | [2501.13379], [2011.11538] |
| Support truncation | Restrict a discrete support to finite \(S_M\) before Gumbel-Softmax | [2003.01847] |
| Competition-set truncation | Keep only selected classes in normalization or sampling | [2112.12433], [2508.03175], [1907.10747], [2501.08563] |
| Output truncation toward one-hot | Boost top class and suppress others, then renormalize | [2508.02387] |

This multiplicity of meanings matters because statements that are exact in one regime are false in another. For example, argmax preservation is exact for comparator replacement at inference, but it does not imply probability preservation, calibration preservation, or unbiased gradient estimation [2201.04562, 1907.10747]. Similarly, truncating the exponential in low-resource hardware is distinct from truncating the class set in sampled softmax or top-\(k\) normalization [2501.13379, 2112.12433].

## 2. Argmax-preserving truncation at inference

A strict inference-time notion of softmax truncation appears in “Reduced Softmax Unit for Deep Neural Network Accelerators,” which formalizes the observation that for logits \(z=(z_1,\dots,z_K)\) with softmax probabilities \(p_i = e^{z_i}/\sum_{j=1}^K e^{z_j}\), the predicted class is unchanged by omitting softmax and taking the maximum logit directly: \(\arg\max_i p_i = \arg\max_i z_i\) [2201.04562]. The proof rests on two facts: the denominator is a positive constant with respect to \(i\) when comparing classes, and the exponential is strictly monotonically increasing. The paper states this as Theorem 1 in the form \(x>y \Rightarrow s(x)>s(y)\), where \(s(\cdot)\) denotes softmax output under a common denominator.

Under this interpretation, truncation means replacing the full softmax datapath—\(K\) exponential evaluations, an adder tree for \(\sum e^{z_j}\), and a divider or reciprocal unit—with a comparator tree plus index tracking [2201.04562]. For \(K\) classes, a balanced tree uses \(O(K)\) comparators over \(O(\log K)\) stages, and the same principle extends to top-\(k\) selection via a selection network or min-heap. The integration point is the DNN output layer: logits feed the comparator directly, and the unit returns the predicted index rather than calibrated probabilities.

This formulation is exact for top-1 and top-\(k\) ranking, but only for inference-only pipelines in which downstream logic consumes class indices rather than probabilities [2201.04562]. Equal logits remain ties under both softmax and comparator-based selection. Numerical stabilization such as subtract-max does not alter the equivalence, because \(p_i = e^{z_i-m}/\sum_j e^{z_j-m}\) preserves ordering. A recurrent misconception is that removing softmax must alter accuracy; in fact, top-1 accuracy is provably unchanged in this setting, whereas confidence scores and any probability-based post-processing are lost [2201.04562].

## 3. Truncating the exponential: Taylor and interpolation approximations

A second usage truncates the internal exponential rather than the class set. In “A Quantitative Evaluation of Approximate Softmax Functions for Deep Neural Networks,” softmax is defined for bounded logits, and the central device is to constrain the exponential input to \(x \in S = (-1,1)\) by scaling the preceding fully connected layer as
\[
\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}, \qquad
y_i = \mathbf{w}_i \cdot \left(\frac{\mathbf{x}}{n}\right) + b_i,
\]
with \(\mathbf{x}, \mathbf{w}_i \in S\), so that \(y_i \in S\) in fixed-point arithmetic [2501.13379]. Within this normalized domain, the paper evaluates Maclaurin truncations
\[
e^x \approx 1+x,\qquad
e^x \approx 1+x+\frac{x^2}{2},\qquad
e^x \approx 1+x+\frac{x^2}{2}+\frac{x^3}{6},
\]
together with Padé approximants and LUT-based linear and quadratic interpolation [2501.13379].

The reported error profile is sharply stratified. On \(x \in (-1,1)\) with \(N=100\) random test points, Taylor order 1 and order 2 both have RMSE near \(3\times 10^{-3}\), whereas Taylor order 3 reduces RMSE to approximately \(4.18\times 10^{-5}\). Linear LUT interpolation yields RMSE \(\approx 3.22\times 10^{-6}\), and quadratic LUT interpolation yields RMSE \(\approx 2.31\times 10^{-7}\), the lowest reported error across the tested methods [2501.13379]. At the same time, the CPU timing study on an Intel Core i7-4710HQ with GCC 7.5.0 shows that Taylor order-3 softmax and Padé 3/1 softmax have very similar, low execution times under `-Ofast`, whereas quadratic LUT interpolation is substantially slower; for size 100, the paper reports approximately \(1.61\times 10^{-6}\,\mathrm{s}\) for Taylor order-3 softmax, \(1.37\times 10^{-6}\,\mathrm{s}\) for Padé 3/1, and \(2.66\times 10^{-4}\,\mathrm{s}\) for LUT quadratic interpolation [2501.13379].

The study is frequently motivated by low-end FPGAs, but it does not report an actual FPGA implementation, target device, bit widths, resource utilization, latency, throughput, or end-to-end model accuracy for LeNet-5 or MobileNet v2 [2501.13379]. It also does not discuss subtract-max stabilization, log-sum-exp, reciprocal approximation for the denominator, clipping, or integration into attention softmax for large logits. Its concrete contribution is therefore numerical approximation of \(e^x\) on a bounded domain, not a full softmax hardware architecture.

A related but distinct line appears in “Exploring Alternatives to Softmax Function,” where Taylor softmax replaces \(\exp(x)\) by the truncated polynomial
\[
T_n(x)=\sum_{k=0}^{n}\frac{x^k}{k!}, \qquad
p_i^{(n)}=\frac{T_n(z_i)}{\sum_{j=1}^{K}T_n(z_j)},
\]
and emphasizes that even \(n\) guarantees positivity of \(T_n(z)\) for all \(z\) [2011.11538]. The paper studies both the exact finite-truncation gradient and an “infinite-series” backpropagation variant with gradient \(p_i^{(n)}-y_i\), then combines Taylor truncation with a soft margin in SM-Taylor softmax [2011.11538]. Its reported best accuracies are \(99.67\%\) on MNIST with SM-Taylor \(n=2, m=0.6\), \(87.47\%\) on CIFAR-10 with SM-Taylor \(n=2, m=0.6\), and \(49.95\%\) on CIFAR-100 with SM-Taylor \(n=4, m=0.6\) [2011.11538]. Here truncation is not primarily about hardware economy; it is an alternative normalization family with different curvature and margin behavior.

## 4. Truncating discrete support for pathwise gradient estimators

In generative modeling, softmax truncation has a third meaning: support truncation before categorical relaxation. “Generalized Gumbel-Softmax Gradient Estimator for Generic Discrete Random Variables” extends the Gumbel-Softmax trick beyond Bernoulli and categorical variables by first truncating a discrete support \(S\) to a finite subset \(S_M \subset S\), defining
\[
p_M(k\mid \theta)=\frac{p(k\mid \theta)}{Z_M(\theta)}, \qquad
Z_M(\theta)=\sum_{j\in S_M}p(j\mid \theta),
\]
then sampling i.i.d. Gumbels \(g_k=-\log(-\log u_k)\) and forming a relaxed categorical
\[
s_k=\frac{\log p_M(k\mid \theta)+g_k}{\tau}, \qquad
y_k=\frac{\exp(s_k)}{\sum_{j\in S_M}\exp(s_j)},
\]
followed by the linear transformation
\[
\tilde{x}=\sum_{k\in S_M} t(k)\,y_k
\]
to obtain a differentiable surrogate in the original outcome space [2003.01847].

The decisive role of truncation is that the Gumbel-Max and Gumbel-Softmax constructions require a finite category set, whereas Poisson, geometric, negative binomial, and related distributions often have infinite support [2003.01847]. The paper’s “special” linear transformation is simply the convex combination of support points, but it is exactly the map that converts a one-hot selection into the corresponding scalar, vector, or tensor outcome. Because the Gumbel variables are independent of \(\theta\), the estimator is pathwise. The derivation also exploits a cancellation property of the softmax Jacobian: any additive term in \(\partial \log p_M(k\mid \theta)/\partial \theta\) that is constant across \(k\) disappears after subtracting the \(y\)-weighted average [2003.01847].

This formulation introduces two separate approximation errors. The first is truncation bias from replacing an infinite-support distribution by \(S_M\); the second is relaxation bias from replacing a discrete sample by the softmax-relaxed vector \(y\) at temperature \(\tau\) [2003.01847]. The paper states that as \(\tau \to 0\), the relaxation approaches the Gumbel-Max limit, but gradients can become unstable. It also notes that straight-through variants can degrade performance empirically in these settings [2003.01847]. In the reported experiments, GenGS variants achieve the best gradient variance and loss across synthetic Poisson, binomial, multinomial, and negative binomial problems, generally yield the lowest negative ELBO in VAEs with Poisson, geometric, and negative binomial latents on MNIST and OMNIGLOT, and achieve the lowest test perplexities in the NVPDEF topic-model setting on 20Newsgroups and RCV1 [2003.01847].

Under this usage, “softmax truncation” does not mean dropping classes for efficiency at the output layer. It is a finite-support construction that makes reparameterization possible for arbitrary discrete random variables [2003.01847].

## 5. Truncating the competing class set during training

A fourth research line truncates the effective competition set in classification losses. One version is hard top-\(k\) support restriction. In “Sparse-softmax: A Simpler and Faster Alternative Softmax Transformation,” only the top-\(k\) logits \(\Omega_k\) receive nonzero mass,
\[
p_i=
\begin{cases}
\frac{e^{z_i}}{\sum_{j\in \Omega_k} e^{z_j}}, & i\in \Omega_k,\\
0, & i\notin \Omega_k,
\end{cases}
\]
with the modified loss
\[
\mathcal{L}_{\text{sparse}}(z,t)=\log\left(\sum_{i\in \Omega_k} e^{z_i}\right)-z_t
\]
to avoid the undefined \(-\log p_t\) when the target lies outside \(\Omega_k\) [2112.12433]. Within regions where \(\Omega_k\) is fixed, gradients are softmax-like on the retained support, zero on non-target masked classes, and \(-1\) on the target if it lies outside the top-\(k\) set [2112.12433]. The paper argues that standard cross-entropy imposes a margin lower bound scaling with \(\log(n-1)\), whereas top-\(k\) truncation heuristically reduces the effective pressure to approximately \(\log(k-1)\). Empirically, on high-dimensional text tasks it reports Micro-F1 improvements such as \(82.65 \rightarrow 83.50\) on WOS-46985 and \(95.19 \rightarrow 95.88\) on OOS-eval, both with \(k=20\) [2112.12433].

An adaptive variant appears in “Adaptive Sparse Softmax: An Effective and Efficient Softmax Variant,” where the retained set is sample-dependent rather than fixed-size. AS-Softmax imposes the probability-margin condition \(p_t-p_{i\neq t}\ge \delta\), masks any non-target class satisfying that inequality, and defines
\[
\tilde{p}_i=\frac{z_i \exp(o_i)}{\sum_{j=1}^{n} z_j \exp(o_j)}
\]
with binary mask \(z_i\), followed by \(\mathcal{L}_{AS}=-\log \tilde{p}_t\) [2508.03175]. If the target outranks every non-target by at least \(\delta\), all competitors are masked and the loss becomes zero. The paper further proposes adaptive gradient accumulation,
\[
\mathrm{steps}_{\mathrm{accum}}=\lambda \cdot \frac{N_{\mathrm{all}}}{N_{\mathrm{all}}-N_{\mathrm{masked}}},
\]
with monotonicity, adjacent-difference, and cap constraints, and reports about \(1.2\times\) training speedup over standard softmax while maintaining effectiveness [2508.03175]. Across text, image, and audio tasks with class sizes from 4 to 5,201, it reports gains such as SST5 with BERT \(51.90 \rightarrow 53.12\), SST5 with RoBERTa \(55.57 \rightarrow 57.29\), and SIGHAN2015 CSC F1 \(70.80 \rightarrow 72.81\) [2508.03175].

Large-output training uses a softer notion of truncation via sampled subsets rather than deterministic support restriction. “Sampled Softmax with Random Fourier Features” formalizes sampled softmax by selecting the true class and \(m\) negatives from \(q(i)\), adjusting sampled logits as \(o'_{i+1}=o_{s_i}-\log(mq_{s_i})\), and optimizing the sampled loss \(-\log p'_t\) [1907.10747]. The key theoretical point is that the gradient estimator is unbiased only if \(q(i)\propto e^{o_i}\), that is, when the sampling distribution matches the exact softmax distribution; practical distributions such as uniform or frequency-based priors are biased [1907.10747]. RF-softmax constructs \(q\) via Random Fourier Features under normalized embeddings so that sampling costs \(O(D\log n)\), and the paper reports wall-time examples such as \(0.6\) ms for RF-softmax with \(D=200\) versus \(6.5\) ms for exact-softmax sampling and \(1.4\) ms for full softmax at \(n=10\mathrm{k}\), batch size \(10\), \(m=10\), \(d=64\) [1907.10747].

“Adaptive Sampled Softmax with Inverted Multi-Index: Methods, Theory and Applications” pushes this sampled-softmax perspective further by decomposing class probability into codeword-level multinomials plus a residual stage under an inverted multi-index [2501.08563]. In its fast proposal, the residual distribution within a codeword cell is replaced by uniform sampling, converting per-query complexity from \(O(ND)\) to \(O(KD+K^2+M)\) after codebook construction [2501.08563]. The paper supplies KL-divergence and gradient-bias bounds, with \(D_{KL}(Q\|P)\le 2\|\tilde{o}\|_\infty\) for the MIDX proposal, and argues that smaller divergence implies faster convergence and better generalization [2501.08563]. It reports strong results on language modeling, recommendation, and extreme classification, with MIDX-rq outperforming static samplers and approaching full softmax at much lower cost [2501.08563].

These methods share a structural idea: they avoid full competition against all classes at every update. Their differences lie in whether truncation is hard or sampled, fixed or adaptive, and whether the resulting estimator is exact, biased, or merely low-bias.

## 6. One-hot-oriented truncation, robustness, and recurrent limitations

A fifth usage modifies the output distribution so that it is explicitly closer to a one-hot vector. In “\(\epsilon\)-Softmax: Approximating One-Hot Vectors for Mitigating Label Noise,” the mapping
\[
p^{(\epsilon)} = g_\epsilon(p)=\frac{m}{m+1}\,\mathbf e_t+\frac{1}{m+1}\,p,
\qquad t=\arg\max_j p_j,
\]
adds \(m\) to the top probability and renormalizes by \(m+1\) [2508.02387]. The paper proves the bound
\[
\epsilon(m)=\frac{\sqrt{1-\frac{1}{K}}}{m+1}
\]
on the \(L_2\) deviation from a one-hot vector, derives the modified cross-entropy
\[
L_{\mathrm{CE}_\epsilon}(y,p)=-\log\left(\frac{p_y+m\mathbf 1\{t=y\}}{m+1}\right),
\]
and shows that the gradient equals the usual \(p-\mathbf e_y\) when \(t\neq y\), but is attenuated by \(p_y/(p_y+m)\) when \(t=y\) [2508.02387]. This acts as soft early stopping on already-correct examples and is analyzed as a noise-tolerant relaxation for almost any loss satisfying the paper’s continuity condition. The reported empirical gains include CIFAR-10 with \(0.8\) symmetric noise, where \(\mathrm{CE}_\epsilon+\mathrm{MAE}\) achieves \(58.96\%\) versus \(18.95\%\) for standard CE, and Clothing1M, where it achieves \(69.85\%\) versus \(67.38\%\) for CE [2508.02387].

Viewed together, the literature dispels several recurring misconceptions. Softmax truncation is not a single standardized method; it is a label applied to inference-time comparator replacement, approximate exponentials, finite-support relaxations, class-set sparsification, sampled normalization, and one-hot-oriented output reshaping [2201.04562, 2501.13379, 2003.01847, 2508.03175, 2508.02387]. Exact top-1 preservation holds only for the comparator-based inference setting, not for sampled or approximate training procedures [2201.04562, 1907.10747]. Extremely accurate approximation of \(e^x\) does not by itself specify how the denominator, reciprocal, or numerical stabilization are handled; in the FPGA-motivated study, these stages are not approximated or implemented in hardware [2501.13379]. Likewise, sampled softmax is not unbiased merely because logits are importance-corrected; unbiasedness requires sampling from the exact softmax distribution, which is itself expensive [1907.10747].

A plausible implication is that method selection must be keyed to the role softmax plays in the target system. If only top-1 or top-\(k\) indices are needed at inference, comparator truncation is exact and architecturally minimal [2201.04562]. If bounded-domain numerical approximation of \(e^x\) is the bottleneck, Taylor order-3 or quadratic LUT interpolation offers an explicit accuracy-speed trade-off [2501.13379]. If the task involves arbitrary discrete latent variables, support truncation plus Gumbel-Softmax supplies a pathwise estimator [2003.01847]. If the challenge is very large output spaces or misalignment between cross-entropy and the argmax test objective, the relevant formulations are sampled softmax, MIDX, sparse-softmax, or AS-Softmax rather than polynomial approximation of the exponential [2112.12433, 2508.03175, 1907.10747, 2501.08563].

Source: https://www.emergentmind.com/topics/softmax-truncation