Global Softmax Weighting Overview

• Global softmax weighting is the method of applying a shared normalization to competing scores, coupling outputs into a probability simplex.
• It underpins diverse systems like self-attention, non-local vision modules, reinforcement learning backups, and mixture-of-experts, ensuring calibrated score distributions.
• Adjustments such as temperature tuning and softmax-free variants improve model stability, efficiency, and overall performance across various applications.

Searching arXiv for recent and foundational papers on global softmax weighting across attention, softmax dynamics, RL, and mixture models. Global softmax weighting denotes the use of a softmax normalizer to convert a set of competing scores into a globally normalized distribution. In its standard form, every weight depends on a shared denominator that sums over all competitors, so the weights are mutually coupled rather than independently scaled. This pattern appears in self-attention and non-local vision modules, in classification heads, in reinforcement-learning backups, in mixture-of-experts gating, in ranking losses, and in more formal settings involving contextual event structures. Across these domains, the central technical question is the same: what is gained, and what is distorted, by forcing scores into a probability simplex through a single global normalizer (Cho et al., 2022, Masarczyk et al., 2 Jun 2025, Asadi et al., 2016).

1. Canonical definition and shared mathematical structure

In attention, global softmax weighting is the row-wise normalization of all query–key similarity scores. For queries $Q \in \mathbb{R}^{n \times d}$, keys $K \in \mathbb{R}^{n \times d}$, and values $V \in \mathbb{R}^{n \times d_v}$, the standard scaled dot-product form is

$$\mathrm{Att}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$

so that for each query $i$,

$$A_{ij}=\frac{\exp(\langle Q_i,K_j\rangle/\sqrt{d})}{\sum_{t=1}^n \exp(\langle Q_i,K_t\rangle/\sqrt{d})}, \qquad O_i=\sum_{j=1}^n A_{ij}V_j.$$

The denominator enforces a global competition over all key positions for each query. The same structure appears in the non-local block, where $$A_{i,j}=\frac{1}{Z(\boldsymbol{x})}f(\boldsymbol{x_i},\boldsymbol{x_j})$$ and $y_i=\sum_{\forall j}A_{i,j}g(\boldsymbol{x_j})$, with the common choice $$f(\boldsymbol{x_i},\boldsymbol{x_j})=\exp\!\big(\frac{1}{\sqrt{C}}\theta(\boldsymbol{x_i})\cdot\phi(\boldsymbol{x_j})\big)$$ and $$Z(\boldsymbol{x})=\sum_{\forall j}f(\boldsymbol{x_i},\boldsymbol{x_j})$$ (Cho et al., 2022).

In classification, the same mechanism acts over class logits. For logits $K \in \mathbb{R}^{n \times d}$0 and temperature $K \in \mathbb{R}^{n \times d}$1,

$K \in \mathbb{R}^{n \times d}$2

The temperature governs the sharpness of global competition, while the norm of the logits induces an “effective temperature”: reducing the logit norm increases the effective temperature seen by softmax. Under cross-entropy with one-hot target $K \in \mathbb{R}^{n \times d}$3, the logit gradient is $K \in \mathbb{R}^{n \times d}$4, so larger $K \in \mathbb{R}^{n \times d}$5 both flattens the output distribution and reduces gradient magnitudes (Masarczyk et al., 2 Jun 2025).

In reinforcement learning, global softmax weighting appears as a backup over all actions at a state:

$K \in \mathbb{R}^{n \times d}$6

As $K \in \mathbb{R}^{n \times d}$7, the operator approaches the mean over actions; as $K \in \mathbb{R}^{n \times d}$8, it approaches the max. In mixture-of-experts, the same pattern is used to distribute mass over experts, typically with affine gating scores $K \in \mathbb{R}^{n \times d}$9 and weights $V \in \mathbb{R}^{n \times d_v}$0 (Asadi et al., 2016, Nguyen et al., 5 Mar 2025).

2. Non-local vision modules: standard practice and geometric failure modes

The most direct analysis of global softmax weighting in vision appears in lightweight non-local neural networks. For an input feature tensor $V \in \mathbb{R}^{n \times d_v}$1, query, key, and value embeddings are formed by learned linear maps $V \in \mathbb{R}^{n \times d_v}$2, $V \in \mathbb{R}^{n \times d_v}$3, and $V \in \mathbb{R}^{n \times d_v}$4. The standard non-local block applies a softmax over all $V \in \mathbb{R}^{n \times d_v}$5 key positions per query, producing a probability distribution over the entire spatial grid. This yields an attention map $V \in \mathbb{R}^{n \times d_v}$6 and output $V \in \mathbb{R}^{n \times d_v}$7 (Cho et al., 2022).

The geometric analysis decomposes the dot product as

$V \in \mathbb{R}^{n \times d_v}$8

The central diagnosis is that softmax normalization makes attention highly reliant on the magnitudes of key vectors. Two arguments are emphasized. First, if both $V \in \mathbb{R}^{n \times d_v}$9 and $$\mathrm{Att}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$0 are unit-norm and softmax is applied, then for a single key vector to simultaneously have minimum attention with more than two queries, those queries must share the same direction, reducing angular variability for near-zero attention. Second, without softmax, orthogonality already drives attention to zero: all queries in the $$\mathrm{Att}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$1-dimensional hyperplane orthogonal to a fixed key achieve zero dot product, preserving much richer angular structure (Cho et al., 2022).

The empirical evidence is correspondingly geometric. Attention maps for standard non-local blocks show distinct vertical lines, interpreted as keys dominating regardless of query. By contrast, attention maps for the softmax-free scaled non-local block do not show vertical lines, suggesting dependence on both queries and keys. Magnitude-only and direction-only ablations sharpen the point. On CIFAR-10 with PreResNet56+3NL, the test errors averaged over 10 seeds were Base 5.73, Mag 5.83, and Dir 6.01, whereas PreResNet56+3Ours yielded Base 5.64, Mag 5.76, and Dir 5.67. On CIFAR-100, PreResNet56+3NL gave Base 25.12, Mag 25.26, and Dir 25.44, while PreResNet56+3Ours gave Base 24.53, Mag 25.20, and Dir 24.68. Direction-only attention therefore degrades substantially under standard global softmax weighting but remains close to baseline in the scaled non-local block (Cho et al., 2022).

The proposed replacement removes softmax and scales the output by $$\mathrm{Att}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$2 in addition to the usual $$\mathrm{Att}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$3 dot-product scaling:

$$\mathrm{Att}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$4

The $$\mathrm{Att}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$5 factor is justified by a variance argument and was observed to prevent divergence. Because the computation can be reordered associatively, the method avoids explicitly forming the $$\mathrm{Att}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$6 attention matrix and yields complexity linear in $$\mathrm{Att}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$7. In multi-head form, the complexity becomes $$\mathrm{Att}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$8, which does not increase with the number of heads and decreases as $$\mathrm{Att}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,$$9 grows for fixed $i$0 (Cho et al., 2022).

The reported gains are not merely asymptotic. With 3 blocks and 4 heads, CIFAR-10 error for PreResNet110 drops from 5.29 with NL to 4.93 with the scaled variant; CIFAR-100 error for WRN-28-10 drops from 18.51 to 18.18; Tiny-ImageNet error for PreResNet50 drops from 34.76 to 34.385. Memory and train step time remain essentially unchanged as heads increase in the scaled model, whereas standard softmax NL grows from 3789 MB to 5229 MB and from 83.11 ms/batch to 100.93 ms/batch when moving from 1 to 4 heads (Cho et al., 2022).

3. Temperature, norm, and representation rank

A broader theory of global softmax weighting treats temperature as the principal control knob for representation geometry. The key claim is that softmax-based deep networks exhibit a rank-deficit bias: they converge to solutions whose pre-softmax logits matrix $i$1 has rank much lower than the number of classes, especially when the effective temperature is high, or equivalently when the logit norm is low. High temperature drives the predicted distribution toward uniformity, creating a symmetric loss regime with low gradient diversity; with fixed $i$2, networks must grow logit norms to escape that symmetry (Masarczyk et al., 2 Jun 2025).

The mechanism is not described purely in terms of entropy. Two norm-growth routes are identified. One is singular value scaling, which increases dominant singular values in $i$3 and $i$4. The other is singular vector alignment across layers, so that top singular directions compound through depth. Early training under high $i$5 exhibits strong alignment of top singular vectors across layers while other directions remain nearly orthogonal. This asymmetry yields compression and collapse simultaneously: effective depth $i$6 decreases, but solution rank $i$7 also decreases, and out-of-distribution generalization at the final layer worsens (Masarczyk et al., 2 Jun 2025).

The same work proves that softmax can amplify rank after normalization even if pre-softmax rank does not change. It also gives a constructive result that rank-2 pre-softmax matrices can yield full-rank post-softmax outputs for sufficiently large scaling. A separate gradient-rank bound shows that low-rank representations bound gradient rank from above, reducing gradient diversity and reinforcing collapse. In this sense, global softmax weighting is not a passive readout; it feeds back into the optimization geometry (Masarczyk et al., 2 Jun 2025).

The empirical results are architecture-wide. On CIFAR-100 with ResNet-34, lowering the logits norm or raising the effective temperature changes $i$8 from 99 to 72, $i$9 from 100% to 68%, and the OOD generalization loss $$A_{ij}=\frac{\exp(\langle Q_i,K_j\rangle/\sqrt{d})}{\sum_{t=1}^n \exp(\langle Q_i,K_t\rangle/\sqrt{d})}, \qquad O_i=\sum_{j=1}^n A_{ij}V_j.$$0 from 16% to 51%. On ImageNet-1k with ResNet-34, $$A_{ij}=\frac{\exp(\langle Q_i,K_j\rangle/\sqrt{d})}{\sum_{t=1}^n \exp(\langle Q_i,K_t\rangle/\sqrt{d})}, \qquad O_i=\sum_{j=1}^n A_{ij}V_j.$$1 falls from 512 to 122, $$A_{ij}=\frac{\exp(\langle Q_i,K_j\rangle/\sqrt{d})}{\sum_{t=1}^n \exp(\langle Q_i,K_t\rangle/\sqrt{d})}, \qquad O_i=\sum_{j=1}^n A_{ij}V_j.$$2 from 100% to 82%, and $$A_{ij}=\frac{\exp(\langle Q_i,K_j\rangle/\sqrt{d})}{\sum_{t=1}^n \exp(\langle Q_i,K_t\rangle/\sqrt{d})}, \qquad O_i=\sum_{j=1}^n A_{ij}V_j.$$3 rises from 5% to 23%. VGG-19 is described as having inherently low initial logits norm and already collapsing at $$A_{ij}=\frac{\exp(\langle Q_i,K_j\rangle/\sqrt{d})}{\sum_{t=1}^n \exp(\langle Q_i,K_t\rangle/\sqrt{d})}, \qquad O_i=\sum_{j=1}^n A_{ij}V_j.$$4, whereas ResNets start higher and tolerate $$A_{ij}=\frac{\exp(\langle Q_i,K_j\rangle/\sqrt{d})}{\sum_{t=1}^n \exp(\langle Q_i,K_t\rangle/\sqrt{d})}, \qquad O_i=\sum_{j=1}^n A_{ij}V_j.$$5 (Masarczyk et al., 2 Jun 2025).

Attention inherits the same temperature logic. With attention weights

$$A_{ij}=\frac{\exp(\langle Q_i,K_j\rangle/\sqrt{d})}{\sum_{t=1}^n \exp(\langle Q_i,K_t\rangle/\sqrt{d})}, \qquad O_i=\sum_{j=1}^n A_{ij}V_j.$$6

low $$A_{ij}=\frac{\exp(\langle Q_i,K_j\rangle/\sqrt{d})}{\sum_{t=1}^n \exp(\langle Q_i,K_t\rangle/\sqrt{d})}, \qquad O_i=\sum_{j=1}^n A_{ij}V_j.$$7 yields spiky, winner-takes-all attention and can starve non-winning tokens of gradient signal, while high $$A_{ij}=\frac{\exp(\langle Q_i,K_j\rangle/\sqrt{d})}{\sum_{t=1}^n \exp(\langle Q_i,K_t\rangle/\sqrt{d})}, \qquad O_i=\sum_{j=1}^n A_{ij}V_j.$$8 yields flatter attention, slow optimization under loss symmetry, and downstream rank collapse. The practical guidance is therefore explicitly two-sided: avoid very low temperature and very high temperature, and control effective temperature through scaling, normalization, and width rather than through temperature alone (Masarczyk et al., 2 Jun 2025).

The same temperature control also mediates an OOD trade-off. High $$A_{ij}=\frac{\exp(\langle Q_i,K_j\rangle/\sqrt{d})}{\sum_{t=1}^n \exp(\langle Q_i,K_t\rangle/\sqrt{d})}, \qquad O_i=\sum_{j=1}^n A_{ij}V_j.$$9 reduces the orthogonality-deviation statistic and improves OOD detection via NECO, but worsens OOD generalization at the final layer. On CIFAR-100$$A_{i,j}=\frac{1}{Z(\boldsymbol{x})}f(\boldsymbol{x_i},\boldsymbol{x_j})$$0CIFAR-10 with ResNet-18, AUROC improves from 63.66% to 77.04% and FPR@95 drops from 90.44% to 80.13%; on CIFAR-10$$A_{i,j}=\frac{1}{Z(\boldsymbol{x})}f(\boldsymbol{x_i},\boldsymbol{x_j})$$1CIFAR-100, AUROC improves from 77.75% to 87.53% and FPR@95 drops from 73.29% to 55.87% (Masarczyk et al., 2 Jun 2025).

4. Efficient and softmax-free attention variants

Several recent attention mechanisms can be read as attempts either to remove global softmax weighting or to reintroduce selected parts of it without paying quadratic cost. One line keeps row-wise normalization but replaces the exponential kernel. In cosFormer, the two properties of softmax attention singled out as essential are the non-negativeness of the attention matrix and a non-linear re-weighting scheme that can concentrate the distribution. Queries and keys are mapped with $$A_{i,j}=\frac{1}{Z(\boldsymbol{x})}f(\boldsymbol{x_i},\boldsymbol{x_j})$$2, and a cosine-based positional re-weighting

$$A_{i,j}=\frac{1}{Z(\boldsymbol{x})}f(\boldsymbol{x_i},\boldsymbol{x_j})$$3

is multiplied into the dot product. The resulting similarity remains non-negative and admits an exact linear decomposition via $$A_{i,j}=\frac{1}{Z(\boldsymbol{x})}f(\boldsymbol{x_i},\boldsymbol{x_j})$$4, so the model achieves $$A_{i,j}=\frac{1}{Z(\boldsymbol{x})}f(\boldsymbol{x_i},\boldsymbol{x_j})$$5 rather than $$A_{i,j}=\frac{1}{Z(\boldsymbol{x})}f(\boldsymbol{x_i},\boldsymbol{x_j})$$6 complexity while preserving explicit row-wise normalization (Qin et al., 2022).

The empirical motivation for those two properties is direct. In the paper’s analysis of softmax properties, a ReLU feature map outperforms identity and LeakyReLU in linear attention: validation loss 1.993 for ReLU versus 2.343 for identity, with QQP 88.86 versus 84.23, SST-2 89.90 versus 76.26, and MNLI 77.86 versus 58.27. On WikiText-103 autoregressive language modeling, cosFormer reports ppl(val)=23.5 and ppl(test)=23.1, compared with 24.5 and 26.2 for the vanilla Transformer. On Long-Range Arena, it achieves the best overall average of 55.23, with state-of-the-art scores on ListOps and Retrieval (Qin et al., 2022).

A different strategy keeps global competition but lifts it from tokens to heads. Softmax Linear Attention restores competition at the head level through dual head-softmax gates $$A_{i,j}=\frac{1}{Z(\boldsymbol{x})}f(\boldsymbol{x_i},\boldsymbol{x_j})$$7 and $$A_{i,j}=\frac{1}{Z(\boldsymbol{x})}f(\boldsymbol{x_i},\boldsymbol{x_j})$$8, where $$A_{i,j}=\frac{1}{Z(\boldsymbol{x})}f(\boldsymbol{x_i},\boldsymbol{x_j})$$9 and $y_i=\sum_{\forall j}A_{i,j}g(\boldsymbol{x_j})$0. The gates act as read competition and write competition over heads conceived as coarse semantic slots. The resulting mechanism preserves linear-time recurrence because the gates are per-token and do not require token-to-token coupling (Xu et al., 2 Feb 2026).

The theory attached to this design emphasizes restored magnitude sensitivity and asymptotic winner-take-all consensus. As gate logits are scaled, the entropy of the head distribution decreases, and in the large-scale limit information flows only when query and key agree on the same head. The reported improvements are strongest in retrieval-heavy settings. On WikiText, perplexity improves from 28.93 to 26.32 for GLA and from 29.58 to 27.79 for RetNet. On real-world retrieval tasks, the average score rises from 19.17 to 26.88 for GLA and from 19.96 to 24.41 for RetNet. On RULER S-NIAH-1 at length 2K, GLA rises from 44.8 to 85.2 and RetNet from 27.8 to 71.2 (Xu et al., 2 Feb 2026).

Taken together, these variants separate different functions that standard global softmax weighting performs simultaneously. Some retain normalization but replace the exponential. Some eliminate token-level global normalization and recover selective competition over a smaller dimension such as heads. Others, such as the scaled non-local block, discard the simplex constraint altogether and rely on variance control and associative computation instead (Qin et al., 2022, Xu et al., 2 Feb 2026).

5. Sequential decision making and expert mixtures

In sequential decision making, the distinction between smooth weighting and stable operators is explicit. The standard Boltzmann softmax backup

$y_i=\sum_{\forall j}A_{i,j}g(\boldsymbol{x_j})$1

interpolates between mean and max, but it is not a non-expansion in the $y_i=\sum_{\forall j}A_{i,j}g(\boldsymbol{x_j})$2 norm. As a result, generalized value iteration with this operator is not guaranteed to be a contraction and can exhibit multiple fixed points, slow update trajectories, or divergence. A two-state, two-action counterexample with discount $y_i=\sum_{\forall j}A_{i,j}g(\boldsymbol{x_j})$3 and $y_i=\sum_{\forall j}A_{i,j}g(\boldsymbol{x_j})$4 shows oscillatory SARSA estimates that fail to converge (Asadi et al., 2016).

The proposed alternative is mellowmax,

$y_i=\sum_{\forall j}A_{i,j}g(\boldsymbol{x_j})$5

which remains differentiable, interpolates between the mean and the max, and crucially is a non-expansion in $y_i=\sum_{\forall j}A_{i,j}g(\boldsymbol{x_j})$6. The corresponding Bellman-style operator is therefore a $y_i=\sum_{\forall j}A_{i,j}g(\boldsymbol{x_j})$7-contraction. In experiments on 200 random MDPs, generalized value iteration termination failures were 8/200 for Boltzmann and 0/200 for mellowmax; multiple fixed points were detected 3/200 times for Boltzmann and 0/200 for mellowmax; average iterations to terminate were approximately 231.65 for Boltzmann and 201.32 for mellowmax (Asadi et al., 2016).

Mixture-of-experts introduces a different issue: the statistical cost of softmax gating. For the standard regression function

$y_i=\sum_{\forall j}A_{i,j}g(\boldsymbol{x_j})$8

the paper on convergence rates shows that parameter estimation is favorable only when the expert class satisfies strong identifiability. Two-layer FFNs with GELU, sigmoid, or tanh satisfy this condition, whereas linear experts do not. Under strong identifiability, exactly specified parameters converge at $y_i=\sum_{\forall j}A_{i,j}g(\boldsymbol{x_j})$9 and over-specified ones at $$f(\boldsymbol{x_i},\boldsymbol{x_j})=\exp\!\big(\frac{1}{\sqrt{C}}\theta(\boldsymbol{x_i})\cdot\phi(\boldsymbol{x_j})\big)$$0. For linear experts, parameter and expert estimation can be slower than any polynomial order, with potential $$f(\boldsymbol{x_i},\boldsymbol{x_j})=\exp\!\big(\frac{1}{\sqrt{C}}\theta(\boldsymbol{x_i})\cdot\phi(\boldsymbol{x_j})\big)$$1 behavior and exponential sample complexity in $$f(\boldsymbol{x_i},\boldsymbol{x_j})=\exp\!\big(\frac{1}{\sqrt{C}}\theta(\boldsymbol{x_i})\cdot\phi(\boldsymbol{x_j})\big)$$2 (Nguyen et al., 5 Mar 2025).

The same analysis finds that dense-to-sparse temperature gating with a linear router is statistically detrimental because of an intrinsic PDE interaction between $$f(\boldsymbol{x_i},\boldsymbol{x_j})=\exp\!\big(\frac{1}{\sqrt{C}}\theta(\boldsymbol{x_i})\cdot\phi(\boldsymbol{x_j})\big)$$3 and the router parameters. Replacing the linear router by a general router $$f(\boldsymbol{x_i},\boldsymbol{x_j})=\exp\!\big(\frac{1}{\sqrt{C}}\theta(\boldsymbol{x_i})\cdot\phi(\boldsymbol{x_j})\big)$$4 that is algebraically independent of the expert restores the fast rates. Hierarchical softmax gating retains polynomial rates with strongly identifiable experts but again becomes statistically unfavorable with linear experts because cross-level parameter interactions slow all relevant rates (Nguyen et al., 5 Mar 2025).

A third mixture setting uses global softmax weighting at the level of expert mixture weights rather than per-sample gating. In AMF–VI–sEMA, Stage 1 trains heterogeneous flow experts independently, and Stage 2 freezes them and updates only global weights by a temperature softmax of average log-likelihoods or averaged responsibilities, followed by simplex Exponential Moving Average:

$$f(\boldsymbol{x_i},\boldsymbol{x_j})=\exp\!\big(\frac{1}{\sqrt{C}}\theta(\boldsymbol{x_i})\cdot\phi(\boldsymbol{x_j})\big)$$5

With defaults $$f(\boldsymbol{x_i},\boldsymbol{x_j})=\exp\!\big(\frac{1}{\sqrt{C}}\theta(\boldsymbol{x_i})\cdot\phi(\boldsymbol{x_j})\big)$$6, the method maintains $$f(\boldsymbol{x_i},\boldsymbol{x_j})=\exp\!\big(\frac{1}{\sqrt{C}}\theta(\boldsymbol{x_i})\cdot\phi(\boldsymbol{x_j})\big)$$7 on all datasets and reports consistent NLL improvements over AMF–VI, such as Rings 2.218 versus 2.350 and Two-moons 1.012 versus 1.074. The design is explicitly data-agnostic at inference time: there is no per-sample gating and no backpropagation through the weights (Wiriyapong et al., 4 Jul 2026).

6. Ranking, classification, and alternative output normalizers

In ranking, global softmax weighting appears as listwise comparison against the full candidate set. For a context $$f(\boldsymbol{x_i},\boldsymbol{x_j})=\exp\!\big(\frac{1}{\sqrt{C}}\theta(\boldsymbol{x_i})\cdot\phi(\boldsymbol{x_j})\big)$$8 with candidates $$f(\boldsymbol{x_i},\boldsymbol{x_j})=\exp\!\big(\frac{1}{\sqrt{C}}\theta(\boldsymbol{x_i})\cdot\phi(\boldsymbol{x_j})\big)$$9 and scores $$Z(\boldsymbol{x})=\sum_{\forall j}f(\boldsymbol{x_i},\boldsymbol{x_j})$$0, the listwise softmax probabilities are $$Z(\boldsymbol{x})=\sum_{\forall j}f(\boldsymbol{x_i},\boldsymbol{x_j})$$1, and the standard listwise loss for positive label $$Z(\boldsymbol{x})=\sum_{\forall j}f(\boldsymbol{x_i},\boldsymbol{x_j})$$2 is $$Z(\boldsymbol{x})=\sum_{\forall j}f(\boldsymbol{x_i},\boldsymbol{x_j})$$3. The global denominator induces globally coupled gradients and Hessian entries $$Z(\boldsymbol{x})=\sum_{\forall j}f(\boldsymbol{x_i},\boldsymbol{x_j})$$4 and $$Z(\boldsymbol{x})=\sum_{\forall j}f(\boldsymbol{x_i},\boldsymbol{x_j})$$5. A second-order Taylor expansion around zero logits produces the RG$$Z(\boldsymbol{x})=\sum_{\forall j}f(\boldsymbol{x_i},\boldsymbol{x_j})$$6 and RG$$Z(\boldsymbol{x})=\sum_{\forall j}f(\boldsymbol{x_i},\boldsymbol{x_j})$$7 losses, which replace the original softmax loss by tractable quadratic forms while preserving DCG consistency. The accompanying ALS optimization has per-iteration complexity $$Z(\boldsymbol{x})=\sum_{\forall j}f(\boldsymbol{x_i},\boldsymbol{x_j})$$8 and converges linearly under the stated conditions (Pu et al., 11 Jun 2025).

In classification, a different reinterpretation of global softmax weighting is negative-focused. W-Softmax removes the bias in the last FC layer, enforces $$Z(\boldsymbol{x})=\sum_{\forall j}f(\boldsymbol{x_i},\boldsymbol{x_j})$$9, and for a sample of class $K \in \mathbb{R}^{n \times d}$00 replaces each negative class weight by

$K \in \mathbb{R}^{n \times d}$01

The positive weight remains $K \in \mathbb{R}^{n \times d}$02. This globally modifies all negatives in the denominator for the current sample, rotating them toward the positive direction and thereby enlarging the decision margin. The reported accuracies increase with $K \in \mathbb{R}^{n \times d}$03 on all tested datasets: on CIFAR-10, standard softmax gives 90.95%, while W-Softmax gives 92.47% for $K \in \mathbb{R}^{n \times d}$04, 92.84% for $K \in \mathbb{R}^{n \times d}$05, and 93.28% for $K \in \mathbb{R}^{n \times d}$06; on LFW face verification, the corresponding numbers are 96.53%, 97.98%, 98.86%, and 98.91% (Li et al., 2019).

The same classification paper also proves a dimensionality result for the last FC layer: for $K \in \mathbb{R}^{n \times d}$07 classes, the minimum number of units that admits the optimal equiangular configuration is $K \in \mathbb{R}^{n \times d}$08. The construction yields pairwise inner products $K \in \mathbb{R}^{n \times d}$09 for $K \in \mathbb{R}^{n \times d}$10, and experiments are reported to show that accuracy saturates around this minimum while larger $K \in \mathbb{R}^{n \times d}$11 increases parameter memory and training time (Li et al., 2019).

A more direct study of output normalization examines alternatives to the exponential. The spherical loss family uses only the sum of logits $K \in \mathbb{R}^{n \times d}$12, the sum of squares $K \in \mathbb{R}^{n \times d}$13, and the target logit $K \in \mathbb{R}^{n \times d}$14. This permits exact output-layer updates in $K \in \mathbb{R}^{n \times d}$15 time per example, independent of output size, by maintaining global summaries of the output weights. Two concrete normalizers are the log-Spherical Softmax,

$K \in \mathbb{R}^{n \times d}$16

and the log-Taylor Softmax,

$K \in \mathbb{R}^{n \times d}$17

The empirical picture is mixed: on MNIST, log-Taylor Softmax reaches error 0.688% versus 0.716% for log-softmax on the official split; on CIFAR-10, log-Taylor Softmax gives error 8.07% versus 8.52% for log-softmax; but on Penn Treebank language modeling, log-softmax achieves perplexity 126.7 versus 147.2 for log-Taylor Softmax (Brébisson et al., 2015).

A separate optimization paper reformulates softmax itself as a regression target. With $K \in \mathbb{R}^{n \times d}$18 and $K \in \mathbb{R}^{n \times d}$19, the objective is

$K \in \mathbb{R}^{n \times d}$20

The Jacobian of softmax is $K \in \mathbb{R}^{n \times d}$21, the Hessian of the regularized objective is shown to be strongly convex under sufficient diagonal regularization, and an approximate Newton method with diagonal sparsification achieves linear convergence with per-iteration time $K \in \mathbb{R}^{n \times d}$22 (Deng et al., 2023).

7. Formal generalizations and hardware realization

In non-Boolean event structures, the term acquires a precise distinction between local normalization and global weight. A context-wise softmax can always normalize probabilities within each context, but it does not automatically define a single global probability weight on atoms shared across contexts. Single-valuedness, or no-disturbance, requires that if an atom $K \in \mathbb{R}^{n \times d}$23 appears in two contexts $K \in \mathbb{R}^{n \times d}$24 and $K \in \mathbb{R}^{n \times d}$25, then the assigned probabilities agree: $K \in \mathbb{R}^{n \times d}$26. In positive coordinates $K \in \mathbb{R}^{n \times d}$27, the gluing condition becomes

$K \in \mathbb{R}^{n \times d}$28

The representation theorem states that every strictly positive admissible weight $K \in \mathbb{R}^{n \times d}$29 admits a single-valued generalized softmax representation, so generalized softmax becomes a coordinate parametrization of the strictly positive part of the admissible-weight polytope. Boundary weights arise as limits, for example through $K \in \mathbb{R}^{n \times d}$30 under the exponential link (Svozil, 15 May 2026).

This formalization separates local normalization from broader structural questions. The paper explicitly states that “Exotic weights that exceed classical or quantum bounds are properties of the event structure and the chosen weight, not of the normalizing link.” In odd cyclic pasted logics, the boundary assignment $K \in \mathbb{R}^{n \times d}$31 and $K \in \mathbb{R}^{n \times d}$32 is admissible, violates the classical independence bound, and for $K \in \mathbb{R}^{n \times d}$33 exceeds the usual Lovász/KCBS quantum value; it is nevertheless obtainable as a limiting softmax coordinate system (Svozil, 15 May 2026).

At the other end of the spectrum, hardware implementations focus on preserving standard global softmax weighting as faithfully as possible under resource constraints. For FPGA-based ViT inference, the core operation remains the row-wise normalization over all tokens:

$K \in \mathbb{R}^{n \times d}$34

The proposed BRAM-free unit approximates only $K \in \mathbb{R}^{n \times d}$35 with a 16-segment piecewise-linear function over $K \in \mathbb{R}^{n \times d}$36, keeps the natural exponential rather than a base-2 substitute, and preserves the pre-trained attention temperature. Implemented on a Xilinx Zynq-7020, the complete attention-row core uses 1444 LUTs, 77 DSPs, and no BRAM; the weight unit alone uses 71 LUTs, 2 DSPs, and 0 BRAM. Hardware-accurate emulation reports accuracy within a $K \in \mathbb{R}^{n \times d}$37 absolute top-1 difference from the exact-softmax reference on ViT-family models (Usman et al., 2 Jul 2026).

These formal and hardware developments underscore a common theme. Global softmax weighting is not exhausted by the familiar “softmax over logits” formula. It is a design principle that joins local scores through a shared normalizer, and its consequences depend on what is being coupled: tokens, classes, actions, experts, contexts, or hardware arithmetic. The recent literature therefore treats it less as a default and more as a structured choice, to be retained when probabilistic normalization and calibrated competition are needed, and to be modified, lifted, or removed when geometry, stability, efficiency, or realizability make the simplex itself the limiting factor (Svozil, 15 May 2026, Usman et al., 2 Jul 2026).