---
title: Evidential Softmax (ev-softmax)
url: https://www.emergentmind.com/topics/evidential-softmax-ev-softmax
type: topic
---

# Evidential Softmax (ev-softmax)

Evidential Softmax (ev-softmax) refers to a class of normalization functions for neural network outputs that combine explicit modeling of epistemic uncertainty with properties of sparsity and multimodality. Unlike conventional softmax or sparsemax, ev-softmax is designed to (1) produce probability vectors that are both sparse and support multiple modes, (2) provide a tractable, closed-form backpropagation mechanism compatible with standard log-likelihood or KL-divergence losses, and (3) mitigate overconfidence and codebook collapse in discrete latent models such as VAEs and vector quantized architectures. There exist two principal lines of ev-softmax: one operating directly as a sparse normalization function for logits [2110.14182], and another, distributional, that arises from the Dirichlet–categorical construction for uncertainty-calibrated discrete representations [2310.05718].

## 1. Mathematical Formulation and Variants

Two mathematically distinct but related formulations are presented in the literature:

### 1.1 Direct Sparse Normalization
Given input logits $v = (v_1, ..., v_K) \in \mathbb{R}^K$, define the arithmetic mean $\bar v = \frac{1}{K} \sum_{i=1}^K v_i$. Ev-softmax assigns nonzero probability only to entries above the mean:
$$
EvSoftmax(v)_k = \frac{1\{v_k \geq \bar v\} \exp(v_k)}{\sum_{j=1}^K 1\{v_j \geq \bar v\} \exp(v_j)}, \quad k=1,...,K,
$$
where $1\{\cdot\}$ is the indicator function. This induces exact zeros for sub-mean logits while preserving exponential weighting among the active set [2110.14182]. To facilitate training with standard probabilistic losses, the "full-support" relaxation introduces a small $\epsilon>0$:
$$
EvSoftmax_{\mathrm{train},\epsilon}(v)_k = 
\frac{(1\{v_k \geq \bar v\} + \epsilon) \exp(v_k)}
{\sum_{j=1}^K (1\{v_j \geq \bar v\} + \epsilon) \exp(v_j)}
$$

### 1.2 Evidential Dirichlet–Categorical Construction
Given encoder outputs $l \in \mathbb{R}^K$ (interpreted as evidence), compute Dirichlet concentration parameters:
$$
\alpha = \exp(l) + 1
$$
The mean class probabilities under Dirichlet$(\alpha)$ are
$$
p_i = \mathbb{E}[\pi_i] = \frac{\alpha_i}{\sum_j \alpha_j}
$$
This "ev-softmax" replaces the conventional softmax but additionally attaches a Dirichlet KL regularizer, penalizing deviations from a uniform prior and thus discouraging overconfident, spiky assignments [2310.05718].

## 2. Properties and Theoretical Characteristics

### 2.1 Sparsity and Multimodality
Ev-softmax explicitly zeros out all actions below the mean, guaranteeing sparse distributions. Unlike sparsemax and entmax, which may collapse multimodal supports into a single dominant mode, ev-softmax preserves all above-mean modes, ensuring interpretability and propagation of multiple hypotheses [2110.14182].

### 2.2 Differentiability and Gradients
On the active support ($v_i \geq \bar v$), the gradient of ev-softmax mirrors that of softmax restricted to the subset:
$$
\frac{\partial\,EvSoftmax(v)_i}{\partial\,v_j} = p_i (\delta_{ij} - p_j) \quad \text{if } v_i, v_j \geq \bar v,\quad 0 \text{ otherwise}
$$
where $\delta_{ij}$ is the Kronecker delta. Under the $\epsilon$-relaxation, gradients are defined everywhere, so ev-softmax can be used seamlessly with backpropagation-based training regimes [2110.14182].

### 2.3 Uncertainty and Regularization
The Dirichlet–categorical variant incorporates a KL penalty:
$$
KL\left[\text{Dir}(\pi | \alpha) \,\|\, \text{Dir}(\pi | 1)\right]
$$
This attracts the concentration vector $\alpha$ toward the uniform Dirichlet when the encoder provides weak evidence, penalizing overconfidence and reducing the risk of degeneracy in the representation (codebook collapse) [2310.05718].

## 3. Training Methodologies and Implementation Considerations

### 3.1 Algorithmic Workflow

The following describes a typical training loop for evidential VAEs:

```python
initialize encoder Eθ, decoder Dϕ, codebook M, temperature τ←1.0, β schedule
for t = 1…T do
  x ← sample minibatch
  l ← Eθ(x)                           # raw logits (evidence)
  e ← exp(clamp(l, max=20))          # for numeric stability
  α ← e + 1
  π ∼ Dirichlet(α)
  z_soft ∼ RelaxedOneHotCategorical(probs=π, temp=τ)
  x̂ ← Dϕ(M, z_soft)
  recon_loss = ‖x – x̂‖²
  kl_dir = KL_Dirichlet(α ‖ 1)
  L = recon_loss + β(t)·kl_dir
  backpropagate L w.r.t. θ, ϕ, M
  τ ← anneal_temperature(t)
  β ← anneal_beta(t)
end for
```
Key details include clamping logits before exponentiation, annealing the Gumbel–Softmax temperature and regularization weight, and using the relaxed categorical in training for differentiability. Standard log-likelihood and KL losses are directly applicable when using the continuous relaxation of the direct ev-softmax [2310.05718][2110.14182].

### 3.2 Practical Recommendations

- Use $\epsilon \approx 10^{-6}$ for the continuous relaxation of direct ev-softmax during training.
- Apply layer normalization or calibration to input logits to control thresholding behavior imposed by mean-subtraction.
- Employ Adam optimizer (lr $= 1\mathrm{e}{-3}$), moderate batch sizes, and numerically robust codebook parameterizations as in standard discrete VAE architectures [2310.05718].
- Clamp logit values pre-exponentiation to prevent overflow.

## 4. Comparison to Other Normalization Functions

| Normalizer   | Support       | Sparsity   | Multimodality | Special Loss Required   |
|--------------|--------------|------------|---------------|------------------------|
| Softmax      | Full          | No         | Yes           | No                     |
| Sparsemax    | Subset        | Yes        | No (collapse) | Yes (hinge/Poisson)    |
| Entmax$_\alpha$ | Interpolated | Yes        | No (collapse) | Yes ($\alpha$-entmax)  |
| Ev-softmax   | Subset        | Yes        | Yes           | No (with $\epsilon$)   |

Ev-softmax uniquely combines support for multimodality and strict sparsity with standard log-likelihood/KL compatibility when using its continuous relaxation. Empirically, it reduces dimensionality of the distribution while maintaining high distributional accuracy and balancing focus and context in attention mechanisms [2110.14182].

## 5. Empirical Performance and Use Cases

### 5.1 Deep Generative Models

- **Conditional VAE on MNIST:** Ev-softmax learned exactly five nonzero modes per class (matching the even/odd structure), outperforming softmax, sparsemax, and entmax in terms of Wasserstein distance to the true prior.
- **VQ-VAE + PixelCNN on tinyImageNet:** Ev-softmax achieved highest top-5/top-10 accuracy with 85% sparsity (using ~77 out of 512 codes on average), surpassing other sparse normalization strategies [2110.14182].

### 5.2 Attention and Sequence Models

- **Transformer NMT (IWSLT’14 EN→DE):** With ev-softmax self-attention, models achieved the highest BLEU (29.4), best ROUGE/METEOR, and a balanced attention focus—attending to $\sim$8 source words on average (vs. all for softmax, 2 for sparsemax, 4 for entmax) [2110.14182].

### 5.3 Evidential Discrete Representation Learning

- **EdVAE discrete VAE:** The evidential layer replaces softmax, prevents codebook collapse, improves reconstruction, and enhances codebook usage compared to dVAE and VQ-VAE. The KL regularizer encourages the network away from overconfident assignments, leading to richer latent usage and more robust representations [2310.05718].

### 5.4 Semi-supervised Learning

- **Semi-supervised VAE on MNIST:** 97.3% classification accuracy with only 1.64 average active classes per prediction (84% sparsity). Competing sparse normalization schemes (e.g., sparsemax/entmax) had reduced accuracy or mode collapse [2110.14182].

## 6. Limitations, Practical Considerations, and Extensions

While ev-softmax is scale- and translation-invariant due to mean-subtraction, calibration of logits is necessary to control thresholding effects. Exact zeros (and the induced kinks at $v_k = \bar v$) are a natural consequence of the hard support, though empirical results indicate these are not a barrier to convergence. The KL regularizer in evidential constructions (Dirichlet–categorical) must be carefully scheduled (e.g., ramp up $\beta$ slowly), and numerical stability precautions (clamping, exponentiation limits) are essential. A plausible implication is that further hybridizations—combining evidential uncertainty with task-specific structured sparsity—may yield new regimes of interpretable and robust discrete modeling in dense and sequence architectures [2310.05718][2110.14182].

Source: https://www.emergentmind.com/topics/evidential-softmax-ev-softmax