Temperature-Scaled Activations in Deep Learning

• Temperature-scaled activations are a technique that applies a positive scalar to activation functions like softmax to modulate output confidence and entropy.
• They underpin methods for post-hoc calibration, adversarial robustness, and self-supervised learning by continuously controlling neural uncertainty.
• Adaptive variants, including per-sample and layer-wise scaling, enhance model performance under distribution shifts and adversarial attacks.

Temperature-scaled activations refer to the practice of inserting a positive scalar “temperature” parameter into the activation functions (most notably the softmax) of neural networks to modulate the confidence, uncertainty, and smoothness of the model’s output distribution. This intervention is foundational in post-hoc calibration, adversarial robustness, self-supervised representation learning, and uncertainty quantification. Modern research has established both the theoretical foundations and a diverse set of practical extensions, ranging from constant to input-adaptive temperature scaling.

1. Mathematical Definition and Core Properties

Given a vector of logits $z = (z_1, \dots, z_K) \in \mathbb{R}^K$, temperature scaling introduces a positive scalar parameter $T > 0$, defining a temperature-scaled softmax: $$p_i(T) = \frac{\exp(z_i/T)}{\sum_{j=1}^K \exp(z_j/T)}$$ In the inverse-temperature parameterization, $\beta = 1/T$, this becomes $$p_i(\beta) = \frac{\exp(\beta z_i)}{\sum_j \exp(\beta z_j)}$$ (Mattei et al., 16 Feb 2026).

Key behaviors:

• As $T \to 0^+$, $p(T)$ becomes increasingly peaked, converging to a point mass at the argmax of $z$.
• As $T \to \infty$, $p(T)$ becomes uniform over classes.
• For many other activation functions, temperature enters as a divisor in the pre-activation argument, e.g., $T > 0$0 (sigmoid) or $T > 0$1 (Tuna et al., 2022).

Entropy of the output distribution, $T > 0$2, increases strictly monotonically with $T > 0$3 as long as logits are not all tied. The derivative $T > 0$4; thus, temperature provides a continuous control over the model’s uncertainty (Mattei et al., 16 Feb 2026).

2. Theoretical Frameworks and Characterizations

Variational and Information-Geometric Perspective: Temperature scaling can be interpreted as the $T > 0$5-projection (information projection) of a given model $T > 0$6 onto the set of distributions with a specified entropy constraint. The unique solution to

$T > 0$7

is given by $T > 0$8 for some $T > 0$9; equivalently, $$p_i(T) = \frac{\exp(z_i/T)}{\sum_{j=1}^K \exp(z_j/T)}$$0 (Mattei et al., 16 Feb 2026).

Algebraic Uniqueness: Among all linear scalers of the form $$p_i(T) = \frac{\exp(z_i/T)}{\sum_{j=1}^K \exp(z_j/T)}$$1 followed by softmax, only temperature scaling ($$p_i(T) = \frac{\exp(z_i/T)}{\sum_{j=1}^K \exp(z_j/T)}$$2, $$p_i(T) = \frac{\exp(z_i/T)}{\sum_{j=1}^K \exp(z_j/T)}$$3, $$p_i(T) = \frac{\exp(z_i/T)}{\sum_{j=1}^K \exp(z_j/T)}$$4) leaves the argmax (hard prediction) unchanged for every input $$p_i(T) = \frac{\exp(z_i/T)}{\sum_{j=1}^K \exp(z_j/T)}$$5—thus uniquely guaranteeing accuracy preservation (Mattei et al., 16 Feb 2026).

Gradient and Optimization Effects: Introducing temperature into the softmax cross-entropy modifies both the magnitude and direction of gradients during training. For loss $$p_i(T) = \frac{\exp(z_i/T)}{\sum_{j=1}^K \exp(z_j/T)}$$6 and class weights $$p_i(T) = \frac{\exp(z_i/T)}{\sum_{j=1}^K \exp(z_j/T)}$$7: $$p_i(T) = \frac{\exp(z_i/T)}{\sum_{j=1}^K \exp(z_j/T)}$$8 where $$p_i(T) = \frac{\exp(z_i/T)}{\sum_{j=1}^K \exp(z_j/T)}$$9 is the softmax under temperature $\beta = 1/T$0 (Xuan et al., 28 Feb 2025). High temperature $\beta = 1/T$1 (or $\beta = 1/T$2) dampens gradient norm, effectively reducing the optimizer step size; it also “de-biases” the update, making it an average over all negative prototypes rather than focusing on the hardest confusion.

3. Extensions: Adaptive, Per-Layer, and Data-Dependent Temperature Scaling

Adaptive Temperature Scaling (ATS): ATS replaces constant temperature with an input- or confidence-dependent function $\beta = 1/T$3, broadening post-hoc calibration capacity. Variants include:

• Parametric: Per-sample temperature via neural networks acting on sorted logits (Balanya et al., 2022).
• Bin-wise: Discrete temperatures per confidence bin (Balanya et al., 2022).
• Entropy-based TS (HTS): $\beta = 1/T$4, with $\beta = 1/T$5 the normalized entropy, granting a simple, interpretable model with demonstrated robustness to limited data (Balanya et al., 2022).

Sample-dependent TS: AdaTS employs a VAE on feature space, producing per-point likelihoods, from which a small MLP predicts $\beta = 1/T$6 for each input, further reducing calibration error and enabling fine-grained confidence correction (Joy et al., 2022). Empirically, AdaTS consistently improves Expected Calibration Error (ECE) and robustness to covariate shift.

Layer-Stack TS (LATES): Rather than scaling only the output logits, LATES pools per-layer probe logits with learned, nonnegative weights $\beta = 1/T$7 (or equivalently, $\beta = 1/T$8 temperatures). The final aggregate is $\beta = 1/T$9, and the softmax is applied to this ensemble. LATES yields consistent improvements in ECE, cross-entropy, Brier score, and AUC, outperforming standard TS, particularly under distribution shift (Khalifa et al., 2022).

4. Temperature Scaling in Self-Supervised and Contrastive Learning

In self-supervised contrastive pipelines (e.g., SimCLR, MoCo), temperature scaling modulates the InfoNCE loss: $$p_i(\beta) = \frac{\exp(\beta z_i)}{\sum_j \exp(\beta z_j)}$$0 where $$p_i(\beta) = \frac{\exp(\beta z_i)}{\sum_j \exp(\beta z_j)}$$1. Lowering $$p_i(\beta) = \frac{\exp(\beta z_i)}{\sum_j \exp(\beta z_j)}$$2 sharpens the distribution (emphasizing hard negatives), while raising $$p_i(\beta) = \frac{\exp(\beta z_i)}{\sum_j \exp(\beta z_j)}$$3 “smooths” it. DySTreSS introduces dynamically scaled, similarity-dependent temperatures: $$p_i(\beta) = \frac{\exp(\beta z_i)}{\sum_j \exp(\beta z_j)}$$4 adjusting the temperature for each pair based on cosine similarity, mitigating over-pushing of false negatives and controlling the uniformity/tolerance trade-off. This approach yields superior cluster compactness and representation quality, with empirical gains in linear-probe and k-NN accuracy, and lower inter-class uniformity (Manna et al., 2023).

5. Adversarial Robustness and Security Implications

Temperature-scaled activations play a substantial role in adversarial robustness:

• Training with high $$p_i(\beta) = \frac{\exp(\beta z_i)}{\sum_j \exp(\beta z_j)}$$5 in the softmax (or alternative activation functions like sigmoid/tanh) effectively zeros out gradients used by gradient-based adversarial attacks during both targeted and untargeted threat models. The key mechanism is that, as $$p_i(\beta) = \frac{\exp(\beta z_i)}{\sum_j \exp(\beta z_j)}$$6, activations saturate, making $$p_i(\beta) = \frac{\exp(\beta z_i)}{\sum_j \exp(\beta z_j)}$$7 for all $$p_i(\beta) = \frac{\exp(\beta z_i)}{\sum_j \exp(\beta z_j)}$$8 (Tuna et al., 2022).
• Empirically, on MNIST and CIFAR-10, models with high-$$p_i(\beta) = \frac{\exp(\beta z_i)}{\sum_j \exp(\beta z_j)}$$9 activations show orders-of-magnitude reductions in attack success rates for FGSM, BIM, DeepFool, and Carlini–Wagner. Clean test accuracy typically drops by <2pp (Tuna et al., 2022).
• For adversarial training, moderate/high $T \to 0^+$0 in the clean loss term additionally improves PGD20 and CW robustness, maintaining 90% accuracy under attack at $T \to 0^+$1 (CIFAR-10/ResNet-50). Clean accuracy remains at or slightly above the standard baseline until extremely high $T \to 0^+$2 (Xuan et al., 28 Feb 2025).

Table: Illustrative Effect of Training Temperature on Performance (CIFAR-10, ResNet-50) (Xuan et al., 28 Feb 2025)

$T \to 0^+$3	Clean Acc (%)	Corruption Acc (%)	PGD20 (%)	CW (%)
0.1	90.05	73.31	0.00	27.8
1	94.26	72.53	0.00	19.2
30	95.26	74.93	91.1	43.4
100	95.05	73.08	94.3	37.3

Increasing $T \to 0^+$4 does not substantially degrade clean accuracy for moderate values (up to $T \to 0^+$5), but dramatically boosts adversarial robustness.

6. Practical Applications and Implementation

Calibration: Post-hoc TS is almost always implemented by dividing output logits by a fitted $T \to 0^+$6 before softmax, optimizing $T \to 0^+$7 against held-out cross-entropy on a validation set. This is convex, extremely fast, and accuracy-preserving (Mattei et al., 16 Feb 2026, Balanya et al., 2022).

Uncertainty Control: Adjusting $T \to 0^+$8 provides a monotonic handle on predictive entropy, allowing continuous tuning between sharp (overconfident) and flat (underconfident) distributions. All major deep learning frameworks support temperature parameterization in softmax.

Generalization to Other Activations: While the canonical use is with softmax, temperature scaling has also been analyzed for sigmoid and tanh in output layers, particularly for adversarial defense (Tuna et al., 2022). Not all activation functions (e.g., ReLU) admit a meaningful temperature parameterization.

Guidelines: Moderate $T \to 0^+$9 yields the best joint trade-off between calibration, robustness, and clean accuracy. For adversarial defense, $p(T)$0–$p(T)$1 is recommended during training on standard image datasets, resetting $p(T)$2 at inference (Xuan et al., 28 Feb 2025, Tuna et al., 2022).

7. Interpretability, Limitations, and Open Directions

Interpretability: Entropy-based temperature scaling enables transparent linkage between predictive entropy and required softmax smoothing, supporting interpretable recalibration even in low-sample regimes (Balanya et al., 2022).

Limitations: Highly expressive or per-sample temperature models (complex neural nets) improve calibration only with large held-out datasets; under limited data, entropy-based or linear scaling is preferable due to generalization robustness (Balanya et al., 2022). For overconfident/underconfident misalignment, only per-sample or entropy-based TS can provide local correction (Joy et al., 2022).

Future Directions: Open problems include:

• Automatic $p(T)$3-scheduling curriculums during training (Xuan et al., 28 Feb 2025).
• Fully generalizing robust calibration on high-dimensional, low-data applications such as medical imaging (Balanya et al., 2022).
• Extending temperature scaling to more activation types and structured output spaces.
• Integrating TS with advanced uncertainty quantification and OOD detection mechanisms.

Temperature-scaled activations provide a theoretically grounded, computationally efficient, and versatile tool for calibrating neural classifiers, modulating predictive uncertainty, improving adversarial robustness, and optimizing feature representations in both supervised and self-supervised paradigms (Mattei et al., 16 Feb 2026, Xuan et al., 28 Feb 2025, Manna et al., 2023, Tuna et al., 2022, Khalifa et al., 2022, Joy et al., 2022, Balanya et al., 2022).