---
title: Temperature Scaling in Neural Networks
url: https://www.emergentmind.com/topics/temperature-scaling-ts
type: topic
---

# Temperature Scaling in Neural Networks

Temperature scaling (TS) denotes the rescaling of logits by a positive scalar before the softmax. In classification, it is a post-hoc calibration method: the trained model weights are held fixed, logits are divided by a temperature \(T>0\), and the resulting probabilities are used to better align confidence with correctness while preserving the hard prediction. In language-model and reinforcement-learning settings, the same transformation is also used to control output sharpness, entropy, and sampling behavior, although its sequence-level effects differ from its classifier-calibration role [2602.14862, 1905.00174].

## 1. Mathematical definition and invariant properties

Let \(z=(z_1,\dots,z_K)\) be the logits of a \(K\)-class model. Temperature scaling replaces the standard softmax by
\[
\hat p_i(T)=\frac{\exp(z_i/T)}{\sum_{j=1}^K \exp(z_j/T)}.
\]
Equivalently, with inverse temperature \(\beta=1/T\),
\[
\pi_{k,\beta}=\mathrm{Softmax}(\beta z)_k
\quad\Longleftrightarrow\quad
\pi_{k,\beta}=\frac{\pi_k^\beta}{\sum_{j=1}^K \pi_j^\beta}.
\]
For \(T=1\), one recovers the base model; for \(T>1\), the distribution softens; for \(T<1\), it sharpens. Because multiplying logits by a positive scalar preserves their ordering, TS does not change the hard prediction \(\arg\max_k p(y=k\mid z)\); in this sense it is accuracy-preserving for the classifier’s decision rule [2602.14862].

The theory in "The Well-Tempered Classifier" gives three elementary properties. First, increasing temperature increases uncertainty in a very general sense. In particular, if \(p_\beta\) is the tempered classifier, then the Shannon entropy satisfies
\[
\frac{d}{d\beta}H\bigl(p_\beta(\cdot\mid z)\bigr)
=
-\,\beta\;\mathrm{Var}_{k\sim p_\beta}[z_k]
\le 0,
\]
so entropy decreases in \(\beta\) and therefore increases in \(T\). The same work further states a majorization result: if \(0<\beta_1<\beta_2\), then \(p_{\beta_2}\) majorizes \(p_{\beta_1}\), implying monotonic behavior for Schur-concave uncertainty measures. Second, TS admits a geometric characterization: among distributions on the simplex with a prescribed entropy, the one minimizing \(\mathrm{KL}(q\|p)\) is exactly a temperature-scaled version of \(p\). Third, among linear post-hoc scalers such as matrix scaling and Dirichlet calibration, TS is the only linear family that does not change the hard predictions [2602.14862].

These results clarify why TS occupies a special position among post-hoc recalibrators. It is not merely a heuristic softmax flattening; it is a one-parameter submodel with a precise entropy monotonicity, an information-projection interpretation, and a unique accuracy-preserving status within linear scaling families [2602.14862].

## 2. Fitting the temperature and measuring calibration

Standard TS uses a labeled validation set \(\{(x_i,y_i)\}_{i=1}^N\) that is disjoint from weight training. The network weights are frozen, and \(T\) is chosen by minimizing the negative log-likelihood (NLL),
\[
\mathrm{NLL}(T)
=
-\,\frac{1}{N}\sum_{i=1}^N \log \hat p_{y_i}(T),
\]
or, in equivalent notation,
\[
T^*=\arg\min_{T>0} L_{\rm TS}(T),\qquad
L_{\rm TS}(T)=-\sum_{i=1}^N \log p_{y_i}(x_i;T).
\]
Because this is a single-parameter optimization, the cited works describe it as convex or strictly convex and solvable by simple line search, gradient-based methods, L-BFGS, or simple gradient descent initialized at \(T=1\) [1905.00174, 2509.24951].

A common external calibration metric is expected calibration error (ECE). With \(M\) confidence bins \(B_1,\dots,B_M\),
\[
\mathrm{ECE}
=
\sum_{m=1}^M
\frac{|B_m|}{N}
\left|
\mathrm{acc}(B_m)-\mathrm{conf}(B_m)
\right|,
\]
where \(\mathrm{acc}(B_m)\) is the empirical accuracy in bin \(m\) and \(\mathrm{conf}(B_m)\) is the mean confidence. The definition operationalizes the ideal calibration condition \(P(\hat y=k\mid \text{confidence}=p)=p\): a perfectly calibrated classifier has confidence frequencies matching empirical correctness frequencies at each confidence level [1905.00174].

The workflow is correspondingly simple. One carves out a held-out calibration split, fits \(T\) solely on that split, and evaluates both predictive metrics and calibration metrics on test data. The medical-imaging study recommends reserving approximately \(10\%\) to \(20\%\) of the data for temperature tuning, not using test labels for calibration, and using approximately \(10\) to \(20\) bins for ECE to avoid extreme sparsity while retaining stable estimates [2509.24951].

## 3. Empirical behavior in deep classifiers

A detailed case study is provided by "Evaluating Temperature Scaling Calibration Effectiveness for CNNs under Varying Noise Levels in Brain Tumour Detection" [2509.24951]. The model is a light CNN with 4 convolutional blocks followed by 2 fully connected layers and a dropout of \(p=0.25\); inputs are \(256\times256\times3\), training uses Adam with learning rate \(3\times10^{-4}\), batch size \(64\), and up to 60 epochs with early stopping. The dataset split is \(80\%\) training, \(10\%\) validation for TS tuning, and \(10\%\) testing. To simulate realistic MRI artifacts, five synthetic noise types are applied only to the test set: Gaussian, Poisson, Salt & Pepper, Speckle, and Uniform noise [2509.24951].

On clean data, the uncalibrated model attains accuracy \(=0.98\), NLL \(=0.079\), and ECE \(=0.016\). After TS, accuracy remains approximately \(0.98\), NLL decreases from \(0.079\) to \(0.077\), and ECE changes from approximately \(0.016\) to \(0.017\). Under noise, the reductions are larger. For Gaussian noise with \(\sigma=0.10\) and \(\mu=0.00\), the uncalibrated model has accuracy \(=0.965\), NLL \(=0.127\), and ECE \(=0.021\); after TS with \(T^*=1.532\), accuracy is \(0.955\) (\(\pm0.01\)), NLL falls to \(0.113\), and ECE falls to \(0.009\), corresponding to NLL \(\downarrow 11\%\) and ECE \(\downarrow 57\%\) without meaningful accuracy loss [2509.24951].

Across other corruptions, the same pattern recurs. For Poisson noise at scale \(1\), NLL changes from \(0.067\) to \(0.065\) and ECE from \(0.010\) to \(0.010\), with \(T^*=1.382\). For Salt & Pepper noise with \(p_{\rm salt}=0.2\) and \(p_{\rm pepper}=0.2\), NLL changes from \(0.149\) to \(0.145\), ECE from \(0.020\) to \(0.013\), and \(T^*=1.499\). Across Uniform and Speckle noise up to scale \(0.1\), NLL reductions are \(0.062\to0.054\) and \(0.082\to0.070\), while ECE reductions are \(0.012\to0.006\) for both; \(T^*\) ranges from approximately \(1.17\) under light noise to approximately \(1.56\) under heavier noise. Repeating optimal noise cases on ResNet50 shows the same trend, with Gaussian \((\sigma=0.02)\) yielding NLL \(0.027\to0.025\) and ECE \(0.008\to0.003\), and Uniform \((\text{scale}=0.05)\) yielding NLL \(0.017\to0.015\) and ECE \(0.007\to0.006\), while accuracy remains approximately \(99\%\) [2509.24951].

The reported operational cost is correspondingly small. TS adds only a single scalar optimization step on the validation set, took less than one second on GPU with L-BFGS in those experiments, requires no changes to the model architecture, and does not retrain weights. This makes it attractive whenever the objective is to improve NLL and ECE without altering the learned decision rule [2509.24951].

## 4. Limitations, failure modes, and interaction with conformal prediction

The simplicity of TS is also its principal limitation. Because it rescales confidence uniformly across all samples, it cannot correct class-specific or region-specific miscalibration. The brain-tumour study therefore notes that under severe distribution shift TS may not suffice and suggests adaptive TS variants or full retraining with calibration-aware loss when the shift is strong [2509.24951].

A second limitation concerns the calibration set. "Attended Temperature Scaling" shows that the standard TS objective depends directly on labels in the held-out validation set, and identifies two concrete failure modes: small or noisy validation sets, and highly accurate networks. If \(N\) is small or labels are corrupted, the estimate of \(T^*\) can become highly unstable; when a network makes almost no errors on the calibration split, there are too few misclassified examples to pull down overconfidence, so TS can remain sub-optimally calibrated [1810.11586].

A third limitation appears when TS is composed with adaptive conformal prediction. "On Temperature Scaling and Conformal Prediction of Deep Classifiers" reports that TS improves the class-conditional coverage of adaptive CP methods, but negatively affects their prediction-set sizes. For CIFAR-100 with ResNet50 at \(\alpha=0.1\), optimal \(T\approx1.52\) leaves LAC set size nearly unchanged at \(1.62\to1.57\), but increases APS set size from \(5.35\) to \(9.34\) and RAPS set size from \(2.68\) to \(4.96\), while coverage remains approximately \(0.90\). The same work proves that APS and RAPS thresholds decrease monotonically with \(T\ge1\), yet the mean set size typically increases, often non-monotonically but with an overall upward trend. Its practical guideline is explicit: do not apply TS before APS or RAPS when the classifier’s top-1 accuracy is below \(1-\alpha\), and note that LAC is unaffected in set size because TS preserves the softmax ranking [2402.05806].

These limits are not contradictions of the core TS theory. They instead delimit the regime in which a single global temperature is well matched to the structure of the miscalibration problem.

## 5. Major variants and extensions

A substantial literature generalizes standard TS without abandoning the central idea of logit rescaling. The variants differ primarily in what data are used to fit the temperature and whether the temperature remains global, becomes sample-dependent, or is structured by class, domain, or network depth [1905.00174, 1810.11586, 2207.06211, 2402.15019, 2308.08366, 2211.10193].

| Variant | Core mechanism | Reported behavior |
|---|---|---|
| UTS | Fit \(T\) on unlabeled samples via class-wise self-selection | Matches or outperforms TS in NLL and ECE |
| ATS | Attend to borderline examples from other classes | More stable with small or noisy validation sets |
| AdaTS | Predict \(T(x)\) per sample | Lowers ECE and AdaECE beyond global TS |
| CTS | Add style/content consistency losses | Improves OOD calibration using only source domains |
| Dual-TS | Fuse class-adaptive and equal-size-bin temperatures | Improves calibration in long-tailed recognition |
| LATES | Learn nonnegative weights across layer probes | Improves calibration and also accuracy/AUC |

UTS removes the need for labeled calibration data. It assumes that high-confidence samples for class \(k\) are likely to come from the true class-\(k\) distribution, constructs sets
\[
M_k=\{x\in V\mid S_k(x)\ge \theta_k\},\qquad \theta_k=\mu_k+\sigma_k,
\]
and minimizes an unsupervised NLL over these pseudo-labeled subsets. On ResNet-110/CIFAR-10, NLL changes from \(0.312\) uncalibrated to \(0.228\) with TS and \(0.207\) with UTS, while ECE changes from \(4.34\%\) to \(4.30\%\) with TS and \(1.47\%\) with UTS. On DenseNet-100/CIFAR-100, NLL changes from \(1.119\) to \(0.886\) with TS and \(0.878\) with UTS, while ECE changes from \(11.97\%\) to \(4.74\%\) with TS and \(2.77\%\) with UTS [1905.00174].

ATS enlarges each class’s calibration subset by including true class-\(k\) samples together with borderline non-\(k\) samples satisfying \(S_{y=k}(x_i)\ge \theta\). The reported effect is robustness when the validation set is small or contains noisy labels. On DenseNet40/CIFAR-10, ECE changes from \(4.09\%\) uncalibrated to \(3.24\%\) with TS and \(0.66\%\) with ATS; on ISIC skin-lesion detection with ResNet200, ECE changes from \(8.37\%\) uncalibrated to \(7.98\%\) with TS and \(1.16\%\) with ATS [1810.11586].

AdaTS replaces the single scalar with a predicted \(T(x)\) for each sample, using a small VAE and an MLP on pseudo-likelihood features. On CIFAR-10 and CIFAR-100, it improves both ECE and AdaECE beyond TS; on CIFAR-10-C, ResNet50 ECE drops from \(18.48\) uncalibrated to \(12.72\) with TS and \(10.86\) with AdaTS [2207.06211].

CTS is designed for out-of-domain calibration when target-domain validation labels are unavailable. It augments the TS NLL with KL consistency terms under style-shifted and content-shifted feature recombinations. On PACS, Office-Home, Digits-DG, and VLCS, the reported average ECE values for CTS are \(4.31\), \(4.75\), \(6.53\), and \(12.90\), compared with \(9.06\), \(9.79\), \(11.08\), and \(14.00\) for source-validation TS and \(3.60\), \(3.05\), \(4.74\), and \(8.69\) for oracle TS using target labels [2402.15019].

Dual-TS addresses long-tailed recognition by combining class-adaptive temperatures \(T_c^{CA}\) with equal-size-bin temperatures \(T_b^{ES}\) through
\[
T_i^*=(T_c^{CA})^\alpha (T_b^{ES})^{1-\alpha}.
\]
On CIFAR-10-LT with imbalance factor \(100\), Dual-TS reports accuracy \(70.3\%\), ECE \(2.67\), and Esbin-ECE \(2.64\), versus \(63.8\%\) accuracy and ECE \(7.46\) for standard TS. On CIFAR-100-LT with imbalance factor \(100\), Dual-TS reports accuracy \(42.3\%\), ECE \(2.61\), and Esbin-ECE \(2.51\), versus \(38.6\%\) accuracy and ECE \(3.66\) for TS [2308.08366].

LATES distributes the scaling operation across all layers by attaching probes after each block and learning a nonnegative weight vector \(\beta\) for their logits. On ImageNet with ResNet50, TS reports ECE \(0.060\), NLL \(1.156\), ACC \(0.738\), and AUC \(0.991\), whereas LATES reports ECE \(0.023\), NLL \(1.039\), ACC \(0.745\), and AUC \(0.991\). Its stated overhead is less than \(1\%\) extra parameters, less than \(0.7\%\) extra FLOPs at inference, and less than \(25\%\) extra training time [2211.10193].

## 6. Temperature scaling beyond classifier calibration

Outside standard classifier calibration, TS appears in three closely related but distinct roles. The first is language-model decoding and test-time scaling. The second is entropy restoration in reinforcement learning. The third is token-level recalibration for semantic uncertainty in question answering [2602.14862, 2510.02611, 2606.00755, 2604.07172].

For autoregressive decoding, token-level temperature rescales next-token logits exactly as in classification. However, "The Well-Tempered Classifier" challenges the common claim that increasing temperature increases diversity for large language models. A full temperature scaling of the joint autoregressive model is intractable, so practitioners apply a myopic token-level version; in that setting, sequence-level entropy or diversity can be non-monotonic in temperature [2602.14862].

At inference time, this does not preclude useful multi-temperature search. "On the Role of Temperature Sampling in Test-Time Scaling" reports that different sampling temperatures solve different subsets of reasoning problems. Averaged over Qwen3 \(0.6\)B, \(1.7\)B, \(4\)B, and \(8\)B on AIME 2024/2025, MATH500, LiveCodeBench, and Hi-ToM, multi-temperature scaling raises Pass.All from \(58.7\%\) to \(66.0\%\), an additional \(+7.3\) points over single-temperature TTS. For Qwen3 \(4\)B on AIME 2025, performance rises from \(60.0\%\) to \(73.3\%\) [2510.02611].

In RL from verifiable rewards, TS can be internalized into the model rather than used only at rollout time. TS-OPSD constructs a self-teacher by applying high-temperature scaling to the model’s own logits, then distills that smoother distribution back into the student. On Qwen3-4B, macro-averaged avg@8 and pass@8 change from approximately \(29.40\) and \(43.27\) for baseline DAPO to approximately \(31.46\) and \(46.05\) for policy reheating with forward KL; on Qwen3-8B they change from approximately \(34.27\) and \(51.60\) to approximately \(36.60\) and \(54.74\). The reported analysis further states that linear CKA between the collapsed and reheated checkpoints remains near \(0.999\), while top-\(k\) overlap and overlapping probability mass both exceed \(0.95\) [2606.00755].

For semantic uncertainty in QA, token-level scalar TS is treated as a post-hoc recalibration method for confidence measures derived from token probabilities. On TriviaQA with Llama-3 and the E-SC measure, ACE changes from \(0.184\) at \(\tau=1\) to \(0.049\) at \(\tau^*\approx1.26\), while AUROC changes from \(0.757\) to \(0.852\). On Natural Questions with Qwen-2.5, ACE changes from \(0.487\) to \(0.155\), while AUROC changes from \(0.690\) to \(0.759\). The same study argues that a single scalar provides a suitable inductive bias, whereas more expressive token-level recalibration methods can overfit [2604.07172].

Taken together, these works show that the phrase “temperature scaling” names a shared logit-rescaling mechanism whose consequences depend strongly on the object being scaled. For classifiers, TS is a one-parameter recalibration that preserves hard predictions and usually improves NLL and ECE. For adaptive prediction sets, domain shift, long-tailed recognition, and sample-specific miscalibration, structured variants modify the fitting procedure or the temperature itself. For language models and RL, the same operation governs sharpness and entropy, but its sequence-level and policy-level consequences are not reducible to the standard classifier-calibration picture.

Source: https://www.emergentmind.com/topics/temperature-scaling-ts