Temperature Scaling for Zero-Shot Calibration

• The paper introduces temperature scaling methods that adjust softmax confidence scores via a single temperature parameter without using target-domain labels.
• Key strategies include pseudo-labeling, unsupervised partitioning, and consistency-based loss functions to optimize the negative log-likelihood and KL divergences.
• Empirical results show significant ECE reductions—up to 60–70% in OOD, vision-language, and multilingual tasks—demonstrating practical, robust calibration.

Temperature scaling for zero-shot calibration refers to a family of post-hoc uncertainty calibration methods designed to adjust the confidence scores of neural classifiers in scenarios where no target-domain (out-of-distribution, OOD) or target-language labels are available at calibration time. These approaches are distinguished by their reliance on only source-domain data, synthetic consistency, model features, or pseudo-label strategies to select the temperature parameter, thus enabling robust calibration under significant domain shift or new tasks—without access to target domain labels or data.

1. Mathematical Foundations and Properties

Temperature scaling introduces a scalar parameter $T > 0$ into the softmax function, modifying the class probability for pre-trained logits $f(x) \in \mathbb{R}^K$ by

$$p_T(y=k \mid x) = \frac{\exp(f_k(x)/T)}{\sum_{j=1}^K \exp(f_j(x)/T)}.$$

As $T \rightarrow \infty$, the output distribution approaches the uniform distribution of maximal entropy; for $T \rightarrow 0$, predictions become highly peaked. The theoretical properties of temperature scaling have been rigorously analyzed: increasing $T$ strictly increases the entropy of the predictive distribution, and temperature scaling is the only affine calibration transform that preserves the model’s hard (arg-max) predictions. Geometrically, the temperature-scaled distribution is the information projection of the original softmax onto the set of distributions with a prescribed entropy (Mattei et al., 16 Feb 2026).

2. Zero-Shot Temperature Selection Principles

In the absence of target-domain or target-language labels, several strategies emerge for zero-shot (unlabeled) temperature selection:

• Self-consistency via pseudo-labels: Infer pseudo-labels (e.g., via the model’s arg-max prediction on each $x_i$), then minimize the expected negative log-likelihood (NLL) against these predictions (Mattei et al., 16 Feb 2026).
• Class-wise unsupervised partitioning: Partition the unlabeled validation pool into sets $M_k$ of samples likely to come from class $k$, then minimize the average NLL on these synthetic class assignments as in Unsupervised Temperature Scaling (UTS) (Mozafari et al., 2019).
• Average-confidence heuristics: Select $T$ to match a specified average confidence or entropy over the unlabeled evaluation set (Mattei et al., 16 Feb 2026).
• Consistency-based and feature-based approaches: Leverage domain-agnostic or multi-domain models to regress or synthesize calibration signals from available data or structures, achieving robust generalization (Yu et al., 2022, Choi et al., 2024).

Each method inherits the convexity, monotonicity, and accuracy-preserving properties of temperature scaling.

3. Consistency-Guided and Multi-Domain Zero-Shot Calibration

Recent advances specifically address OOD and zero-shot settings by exploiting multiple source domains or sample-level mutual supervision:

• Consistency-Guided Temperature Scaling (CTS): For multi-domain image classification, CTS constructs calibration objectives that include not just the standard NLL on source-domain samples, but also “style” and “content” consistency losses. These losses compare the model’s predictions under synthetic perturbations of internal feature representations—formed by decomposing features into style and content using statistical moments and normalization (e.g., AdaIN). The optimal $f(x) \in \mathbb{R}^K$0 is chosen to minimize

$f(x) \in \mathbb{R}^K$1

where $f(x) \in \mathbb{R}^K$2 and $f(x) \in \mathbb{R}^K$3 are KL divergences between the original softmax outputs and those from style- or content-swapped features (Choi et al., 2024). CTS operates entirely with source-domain data, requiring no OOD samples at calibration time.

• Multi-Domain Temperature Scaling (MD-TS): MD-TS first fits a temperature for each source domain, then regresses these temperature values against the penultimate-layer embeddings to learn a linear temperature function $f(x) \in \mathbb{R}^K$4. This function can be directly applied to unseen domains by evaluating on their feature representations. The theoretical domain adaptation bound guarantees that if the learned temperature map performs well on sources and the target is not too far in feature-space divergence, calibration will generalize (Yu et al., 2022).

These approaches outperform single-domain temperature scaling in OOD average calibration error (ECE), often by substantial margins, and approach the performance of oracle schemes that are fitted directly on labeled target-domain data (Choi et al., 2024, Yu et al., 2022).

4. Zero-Shot Calibration in Vision-Language and Multilingual Models

Temperature scaling has been adapted to zero-shot settings in vision-language and multilingual models:

• Vision-LLMs (CLIP/Zero-Shot Inference): CLIP-style models are calibrated by learning a global $f(x) \in \mathbb{R}^K$5 on a single auxiliary labelled dataset compatible with the model’s pre-training. The fixed $f(x) \in \mathbb{R}^K$6 is then used for arbitrary zero-shot tasks and prompt combinations, substantially reducing ECE (typically by 50–70%) without compromising the “open-vocabulary” deployment regime (LeVine et al., 2023). In adversarial settings, softening the ground-truth label distribution using temperature scaling, as in AGFT, further preserves calibration and cross-modal alignment under robustness penalties (Cui et al., 31 Mar 2026).
• Multilingual LLMs: Temperature scaling is shown to halve the ECE in zero-shot cross-lingual evaluation by learning $f(x) \in \mathbb{R}^K$7 on the pivot (e.g., English) development set and applying it unchanged to all target languages. If a small development set is available in the target language, language-specific temperature scaling can yield further improvements (Ahuja et al., 2022).

Model/Setting	Zero-Shot TS Approach	Typical ECE Reduction
Image classifier, OOD	CTS or MD-TS	$f(x) \in \mathbb{R}^K$840–60%
Vision-language (CLIP)	Global $f(x) \in \mathbb{R}^K$9 per pre-train	$$p_T(y=k \mid x) = \frac{\exp(f_k(x)/T)}{\sum_{j=1}^K \exp(f_j(x)/T)}.$$050–70%
Multilingual LLMs	Pivot-language global $$p_T(y=k \mid x) = \frac{\exp(f_k(x)/T)}{\sum_{j=1}^K \exp(f_j(x)/T)}.$$1	$$p_T(y=k \mid x) = \frac{\exp(f_k(x)/T)}{\sum_{j=1}^K \exp(f_j(x)/T)}.$$250%

5. Empirical Results and Calibration Metrics

The primary metric for calibration assessment is Expected Calibration Error (ECE), computed as: $$p_T(y=k \mid x) = \frac{\exp(f_k(x)/T)}{\sum_{j=1}^K \exp(f_j(x)/T)}.$$3 where $$p_T(y=k \mid x) = \frac{\exp(f_k(x)/T)}{\sum_{j=1}^K \exp(f_j(x)/T)}.$$4 denotes bins of predicted confidence, and $$p_T(y=k \mid x) = \frac{\exp(f_k(x)/T)}{\sum_{j=1}^K \exp(f_j(x)/T)}.$$5 and $$p_T(y=k \mid x) = \frac{\exp(f_k(x)/T)}{\sum_{j=1}^K \exp(f_j(x)/T)}.$$6 are the average confidence and accuracy in each bin. All temperature scaling methods preserve the underlying model's classification accuracy, as the arg-max prediction is invariant to scaling.

In zero-shot OOD calibration:

• CTS reduces ECE on held-out domains to 4–7% from baseline values of ~9–14%, closely approaching the oracle with access to target labels (3–5%) (Choi et al., 2024).
• MD-TS reduces OOD ECE to 4–10% from single-domain TS values of 6–14% (Yu et al., 2022).
• CLIP with global zero-shot TS achieves ECE reductions from 26.5% to 6–8% on challenging evaluation datasets (LeVine et al., 2023).
• Unsupervised Temperature Scaling achieves similar or slightly lower ECE compared to supervised TS in image classification benchmarks, with ECE improvements of 1–2 percentage points on standard datasets (Mozafari et al., 2019).
• In multilingual models, global TS reduces ECE from 13.3% to 6.7% on XNLI, from 20.1% to 16.0% on XCOPA, and Self-TS yields further reductions, when a small labeled set is available (Ahuja et al., 2022).

6. Practical Implementation and Limitations

Zero-shot temperature scaling incurs negligible computational overhead and requires only simple optimization routines for a single parameter or a low-dimensional regression. For CTS, one picks a model layer for style/content decomposition and tunes two hyperparameters $$p_T(y=k \mid x) = \frac{\exp(f_k(x)/T)}{\sum_{j=1}^K \exp(f_j(x)/T)}.$$7 on source-domain validation data. For UTS and self-consistency methods, a moderate-sized unlabeled calibration pool is recommended to ensure robustness, and thresholds for partitioning may require adjustment for class imbalance (Mozafari et al., 2019, Choi et al., 2024). In vision-language settings, the calibration parameter is fixed per architecture and pre-training corpus, and incurs no per-task tuning (LeVine et al., 2023).

Practical limitations include possible suboptimal results when the calibration pool poorly matches the deployment distribution, sensitivity to class imbalance in unsupervised partitioning, and the inability of global-$$p_T(y=k \mid x) = \frac{\exp(f_k(x)/T)}{\sum_{j=1}^K \exp(f_j(x)/T)}.$$8 methods to cope with highly non-homogeneous feature spaces without further adaptation.

7. Significance and Outlook

Temperature scaling methods for zero-shot calibration constitute a principled, theoretically robust, and practical response to the challenge of reliable uncertainty quantification in domains where target data is unavailable or expensive to label. Recent developments—such as consistency-guided calibration, multi-domain regression, and calibration of vision-language and multilingual models—demonstrate that temperature scaling, combined with domain-exploiting or self-consistency objectives, closes much of the performance gap between supervised and zero-shot calibration. These methods are widely adopted for trustworthy deployment in safety-critical, open-vocabulary, and multi-lingual systems and remain an area of active extension to richer post-hoc calibration schemes and further generalized consistency frameworks (Choi et al., 2024, Yu et al., 2022, LeVine et al., 2023, Mattei et al., 16 Feb 2026, Mozafari et al., 2019).