Data-Point Tempering Overview

• Data-point tempering is a technique that assigns individual temperatures to observations to systematically control their contribution in probabilistic inference.
• It reformulates likelihoods or margins by incorporating temperature parameters, thereby adjusting optimization geometry and enhancing model robustness.
• Applications span Bayesian inference, supervised learning, and MCMC subsampling, enabling targeted calibration and improved predictive performance.

Data-point tempering denotes a family of procedures in which individual observations, groups of observations, or continuously indexed fractions of data are assigned temperatures or inverse temperatures so that their contributions to inference or learning are no longer uniform. In Bayesian formulations, this commonly means replacing a factorized likelihood by powered terms such as $p(x_i,z_i\mid\beta)^{1/T_i}$ or $p(y_i\mid\theta)^{\tau_i}$; in discriminative models it can mean inserting group- or sample-specific inverse temperatures inside margins or logits; in post-hoc calibration it means predicting a temperature $T(x)$ for each input and applying $\sigma(s(x)/T(x))$ rather than a single global scaling. Across these settings, the common purpose is to soften or sharpen the effect of particular datapoints, thereby altering optimization geometry, robustness, calibration, or sequential belief updates [(Mandt et al., 2014); (Lu et al., 2022); (Joy et al., 2022)].

1. Formal scope and recurring mechanisms

The most direct probabilistic definition appears in tempered Bayesian inference. For a prior $p(\beta)$ and likelihood $p(x,z\mid\beta)$, a global temperature $T\ge 1$ yields a tempered joint

$$p_T(\beta,z,x)\propto p(z,x\mid\beta)^{1/T}p(\beta),$$

so that large $T$ flattens the likelihood and weakens the influence of data relative to the prior. Data-point tempering is the local generalization of this construction: instead of one shared $T$, each observation receives its own $p(y_i\mid\theta)^{\tau_i}$0, and the posterior or surrogate objective acquires per-observation weights $p(y_i\mid\theta)^{\tau_i}$1. In exponential-family models this can often be written either as $p(y_i\mid\theta)^{\tau_i}$2 or as a likelihood with natural parameter $p(y_i\mid\theta)^{\tau_i}$3, making local temperature a direct measure of how strongly datapoint $p(y_i\mid\theta)^{\tau_i}$4 is tied to the model (Mandt et al., 2014).

Outside latent-variable Bayesian models, the same pattern reappears in different algebraic forms. In overparameterized classification with exponential-tailed losses, importance tempering rescales the margin inside the exponential,

$p(y_i\mid\theta)^{\tau_i}$5

so that the learned decision boundary obeys heterogeneous constraints $p(y_i\mid\theta)^{\tau_i}$6. In post-hoc calibration, sample-dependent adaptive temperature scaling predicts a distinct $p(y_i\mid\theta)^{\tau_i}$7 for each input and outputs $p(y_i\mid\theta)^{\tau_i}$8. In subsampling-based MCMC, the “temperature” is implemented by changing how many datapoints are conditioned on: hotter auxiliary distributions are posteriors on recursively smaller subsets, with $p(y_i\mid\theta)^{\tau_i}$9. These constructions differ in where temperature is inserted—likelihood factors, margins, logits, or subset size—but all implement observation-level control over information flow [(Lu et al., 2022); (Joy et al., 2022); (Meent et al., 2014)].

2. Local variational tempering and probabilistic weighting

The canonical probabilistic realization of data-point tempering is local variational tempering (LVT), introduced as an extension of global deterministic annealing and variational tempering for conditionally conjugate exponential-family models. Starting from

$T(x)$0

LVT augments the model with local temperatures $T(x)$1 and defines

$T(x)$2

A discrete implementation uses a temperature grid $T(x)$3, one-hot variables $T(x)$4, and a mean-field approximation

$T(x)$5

The local temperature update is

$T(x)$6

and the global update weights each datapoint through

$T(x)$7

The interpretation is explicit: large $T(x)$8 flattens the local likelihood, produces more uniform local responsibility distributions, and reduces the contribution of datapoint $T(x)$9 to global sufficient statistics. The paper states that “Local temperature describes the likelihood that a particular data point came from the non-tempered model. Outliers can be better explained in this model by assigning them a high local temperature.” In this sense, LVT is both annealing and robust reweighting: early in optimization many datapoints may remain hot, while well-explained datapoints cool toward $\sigma(s(x)/T(x))$0. Because embedding temperature as $\sigma(s(x)/T(x))$1 breaks exact conjugacy for the global update, the method adopts an approximation that matches only the first component of the sufficient statistics and treats datapoints as weighted contributions from the $\sigma(s(x)/T(x))$2 model. Empirically, on LDA for NYTimes, arXiv, and Wikipedia, LVT consistently outperformed standard stochastic variational inference, deterministic annealing, and global variational tempering in held-out predictive likelihood; in the same paper, global variational tempering also improved predictive likelihoods and removed the need for a hand-tuned cooling schedule (Mandt et al., 2014).

3. Tempering margins and likelihoods in supervised learning

In overparameterized classification, data-point tempering is motivated less by posterior multimodality than by the implicit bias of gradient descent. Importance weighting multiplies the loss,

$\sigma(s(x)/T(x))$3

but leaves the asymptotic max-margin constraints unchanged. Importance tempering instead rescales the margin inside the exponential and thereby changes the limiting geometry. For binary classification, gradient flow on

$\sigma(s(x)/T(x))$4

converges in direction to a cost-sensitive margin problem with constraints

$\sigma(s(x)/T(x))$5

The method is group-level by design—groups may be classes or label–spurious-attribute combinations—but the analysis extends directly to per-example inverse temperatures $\sigma(s(x)/T(x))$6. Under binary label shift, the paper proves a “square-root rule,” $\sigma(s(x)/T(x))$7, implying that minority classes receive smaller inverse temperature and hence larger required margin. In multiclass settings, tempering the classifier logits, rather than the features, is especially consequential: IT(W) yields maximal separation of minority classes, avoids minority collapse in the asymptotic step-imbalance regime, and substantially improves worst-group accuracy. On imbalanced CIFAR-10 with label shift 1:100, reported worst-group accuracies were 21.5% for ERM, 33.2% for importance weighting, and 57.2% for importance tempering; on CelebA with ResNet-50, the reported worst-group accuracies were 41.1% for ERM, 82.1% for importance weighting, 88.3% for Group DRO, and 89.1% for importance tempering (Lu et al., 2022).

A distinct but related line of work studies likelihood tempering in Bayesian classification. One paper argues that the standard categorical likelihood in Bayesian neural networks implicitly encodes very high aleatoric uncertainty and that data augmentation in SGLD or variational inference softens the likelihood, inducing underconfidence. If $\sigma(s(x)/T(x))$8 denotes a finite augmentation set, the limiting posterior under augmentation becomes

$\sigma(s(x)/T(x))$9

so each augmented observation is effectively hot-tempered. In this account, cold posteriors with $p(\beta)$0 act as a correction for augmentation-induced underconfidence, and a Dirichlet observation model provides an explicit alternative by controlling aleatoric uncertainty through concentration parameters rather than temperature (Kapoor et al., 2022).

A more detailed account of augmentation-induced misspecification treats multiple augmentations of the same datapoint as correlated rather than i.i.d. In a Gaussian mean-estimation model with $p(\beta)$1 additive augmentations per original sample, the exact posterior under the correlated likelihood is recovered by a tempered i.i.d. likelihood with

$p(\beta)$2

This gives temperature a direct effective-sample-size interpretation: when augmentations are nearly redundant, $p(\beta)$3 approaches $p(\beta)$4; when they are highly informative, $p(\beta)$5 approaches $p(\beta)$6. The same paper links the optimal temperature to invariance: models built from group convolutions and aligned with the augmentation group require stronger tempering because augmentations contribute less new information. This suggests that, in Bayesian deep learning, cold posterior effects can arise from a mismatch between augmentation-induced correlation and the i.i.d. likelihood used in training, rather than from a failure of Bayesian conditioning as such (Bachmann et al., 2022).

4. Sample-dependent temperatures for calibration and hardness

Post-hoc calibration provides a further meaning of data-point tempering: temperature becomes a learned function of the input rather than a latent variable in a generative model. Sample-dependent adaptive temperature scaling starts from logits $p(\beta)$7 of a frozen classifier and replaces global temperature scaling by

$p(\beta)$8

The temperature $p(\beta)$9 is predicted from a low-dimensional uncertainty representation derived from a VAE fitted in feature space. The VAE uses class-conditional Gaussian priors $p(x,z\mid\beta)$0, and the temperature network receives the vector of pseudo log-likelihoods

$p(x,z\mid\beta)$1

then outputs $p(x,z\mid\beta)$2. Training is post-hoc on a held-out calibration set with the base classifier frozen and objective

$p(x,z\mid\beta)$3

The rationale is that calibration error is heterogeneous across samples: some predictions are overconfident, others underconfident, and a global scalar cannot correct both simultaneously. The learned $p(x,z\mid\beta)$4 therefore acts as a selective entropy modulator. Low $p(x,z\mid\beta)$5 corresponds to easy, typical, high-confidence samples; high $p(x,z\mid\beta)$6 corresponds to hard, atypical, or likely misclassified samples whose confidence should be flattened. The paper reports that on CIFAR-10 with WideResNet-28-10 and cross-entropy training, ECE was approximately 2.15 with no scaling, 0.93 with vanilla temperature scaling, and 0.76 with adaptive scaling; on CIFAR-100, the corresponding ECEs were approximately 5.76, 3.76, and 2.95. Similar relative improvements were reported for ResNet-50, Tiny-ImageNet, corrupted CIFAR-C data, and AURRA-based misclassification rejection. The method is explicitly post-hoc, uses a held-out calibration set, and leaves accuracy essentially unchanged, but its effectiveness depends on the quality of the VAE density model and on the availability of calibration data (Joy et al., 2022).

5. Subsampling, filtering, and surrogate models

A different interpretation of data-point tempering appears in MCMC tempering by subsampling. Instead of scaling the full likelihood by a temperature, auxiliary distributions condition on recursively smaller subsets: $p(x,z\mid\beta)$7 For i.i.d. data, $p(x,z\mid\beta)$8 approximates $p(x,z\mid\beta)$9, so a subset-size ladder approximates a conventional temperature ladder. Subsampled parallel tempering fixes nested subsets $T\ge 1$0 for the duration of a run, whereas subsampled tempered transitions resample the subsets for each proposal. In both cases, only the cold level uses the full dataset, so the target full posterior remains exact even though the hot auxiliary distributions are approximate. The main benefit is computational: for Gaussian process regression at $T\ge 1$1, reported runtimes relative to HMC were 1.73 for subsampled parallel tempering and 2.35 for subsampled tempered transitions, versus 6.03 for parallel tempering and 9.19 for tempered transitions; in the same experiments, effective sample size per second often favored the subsampled schemes, especially when cost scaled superlinearly with $T\ge 1$2 (Meent et al., 2014).

In sequential inference, the tempered Bayes filter extends per-datapoint likelihood tempering to hidden Markov models. With tempering parameters $T\ge 1$3, the belief recursion becomes

$T\ge 1$4

Here $T\ge 1$5 is likelihood tempering in the usual data-point sense, $T\ge 1$6 tempers the trajectory posterior, and $T\ge 1$7 tempers the marginalized belief. The paper shows that likelihood tempering changes the balance between data and prior, full-posterior tempering changes the entropy of the trajectory distribution, and the path $T\ge 1$8 interpolates between the Bayes filter and the MAP filter. In the linear Gaussian case, the resulting tempered Kalman filter preserves Gaussianity and rescales effective process and measurement information in closed form (Zutphen et al., 2 Dec 2025).

Bayesian optimization offers a third sequential interpretation. A tempered posterior

$T\ge 1$9

downweights each observation when fitting a GP surrogate. For GP regression, this is exactly equivalent to inflating the observation noise variance from $$p_T(\beta,z,x)\propto p(z,x\mid\beta)^{1/T}p(\beta),$$0 to $$p_T(\beta,z,x)\propto p(z,x\mid\beta)^{1/T}p(\beta),$$1, so the posterior mean moves less aggressively and the posterior variance contracts more slowly in densely sampled regions. The paper proves cumulative regret bounds for generalized expected improvement under the tempered posterior and states that tempering yields strictly sharper worst-case regret guarantees than the standard posterior $$p_T(\beta,z,x)\propto p(z,x\mid\beta)^{1/T}p(\beta),$$2. It also proposes a prequential rule that decreases $$p_T(\beta,z,x)\propto p(z,x\mid\beta)^{1/T}p(\beta),$$3 when realized prediction errors exceed model-implied uncertainty and returns $$p_T(\beta,z,x)\propto p(z,x\mid\beta)^{1/T}p(\beta),$$4 toward one as calibration improves, making temperature an online estimate of how much recent datapoints should be trusted (Li et al., 11 Jan 2026).

6. Predictive consequences, asymptotics, and unresolved issues

The effect of data-point tempering on prediction depends strongly on regime. For power posteriors

$$p_T(\beta,z,x)\propto p(z,x\mid\beta)^{1/T}p(\beta),$$5

or more generally weighted log-likelihoods with per-observation weights $$p_T(\beta,z,x)\propto p(z,x\mid\beta)^{1/T}p(\beta),$$6, the predictive distribution can change materially in small samples because tempering alters posterior spread and the balance between prior and data. But under standard concentration conditions—local Lipschitz behavior in Hellinger distance and posterior contraction around the pseudo-true parameter—the tempered predictive becomes asymptotically equivalent to the plug-in predictive, uniformly over any compact interval of positive temperatures. The paper formalizes this with total-variation and Kullback–Leibler bounds and concludes that, in moderate-to-large samples, tempering does not impact posterior predictions. In this sense, data-point tempering is chiefly a finite-sample and robustness device rather than an asymptotic predictor optimizer (McLatchie et al., 2024).

A further methodological development studies analyticity of tempered expectations along continuous power-posterior paths,

$$p_T(\beta,z,x)\propto p(z,x\mid\beta)^{1/T}p(\beta),$$7

and shows that $$p_T(\beta,z,x)\propto p(z,x\mid\beta)^{1/T}p(\beta),$$8 is often real-analytic on $$p_T(\beta,z,x)\propto p(z,x\mid\beta)^{1/T}p(\beta),$$9. The same work does not discuss data-point tempering or partial posteriors by name, but it explicitly notes that its results apply to general geometric tempering $T$0 and sketches how data-fraction paths can be embedded into such continuous constructions. This suggests that extrapolation and smoothing of expectations along a data-tempering path may be possible whenever the path admits a continuous geometric parameterization, although that implication is not established as a theorem for discrete partial-posterior schemes (Xi et al., 15 Sep 2025).

Several limitations recur across the literature. Temperature selection is repeatedly identified as nontrivial: deterministic annealing requires a cooling schedule, importance tempering has no general closed form under spurious correlation, and sample-dependent calibration depends on the quality of a learned feature-space density model [(Mandt et al., 2014); (Lu et al., 2022); (Joy et al., 2022)]. Local tempering can also add approximation error or complexity: LVT breaks exact conjugacy and therefore uses an approximate global update; subsampling tempering improves cost but offers no universal guarantee of greater efficiency; per-example temperature learning in overparameterized discriminative models remains an open problem. These caveats delimit the concept’s current status. Data-point tempering is not a single algorithm but a unifying design principle: let the effect of data vary across observations, and use temperature to express that variation in a way that is computationally tractable and inferentially useful [(Mandt et al., 2014); (Meent et al., 2014); (Lu et al., 2022)].