Papers
Topics
Authors
Recent
Search
2000 character limit reached

MMD-Balls as Credal Sets: A PAC-Bayesian Framework for Epistemic Uncertainty in Test-Time Adaptation

Published 20 May 2026 in cs.LG and stat.ML | (2605.21783v1)

Abstract: Test-time adaptation (TTA) methods improve model performance under distribution shift but lack formal guarantees connecting shift magnitude to prediction reliability. We develop a PAC-Bayesian framework yielding generalization bounds explicitly parameterized by the maximum mean discrepancy (MMD) between source and target distributions. Our principal contribution is interpreting MMD-balls around the source distribution as credal sets in Walley's imprecise probability theory, yielding natural epistemic uncertainty quantification. We establish: (i) a PAC-Bayesian bound with an MMD-dependent shift penalty under an RKHS-Lipschitz loss assumption; (ii) a finite-sample version via MMD concentration; (iii) a uniform worst-case risk bound over all distributions in the credal set, with a lower-upper risk decomposition; and (iv) geodesic preservation bounds explaining why kernel-guided adaptation protects local feature geometry. The credal set interpretation separates epistemic from aleatoric uncertainty and provides a principled decision criterion for when adaptation is warranted.

Authors (1)

Summary

  • The paper develops a PAC-Bayesian framework that explicitly quantifies epistemic uncertainty via Maximum Mean Discrepancy (MMD) in test-time adaptation (TTA) whilst providing a new safety guarantee.
  • The framework introduces a credal set interpretation where the credibility set of a distribution is defined by its MMD to the source distribution as per imprecise probability theory, leading to an MMD-shift penalty.
  • The proposed PAC-Bayesian bound yields a worst-case risk certificate over the credal set which gives a principled method to determine when TTA is appropriate and links MMD to risk tolerance via a coverage level.

This paper develops a PAC-Bayesian framework for test-time adaptation (TTA) that produces generalization bounds explicitly parameterized by the maximum mean discrepancy (MMD) between source and target distributions, and interprets MMD-balls around the source distribution as credal sets in Walley's imprecise probability theory (2605.21783). The central motivation is that existing TTA methods—TENT, EATA, SAR, MEMO and successors—improve accuracy under distribution shift but provide no formal guarantee about when predictions should be trusted or how risk scales with shift magnitude. The paper's stated contribution is to fill this gap by connecting PAC-Bayesian generalization, kernel mean embeddings, and imprecise probability theory within a single analysis.

Setting and assumptions

The framework assumes covariate shift: the target preserves the source conditional Pt(y∣x)=Ps(y∣x)P_t(y \mid x) = P_s(y \mid x) while the marginal over features changes. Under this condition, the conditional expected loss L(w,x)=Ey∼P(y∣x)[ℓ(w,x,y)]L(w, x) = E_{y \sim P(y|x)}[\ell(w,x,y)] is the same function for both distributions, so only the measure over xx differs. The paper uses a learned RBF kernel on encoder outputs, kθ(x,y)=exp⁡(−γ∥fθ(x)−fθ(y)∥2)k_\theta(x, y) = \exp(-\gamma\|f_\theta(x) - f_\theta(y)\|^2), which is bounded in [0,1][0,1] and universal (hence characteristic) when the encoder Jacobian has full rank almost everywhere.

Two structural assumptions carry the main results. The first, an RKHS-Lipschitz loss assumption, requires that L(w,⋅)L(w, \cdot) belongs to the RKHS with norm bounded by a constant LHL_H. The paper concedes that this is stronger than smoothness: universality of the RBF kernel guarantees approximation but not bounded RKHS norm, and rigorous verification for deep networks with softmax cross-entropy remains open. The authors propose an empirical check via kernel ridge regression diagnostics (the squared norm α^⊤Kα^\hat{\alpha}^\top K \hat{\alpha} of the fitted loss function) and note that the proofs extend to a weaker posterior-averaged version, Ew∼ρ[∥L(w,⋅)∥H]≤LHE_{w \sim \rho}[\|L(w,\cdot)\|_H] \leq L_H, via Jensen's inequality. The second assumption, used for geometric results, requires the encoder to factor as fθ=W⋅ϕθf_\theta = W \cdot \phi_\theta with L(w,x)=Ey∼P(y∣x)[ℓ(w,x,y)]L(w, x) = E_{y \sim P(y|x)}[\ell(w,x,y)]0; the paper argues this holds approximately in the NTK regime, under MMD regularization, or under spectral normalization, reporting typical operator norms of 10–20 for spectrally normalized ResNet-50 as informal evidence.

The PAC-Bayesian bound with MMD shift penalty

The core result combines three steps: the classical PAC-Bayesian theorem applied to source data, a Cauchy–Schwarz argument bounding L(w,x)=Ey∼P(y∣x)[ℓ(w,x,y)]L(w, x) = E_{y \sim P(y|x)}[\ell(w,x,y)]1 using the reproducing property, and a union bound. With probability at least L(w,x)=Ey∼P(y∣x)[ℓ(w,x,y)]L(w, x) = E_{y \sim P(y|x)}[\ell(w,x,y)]2:

L(w,x)=Ey∼P(y∣x)[ℓ(w,x,y)]L(w, x) = E_{y \sim P(y|x)}[\ell(w,x,y)]3

The bound decomposes into three interpretable terms: the empirical source risk (aleatoric difficulty), the KL-based complexity penalty (estimation uncertainty), and a linear epistemic shift penalty. Two properties follow immediately: at zero shift the standard PAC-Bayesian guarantee is recovered exactly, and as shift grows the bound degrades linearly rather than collapsing. Because the shift penalty is computable given the kernel and monotone in shift magnitude, it functions as a direct epistemic uncertainty measure.

A finite-sample variant replaces the population MMD with the unbiased estimator computed from L(w,x)=Ey∼P(y∣x)[ℓ(w,x,y)]L(w, x) = E_{y \sim P(y|x)}[\ell(w,x,y)]4 source and L(w,x)=Ey∼P(y∣x)[ℓ(w,x,y)]L(w, x) = E_{y \sim P(y|x)}[\ell(w,x,y)]5 target samples, adding the concentration width L(w,x)=Ey∼P(y∣x)[ℓ(w,x,y)]L(w, x) = E_{y \sim P(y|x)}[\ell(w,x,y)]6. This achieves the minimax-optimal L(w,x)=Ey∼P(y∣x)[ℓ(w,x,y)]L(w, x) = E_{y \sim P(y|x)}[\ell(w,x,y)]7 rate for kernel mean embedding estimation established by Tolstikhin et al., so no uniformly faster rate is possible; practically, the bound tightens automatically as more target samples arrive during deployment. Every term is computable from available data, with the estimator costing L(w,x)=Ey∼P(y∣x)[ℓ(w,x,y)]L(w, x) = E_{y \sim P(y|x)}[\ell(w,x,y)]8.

Credal sets, worst-case risk, and uncertainty decomposition

The conceptual contribution is the interpretation of the MMD-ball L(w,x)=Ey∼P(y∣x)[ℓ(w,x,y)]L(w, x) = E_{y \sim P(y|x)}[\ell(w,x,y)]9 as a credal set: the family of distributions indistinguishable from the source at resolution xx0. For a characteristic kernel the set is convex and weakly closed—convexity follows from linearity of the mean embedding, closure from continuity of embeddings under weak convergence—and these properties are what make coherent imprecise-probabilistic reasoning applicable.

Within this structure, the paper establishes a uniform worst-case bound:

xx1

Because the right-hand side does not depend on the particular xx2, the same certificate covers every adversarial distribution inside the ball—a strictly stronger statement than a point estimate of target risk. A complementary lower bound, built on the PAC-Bayesian lower bound of Germain et al., yields a two-sided imprecision interval whose width satisfies

xx3

The interval supports a concrete decision rule for TTA: given a risk tolerance xx4, adaptation is warranted exactly when the interval straddles the threshold (xx5); if the lower risk already exceeds xx6, adaptation cannot help, and if the upper risk is below it, adaptation is unnecessary. The width itself separates epistemic from aleatoric components—the former comprising the estimation term xx7 and the shift penalty xx8—addressing directly the aleatoric/epistemic separation argued for by Hüllermeier and Waegeman. The paper also connects the choice of xx9 to the asymptotic null distribution of the MMD two-sample test, so rejecting kθ(x,y)=exp⁡(−γ∥fθ(x)−fθ(y)∥2)k_\theta(x, y) = \exp(-\gamma\|f_\theta(x) - f_\theta(y)\|^2)0 at level kθ(x,y)=exp⁡(−γ∥fθ(x)−fθ(y)∥2)k_\theta(x, y) = \exp(-\gamma\|f_\theta(x) - f_\theta(y)\|^2)1 calibrates the credal set radius.

Geodesic preservation and rare-class robustness

Under the bounded-feature-map assumption, the paper derives a local distortion bound for expected geodesic distance in the kernel-induced metric:

kθ(x,y)=exp⁡(−γ∥fθ(x)−fθ(y)∥2)k_\theta(x, y) = \exp(-\gamma\|f_\theta(x) - f_\theta(y)\|^2)2

using a local linear expansion of geodesic distance for nearby points and the reverse triangle inequality through the mean feature vectors. The stated implication is a formal argument for why MMD-bounded (kernel-guided) adaptation resists rare-class collapse: entropy minimization à la TENT can flatten low-density regions where rare classes reside, whereas the distortion bound controls geometric change independently of class frequency, since it depends only on population-level MMD. It should be noted this is a theoretical argument; the paper reports no experiments demonstrating rare-class robustness empirically.

Relation to conformal prediction

The paper distinguishes its framework from adaptive conformal inference: conformal methods quantify predictive (largely aleatoric) uncertainty at the level of individual prediction sets, while credal width quantifies distributional epistemic uncertainty at the population level. A proposed combination adapts the conformal coverage level as kθ(x,y)=exp⁡(−γ∥fθ(x)−fθ(y)∥2)k_\theta(x, y) = \exp(-\gamma\|f_\theta(x) - f_\theta(y)\|^2)3, with kθ(x,y)=exp⁡(−γ∥fθ(x)−fθ(y)∥2)k_\theta(x, y) = \exp(-\gamma\|f_\theta(x) - f_\theta(y)\|^2)4 calibrated from the fraction of the upper risk attributable to the shift penalty. The coverage properties of this hybrid are not analyzed formally and are flagged as future work, so this component remains a proposal rather than a result.

Limitations and open questions

Several limitations are acknowledged plainly. The RKHS-Lipschitz loss assumption is non-trivial for deep networks and unverified rigorously; the average-norm relaxation is easier to check but weakens the per-parameter guarantee. The bounded-feature-map assumption holds only approximately for standard architectures, and tighter architecture-specific bounds (e.g., for ResNets and Vision Transformers) remain to be developed. The kθ(x,y)=exp⁡(−γ∥fθ(x)−fθ(y)∥2)k_\theta(x, y) = \exp(-\gamma\|f_\theta(x) - f_\theta(y)\|^2)5 MMD concentration rate, though minimax-optimal, may be loose in practice; whether adaptive bandwidth selection yields tighter data-dependent rates is left open. The geodesic analysis relies on a second-order local approximation valid only in small neighborhoods. Finally, all results are theoretical: the paper contains no empirical evaluation of the bounds' tightness or of the proposed abstention and calibration criteria on benchmark TTA datasets.

Conclusion

This work provides a computable PAC-Bayesian generalization bound for TTA with an explicit linear MMD shift penalty, a minimax-optimal finite-sample version, and a uniform worst-case risk certificate over MMD-induced credal sets with a two-sided imprecision interval separating epistemic from aleatoric uncertainty. The credal interpretation supplies a principled criterion for deciding when adaptation is warranted and links kernel two-sample testing to imprecise probability. The strength of these guarantees rests on assumptions—the RKHS membership of the loss and the bounded feature-map factorization—that hold only approximately for realistic deep networks, and the absence of empirical validation means the practical tightness of the bounds remains an open empirical question.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.