Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantification

Abstract: Approximate Bayesian inference typically revolves around computing the posterior parameter distribution. In practice, however, the main object of interest is often a model's predictions rather than its parameters. In this work, we propose to bypass the parameter posterior and focus directly on approximating the posterior predictive distribution. We achieve this by drawing inspiration from self-training within self-supervised and semi-supervised learning. Essentially, we quantify a Bayesian model's predictive uncertainty by refitting on self-predicted data. The idea is strikingly simple: If a model assigns high likelihood to self-predicted data, these predictions are of low uncertainty, and vice versa. This yields a deterministic, sampling-free approximation of the posterior predictive. The modular structure of our Self-Supervised Laplace Approximation (SSLA) further allows us to plug in different prior specifications, enabling classical Bayesian sensitivity (w.r.t. prior choice) analysis. In order to bypass expensive refitting, we further introduce an approximate version of SSLA, called ASSLA. We study (A)SSLA both theoretically and empirically in regression models ranging from Bayesian linear models to Bayesian neural networks. Across a wide array of regression tasks with simulated and real-world datasets, our methods outperform classical Laplace approximations in predictive calibration while remaining computationally efficient.

• The paper introduces SSLA and ASSLA methods that replace Monte Carlo sampling with a deterministic Laplace-style approximation for the posterior predictive distribution.
• It decomposes the log-posterior-predictive into interpretable increments, providing theoretical guarantees and enabling modular integration with various priors.
• Empirical results across synthetic and real-world regression tasks demonstrate improved calibration, reduced computational cost, and competitive performance compared to traditional approaches.

Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantification

Overview

This paper introduces the Self-Supervised Laplace Approximation (SSLA) and its approximate variant (ASSLA) as novel methods for deterministic, sampling-free Bayesian uncertainty quantification, focusing directly on approximating the posterior predictive distribution (PPD) rather than parameter posteriors. This approach leverages a self-training mechanism: predictions are assessed by their own refit likelihood, with low-likelihood self-predicted data identified as high-uncertainty regions. The core innovation is the bypassing of explicit (often intractable) posterior computation, replacing Monte Carlo integration with analytic and highly modular Laplace-style approximations to the PPD. The methods provide theoretical guarantees, control over approximation error, and strong empirical performance across a range of regression settings where traditional Bayesian approximations are computationally prohibitive or poorly calibrated.

Methodological Contributions

The SSLA decomposes the log-posterior-predictive into interpretable increments: log-likelihood, log-prior, and curvature (log-determinant of Fisher information), evaluated at both the original fit and a model refit on the self-predicted label. The main idea is to quantify uncertainty by the incremental change induced by augmenting the training data with the self-predicted sample, eschewing any unwarranted independence assumptions between the prediction and existing data.

ASSLA further approximates this procedure by exploiting Lipschitz continuity and local linearization, establishing that the new pseudo-optimum induced by the augmented sample can be replaced by the original MAP estimate with vanishing error as the sample size grows. This critical reduction eliminates expensive re-optimization, retaining leading-order likelihood and curvature effects, and results in a deterministic, sampling-free method whose computational cost rivals classical plug-in Laplace approximations.

The methods are inherently modular with respect to the prior, enabling sensitivity analysis and integration into credal Bayesian deep learning (CBDL) settings and more general robust Bayesian inference paradigms.

Theoretical Guarantees

The analysis rigorously establishes the consistency and approximation bounds under standard regularity conditions. Lemma 1 shows that the difference between the original and refit optima is $O(n^{-1})$; Theorem 1 and its corollaries extend this to show that the likelihood and curvature increments dominate the approximation error, while the prior increment is subleading and can be neglected in ASSLA.

This results in precise analytic control over the deterministic approximation to the PPD. Critically, the justification holds for both classical parametric and overparameterized deep learning regimes, given bounded curvature and Lipschitz continuity, and is agnostic to the functional form of the model family.

Numerical Results

The paper offers extensive empirical validation:

• Conjugate Models: On analytic PPDs (normal-normal, Poisson-gamma, conjugate linear regression), both SSLA and ASSLA yield numerical accuracy nearly indistinguishable from true analytic posteriors for datasets with $20 \leq n \leq 10^5$, with entropy differences close to zero. Numerical instabilities in ASSLA for $n \geq 10^6$ are attributed to finite-precision floating-point cancellation rather than methodological flaws.
• Synthetic Heteroscedastic Regression: On controlled neural regression tasks with input-dependent variance, ASSLA displays tight, sharp credible regions, outperforming classical Laplace—which is overly conservative—on negative log-likelihood (NLL) and continuous ranked probability score (CRPS), though displaying mild undercoverage at the widest intervals. SSLA and variational inference deliver competitive calibration; Hamiltonian Monte Carlo (HMC), while theoretically correct, is computationally costly and can underperform in empirical calibration due to practical linearization limitations.

Method	Coverage 95%	Coverage 75%	NLL (↓)	CRPS (↓)	Computational Cost
ASSLA	High	High–Mid	Low	Low	Minimal
SSLA	High	High	Moderate	Low	Moderate–High
Laplace	Maximal	Maximal	High	High	Minimal
VI	Balanced	Balanced	Very Low	Very Low	Moderate–High
HMC	Slightly Low	Moderate	Moderate	Moderate	Very High

• Real-World Regression: On UCI ML Repository datasets, ASSLA yields competitive or superior calibration and sharpness compared to both SSLA and classical Laplace. Overconcentration (undercoverage) does occur in low signal-to-noise or high prior-data conflict settings, exposing scenarios where fallback to more conservative strategies is warranted. Otherwise, ASSLA consistently offers better-calibrated, sharper uncertainty regions, with reduced NLL and CRPS across diverse domains (Auto MPG, Concrete Strength, Wine Quality, etc.).

Implications and Future Directions

This work has both practical and theoretical implications. By decoupling posterior predictive approximation from Monte Carlo sampling, (A)SSLA provides a scalable route to uncertainty quantification in high-dimensional Bayesian deep learning, where traditional sampling or variational methods are prohibitively costly or fail to deliver well-calibrated uncertainties.

The prior modularity enables straightforward integration with recent advances in imprecise or credal probability frameworks, supporting multi-prior and robust Bayesian inference even in combinatorially complex prior specification scenarios.

Future refinements may focus on:

• Further improving numerical stability in very large data regimes (addressing floating-point cancellation in curvature terms).
• Extending covariance structure approximations (e.g., block-diagonal, low-rank) to further scale the method.
• Application to out-of-distribution detection, model selection, or active learning, leveraging the efficient predictive uncertainty estimation.
• Integration with imprecise probability tools for robust model selection and sensitivity analysis.
• Analytic or hybrid strategies for full-network versus last-layer Bayesian correction.

Conclusion

Self-Supervised Laplace Approximation (SSLA) and its approximate form (ASSLA) represent a principled, deterministic alternative for Bayesian uncertainty quantification, especially suited to deep learning settings where computational and calibration limitations of existing methods are acute. Theoretical analysis underpins the validity of the approximation, and empirical evidence demonstrates competitive or improved calibration with substantial computational savings. Limitations persist in pathological data-prior regimes and for extremely large-scale inference, which motivates ongoing improvement. Nevertheless, the proposed approach substantially advances the tractability and fidelity of Bayesian predictive uncertainty estimation in modern machine learning workflows.

Reference: "Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantification" (2605.12208).