---
title: Last-Layer Laplace Approximation
url: https://www.emergentmind.com/topics/last-layer-laplace-approximation
type: topic
---

# Last-Layer Laplace Approximation

The last-layer Laplace approximation is a methodology in approximate Bayesian inference where the Bayesian treatment is restricted solely to the final (linear) layer of a deep neural network, with all earlier layers held fixed at their maximum a posteriori (MAP) estimates. The technique yields a tractable Gaussian approximation for the posterior over the last-layer weights, enabling efficient uncertainty quantification and fast Bayesian updates—especially advantageous in scenarios where full network retraining is computationally prohibitive. It has become a key tool for scalable Bayesian deep learning, efficient incremental learning, active learning query strategies, and conformal prediction with calibrated uncertainty [2210.06112, 2512.01384, 2403.10671].

## 1. Bayesian Setup and Motivation

Given network parameters $\omega \in \mathbb{R}^P$ and dataset $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$, the Bayesian posterior is $p(\omega|\mathcal{D}) \propto p(\omega) \prod_{i=1}^N p(y_i|x_i, \omega)$. For modern networks, $P$ is typically large, rendering posterior inference computationally intractable. The last-layer Laplace approach freezes the representation parameters $\theta$ (all layers except the last), and focuses on the posterior over the final weights $w \in \mathbb{R}^D$, often after a feature extraction step $\varphi(x) \in \mathbb{R}^D$. The marginal posterior then simplifies to $p(w | \mathcal{D}, \theta) \propto p(w) \prod p(y_i | \varphi(x_i), w)$, reducing parameter dimensionality from $P$ to $D$ [2210.06112, 2512.01384, 2403.10671].

## 2. Derivation of the Gaussian Posterior (Laplace Approximation)

The last-layer Laplace approximation proceeds by expanding the log posterior density $\log p(w|\mathcal{D})$ to second order around its mode (MAP estimate) $\mu$:

- **MAP estimation:** $\mu = \arg\max_w \left[\log p(w) + \sum_{i=1}^N \log p(y_i|\varphi(x_i), w)\right]$  
  With a Gaussian prior $p(w) = \mathcal{N}(w; 0, \lambda^{-1}I)$ and suitable likelihood (e.g., Gaussian for regression; logistic for binary classification).

- **Hessian computation:** Negative log-posterior Hessian at $w = \mu$ is  
  $H := -\nabla^2_w \log p(w | \mathcal{D})|_{w=\mu}$  
  For a regression model with homoscedastic noise $\sigma^2$, $H = (1/\sigma^2)\Phi^\top \Phi + \lambda I$, where $\Phi$ is the design matrix [2512.01384]. For logistic regression, $H = \lambda I + \sum_i g_i \varphi(x_i)\varphi(x_i)^\top$, with $g_i = \sigma(\varphi_i^\top \mu)(1 - \sigma(\varphi_i^\top \mu))$.

- **Gaussian approximation:** The posterior is approximated as $q(w) = \mathcal{N}(w; \mu, \Sigma)$, with $\Sigma = H^{-1}$ [2210.06112, 2512.01384, 2403.10671].

## 3. Efficient Hessian Inversion and Fast Bayesian Updates

- **Low-rank updates:** Direct inversion of $H \in \mathbb{R}^{D \times D}$ costs $O(D^3)$. However, $H$ is a low-rank modification of the prior precision, so the Sherman–Morrison–Woodbury identity or sequential rank-one updates permit inversion in $O(N D^2)$ or $O(D^2)$ per update:
  - Initialize $\Sigma_0 = (\lambda I)^{-1}$.
  - Sequentially update with each rank-one term:  
    $\Sigma_i = \Sigma_{i-1} - g_i (\Sigma_{i-1}\varphi_i)(\Sigma_{i-1}\varphi_i)^\top / \beta_i$ with $\beta_i = 1 + g_i \varphi_i^\top \Sigma_{i-1} \varphi_i$.
  - For batch updates with new data, compute posterior update formulas for mean and covariance [2210.06112].

- **Incremental updates:** Given prior $q(w|\mathcal{D}) = \mathcal{N}(\mu, \Sigma)$ and new data batch $\mathcal{D}^+$, perform a single Laplace step to yield new mean and covariance:
  - $\Delta = \sum_{(x, y) \in \mathcal{D}^+} (\sigma(\varphi(x)^\top\mu) - y)\varphi(x)$
  - Update $\Sigma^+$ and $\mu^+$ using the rank-one formulae; iterate as needed for numerical stability.
  - Prediction at $x$ involves computing $\varphi(x)^\top \Sigma \varphi(x)$ [2210.06112].

- **Computational advantage:** Last-layer Laplace avoids retraining the full model or last layer entirely. Updates can be $10$–$100\times$ faster than gradient-based retraining for moderate $M$ and $D$; prediction is $O(D^2)$ versus multiple forward passes for MC methods [2210.06112].

## 4. Predictive Distributions and Uncertainty Decomposition

- **Posterior predictive:** For new $x$ with feature map $\varphi(x)$, the predictive distribution is  
  $p(y | x, \mathcal{D}) = \mathcal{N}(\varphi(x)^\top \mu, \sigma^2 + \varphi(x)^\top \Sigma \varphi(x))$
  for regression [2512.01384].  
  The mean $\varphi(x)^\top \mu$ is the standard point prediction; the variance decomposes into
    - $\sigma^2$: aleatoric (data) uncertainty,
    - $\varphi(x)^\top \Sigma \varphi(x)$: epistemic (model) uncertainty.

- **Diagnostics:** Fractional epistemic share $r(x) = \varphi(x)^\top \Sigma \varphi(x)/(\sigma^2 + \varphi(x)^\top \Sigma \varphi(x))$ and overall posterior spread $\mathrm{tr}(\Sigma)$ are useful diagnostics for model uncertainty and interval tightness [2512.01384].

## 5. Applications: Incremental Learning, Active Learning, and Conformal Prediction

| Application Area                | Role of Last-Layer Laplace Approximation            | Cited Paper     |
|---------------------------------|-----------------------------------------------------|-----------------|
| Incremental/Online Learning     | Enables closed-form Bayesian updates on arrival of new data batches, faster than retraining | [2210.06112]    |
| Active Learning                 | Permits fully sequential acquisition strategies (e.g., Uncertainty Sampling, BALD, Query-by-Committee) with per-point model update and uncertainty estimation | [2210.06112]    |
| Conformal Prediction            | Used in CLAPS to yield efficiency-improved conformal intervals with split-conformal calibration; intervals adapt to epistemic uncertainty | [2512.01384]    |
| Uncertainty Quantification      | Provides fast, scalable Gaussian posterior over outputs for downstream decision-making | [2403.10671]    |

**Active learning context:** The method enables the use of computationally intensive acquisition functions (e.g., BALD) after each new label is acquired, rather than in large pre-selected batches. Empirically, SNGP-LA yields 2–5% better learning curves than traditional top-batch-based batch selection [2210.06112].  
**Conformal prediction context:** CLAPS combines the Gaussian posterior from last-layer Laplace with a two-sided posterior CDF conformity score, producing tighter prediction intervals with finite-sample coverage, particularly effective when epistemic variance is nontrivial [2512.01384].

## 6. Implementation Notes and Practical Considerations

- **Cholesky and Hessian-vector products:** For width $d$ final-layer, Cholesky-based inversion is feasible unless $d$ is extremely large. For large $d$, conjugate-gradient (CG) methods can be used for Hessian solves [2403.10671, 2512.01384].
- **Curvature approximations:** In practice, the Gauss–Newton approximation is often used for the Hessian [2403.10671]. For further efficiency, diagonal, Kronecker-factored, or low-rank approximations can be applied.
- **Numerical stability:** Additional damping is applied to ensure positive-definiteness of the Hessian; fallback strategies are employed when the Hessian is ill-conditioned [2403.10671].
- **Scalability:** The technique is computationally viable in settings where inverting the full-network Hessian is infeasible—applicable for last-layer dimensions up to several thousand.

## 7. Limitations and Theoretical Considerations

- **Representation uncertainty ignored:** Only last-layer weights are treated Bayesian. Uncertainty in the backbone is not captured; the method assumes feature extractor parameters are well-optimized and certain [2210.06112, 2512.01384].
- **Posterior contraction:** As the data set size $n$ increases, epistemic uncertainty diminishes ($\varphi(x)^\top \Sigma \varphi(x) \to 0$), so benefits over mean-based or residual-based methods vanish [2512.01384].
- **Homogeneity assumptions:** Many approaches assume homoscedastic Gaussian noise; strongly heteroscedastic settings require explicit modeling via scale or quantile heads [2512.01384].
- **Fidelity vs. Scalability trade-off:** High modeling fidelity would require full Laplace approximations over all parameters, which is computationally prohibitive. Last-layer Laplace trades off some fidelity for tractability [2403.10671].
- **Accuracy depends on feature representation:** The posterior variance is only meaningful to the extent that the frozen representation $\varphi(x)$ is expressive and robust [2512.01384].

## References

- “Efficient Bayesian Updates for Deep Learning via Laplace Approximations” [2210.06112]
- “CLAPS: Posterior-Aware Conformal Intervals via Last-Layer Laplace” [2512.01384]
- “Variation Due to Regularization Tractably Recovers Bayesian Deep Learning” [2403.10671]
- “An Extended Simplified Laplace strategy for Approximate Bayesian inference of Latent Gaussian Models using R-INLA” [2203.14304]

Source: https://www.emergentmind.com/topics/last-layer-laplace-approximation