Corrected Integrated Laplace Approximation for Bayesian Inference in Latent Gaussian Models

Abstract: Latent Gaussian models (LGMs) are a popular class of Bayesian hierarchical models that include Gaussian processes, as well as certain spatial models and mixed-effect models. Efficient Bayesian inference of LGMs often requires marginalizing out the latent variables. For LGMs with a non-Gaussian likelihood, exact marginalization is not possible and a popular approach is to do approximate marginalization with an integrated Laplace approximation (ILA). Using ILA produces an approximate posterior which, in some settings, can differ significantly from the correct posterior, which impacts downstream applications. We propose an importance sampling scheme to correct the error introduced by ILA. By increasing the number of samples in importance sampling, the posterior with ILA converges to the correct posterior. This idea is realized with various techniques, including pseudo-marginalization, quasi-Monte Carlo and randomized quasi-Monte Carlo. We implement our methods in an automatic differentiation framework to support gradient-based algorithms when doing inference on the hyperparameters. For the latter, we specifically consider the use of Hamiltonian Monte Carlo. We demonstrate the benefits of reduced error in various applied models.

• The paper introduces unbiased estimators (PM-ADLA, QMC-ADLA, RQMC-ADLA) that correct posterior bias from traditional Laplace approximations in latent Gaussian models.
• It leverages importance sampling combined with quasi-Monte Carlo methods to achieve asymptotic unbiasedness and reduced variance in high-dimensional settings.
• Empirical results demonstrate robust performance in bias reduction and improved effective sampling across various complex hierarchical models.

Corrected Integrated Laplace Approximation for Bayesian Inference in Latent Gaussian Models

Overview

This paper ("Corrected Integrated Laplace Approximation for Bayesian Inference in Latent Gaussian Models" (2605.20345)) addresses the persistent issue of posterior approximation error inherent to the Integrated Laplace Approximation (ILA) for latent Gaussian models (LGMs). The work proposes a set of novel algorithms that leverage importance sampling, quasi-Monte Carlo (QMC), and randomized quasi-Monte Carlo (RQMC) methodologies to correct the bias introduced by Laplace-based marginalization. Crucially, the methods combine efficiency, automated differentiation for gradient-based inference (notably HMC), and asymptotic correctness, providing practical solutions for high-dimensional and non-Gaussian likelihood scenarios.

Integrated Laplace Approximation and Its Limitations

ILA and its instantiations such as INLA have become standard tools for LGMs with non-Gaussian likelihoods, enabling approximate marginalization of high-dimensional latent variables $z$, thus reducing the complexity of the inference problem to the hyperparameters $\theta$. Automatic differentiation enables scalable gradient computation for HMC targeting the approximate marginal posterior, as in ADLA. However, the Laplace approximation can yield significant error in the marginal posterior, which is neither controlled nor vanishes with increased sampling. The only unbiased alternative—exact MCMC over the full joint $(\theta,z)$—is computationally prohibitive for high-dimensional $z$.

Correcting Laplace Marginalization: Importance Sampling Framework

The central contribution is the recognition that the marginal likelihood approximation produced by ILA can be interpreted as an importance sampling estimator with a single sample, and thus increased samples allow convergence to the correct posterior. The authors design unbiased estimators:

• Pseudo-Marginal ADLA (PM-ADLA): Samples auxiliary Gaussian variables and targets an extended posterior using pseudo-marginal techniques.
• QMC-ADLA: Incorporates low-discrepancy sequences (e.g., Sobol) to reduce importance sampling variance and bias in the marginal estimator.
• RQMC-ADLA: Unifies QMC and pseudo-marginalization, yielding unbiasedness and computational efficiency without increasing dimensionality.

These estimators are amenable to reverse-mode automatic differentiation (JAX implementation), enabling scalable inference via gradient-based samplers on extended or modified posteriors.

Numerical Performance and Bias Correction

Extensive experiments demonstrate the strengths and tradeoffs of the proposed methods across LGMs, including Gaussian processes, sparse kernel interaction models, and mixed-effects models. All methods eliminate divergent transitions present with direct HMC on the unmarginalized model, maintaining robust geometry in the posterior. However, naive ILA-based HMC (ADLA) shows persistent bias, particularly for parameters associated with the latent structure.

Figure 1: Error convergence for estimating means of parameters in a Poisson Gaussian process, highlighting the reduced posterior error of PM-ADLA, QMC-ADLA, and RQMC-ADLA relative to ADLA and HMC.

Figure 2: Comparative running time and effective sample size per minute, illustrating the tradeoff between computational throughput and posterior correctness across methods.

Figure 3: Bias correction in a sparse kernel interaction model; error curves demonstrate that increased QMC samples and RQMC effectively minimize approximation bias.

Figure 4: Posterior distributions compared to ground truth for challenging latent parameters, evidencing the improved tail behavior of corrected estimators.

Theoretical Guarantees and Algorithmic Implications

The paper provides formal proofs of unbiasedness and asymptotic convergence of the corrected estimators to the true marginal posterior. The error as a function of the number of importance samples decays as $\mathcal{O}(n^{-1})$ (IS) or $\mathcal{O}(n^{-2})$ (QMC, under suitable smoothness conditions). RQMC further reduces variance while maintaining computational efficiency. Posterior recovery for latent variables $z$ is refined by reweighting Laplace samples, yielding asymptotically correct joint samples $(\theta, z)$.

RQMC-ADLA, while continuous in $\theta$, is discontinuous in the QMC shift variable due to the modulo operator—a limitation that the authors circumvent using Metropolis-within-Gibbs sampling, with performance validated numerically.

Practical and Theoretical Implications

The corrected ILA methodology provides a robust pathway for scalable Bayesian inference in complex hierarchical models, especially where exact marginalization is infeasible. The reduction of approximation bias allows the deployment of automatic differentiation and gradient-based samplers in probabilistic programming with practical guarantees on posterior correctness, eliminating the need for model-specific Laplace error diagnostics. The variance estimates from importance sampling can give on-the-fly diagnostic feedback.

Figure 5: Error convergence for various corrected methods in a Poisson Gaussian process, reinforcing the theoretical predictions on bias decay with increasing $n$.

Future Directions

Extensions include:

• Application to latent models beyond Gaussian structure, requiring new proposal distributions and differentiation routines.
• Integration with variational inference, as demonstrated in recent work in the presence of symmetry.
• High-performance implementation in widespread probabilistic programming frameworks (Stan, TMB, BlackJAX).
• Leveraging importance sampling variance estimates for adaptive selection of $\theta$0 and model diagnostics.

Conclusion

The work rigorously defines, analyzes, and empirically validates corrected integrated Laplace approximation algorithms for Bayesian inference in latent Gaussian models. By incorporating importance sampling and QMC/RQMC, the methods bridge the gap between computational efficiency and posterior correctness, making them a valuable tool for high-dimensional and complex hierarchical Bayesian modeling. The theoretical guarantees and practical performance underscore their utility and suggest future avenues for generalizing to broader model classes and inference paradigms.