Low Rank Bayesian Inference

• Low rank Bayesian inference is a collection of methods that leverages low-dimensional factorizations to mitigate the curse of dimensionality in high-dimensional statistical problems.
• Key techniques include low-rank surrogate modeling, constrained variational approaches, and tensor factorizations, all of which enhance both computational efficiency and stability.
• These methods find applications in high-dimensional PDEs, inverse problems, and deep learning, delivering significant computational savings and robust uncertainty quantification.

Low rank Bayesian inference refers to a collection of statistical techniques that exploit low-rank structure—typically in matrices or tensors—within the Bayesian inference paradigm. These methods have emerged as a key strategy for rendering Bayesian analysis tractable in high-dimensional or high-order settings, allowing for both efficient uncertainty quantification and scalable computation. The central concept is to restrict attention to a class of models or surrogates whose dependence on the underlying random variables is expressed via low-rank factorizations, separable representations, or subspace projections, thereby avoiding the curse of dimensionality and enabling uncertainty quantification in otherwise intractable problems.

1. Low-Rank Surrogate Modeling for Bayesian Inference

A central paradigm for efficient Bayesian inference in high-dimensional stochastic modeling is the construction of low-rank separated representation surrogates. The surrogate expresses a vector-valued stochastic function, $\mathbf{u}(\xi, \bm{y})$—where $\xi$ is a physical variable and $\bm{y} = (y_1,\ldots,y_d)$ a high-dimensional random input—as a sum of rank-one tensor products: $$\mathbf{u}(\xi, \bm{y}) = \sum_{l=1}^{r} s_l\, \mathbf{u}_0^l(\xi) \prod_{i=1}^d u_i^l(y_i) + \bm{\varepsilon},$$ where $r$ is the separation rank and each $u_i^l(y_i)$ is a univariate function represented in a spectral basis (e.g., polynomial chaos). All surrogate parameters are determined by regularized alternating least squares regression, incorporating Tikhonov regularization—specifically with a roughening gradient-based penalty matrix—to enforce stability and suppress overfitting. A perturbation-based error indicator (PEI) is defined to assess model sensitivity and thereby select the optimal pair $(r, M)$, where $M$ is the truncation level of the univariate polynomial expansion.

These surrogate models dramatically reduce computational cost in Monte Carlo sampling: the number of training samples grows linearly with dimension $d$, and the computational complexity is quadratic in $d$. Application to high-dimensional PDEs (e.g., a 41-dimensional elliptic equation and a 21-dimensional stochastic cavity flow) demonstrates relative errors of $10^{-3}$ in mean/standard deviation estimates and order-of-magnitude speedup over scalar surrogates (Validi, 2013).

2. Constrained and Hierarchical Bayesian Low-Rank Inference

Bayesian low-rank inference techniques frequently capitalize on constrained formulations and structured priors that promote low-rankness:

• Constrained Variational Inference: Optimization-based approaches (e.g., matrix-variate multitask regression) impose low-rank constraints using the nuclear norm, typically in the form $\|W\|_* = \sum_i \sigma_i(W)$ for a weight matrix $W$. Constrained Bayesian inference is implemented via parametric optimization over the exponential family, allowing nonconvex constraint sets and sidestepping dual Fenchel formulations. Nuclear norm–constrained posteriors are coupled with precision matrix estimation for both rows and columns, incorporating $L_1$ penalties to recover sparse conditional independence structures (Koyejo et al., 2013).
• Hierarchical Priors: Latent variable models for low-rank matrix learning adopt Gaussian priors with precision (inverse covariance) matrices that are themselves random variables with Wishart or Gamma hyperpriors. Marginalizing over these yields heavy-tailed priors on the latent matrix (e.g., log-determinant, Schatten-$s$ or nuclear norm penalties),

$p(X) \propto |W^{-1} + X X^T|^{-(\nu+N)/2},$

directly promoting sparsity in the singular value spectrum. Efficient variational inference (with GAMP) or EM schemes enable learning of both the low-rank factors and the precision structure (Yang et al., 2017, Sundin et al., 2015).

3. Low-Rank Bayesian Inference in Inverse Problems and Covariance Approximation

For Bayesian inference on high-dimensional PDE-constrained inverse problems, direct computations with the posterior covariance, $(H + \Gamma_{\rm pr}^{-1})^{-1}$, are infeasible. A key insight is that the data are often informative only in a low-dimensional subspace. This motivates optimal low-rank corrections to the prior covariance: $$\widehat{\Sigma}_{\rm post} = \Gamma_{\rm pr} - \sum_{i=1}^r \frac{\delta_i^2}{1+\delta_i^2} w_i w_i^T,$$ where $(\delta_i^2, w_i)$ are the top generalized eigenpairs of the Hessian/prior pencil $(H, \Gamma_{\rm pr}^{-1})$. Such an update is theoretically optimal for a wide class of loss functions, including KL divergence and the Förstner metric (Spantini et al., 2014). The posterior mean admits an analogous low-rank affine approximation in terms of the data.

Efficient implementation in large-scale problems is enabled by low-rank decomposition of the Hessian through partial eigendecomposition, randomized SVD, or Arnoldi/Lanczos iteration using matrix–vector products facilitated by forward/adjoint PDE solves (Benner et al., 2017, Brown et al., 2016). For space–time PDEs, rank truncation allows complexity reduction from $\mathcal{O}(n_x n_t)$ to $\mathcal{O}(n_x + n_t)$. These advances make sampled or marginal posterior inference feasible in high-dimensional Bayesian inverse problems (e.g., X-ray tomography, image deblurring, NMR relaxometry) and enable rapid uncertainty quantification.

4. Bayesian Low-Rank Tensor Factorization and High-Order Structure

Several Bayesian low-rank techniques address structured data in the form of tensors (order-3 or higher):

• Bayesian Tensor Decompositions: Tensor ring (TR) or higher-order tensor SVD (t-SVD) decompositions are imposed with hierarchical sparsity-inducing (ARD) priors, often leading to core parameters following Student-T distributions. This automatic relevance determination "prunes" unnecessary latent components, infers the tensor multi-rank, and addresses overfitting (Long et al., 2020, Liu et al., 2023). Bayesian learning proceeds via variational inference, which updates latent factors, sparsity parameters, and noise models simultaneously.
• Bayesian Alternating Linear Scheme: A probabilistic reinterpretation of the alternating linear scheme (ALS) in tensor decompositions models each core as a Gaussian posterior. With a block-coordinate approach, measurement noise and prior knowledge are incorporated, and uncertainty quantification is achieved via the unscented transform in tensor train format (Menzen et al., 2020).
• Robustness and Exact Rank Selection: For robust principal component analysis in high-order tensors, explicit modeling of both sparse and dense noise components is combined with t-SVD–based multi-rank factorization. Variational Bayesian inference with ARD priors allows fine-grained adaptation to the intrinsic structure of the data, resulting in superior denoising and inpainting performance in multi-modal imaging tasks (Liu et al., 2023).

5. Low-Rank Bayesian Inference in Deep Learning and LLMs

Recent advances apply low-rank Bayesian strategies to large neural networks, particularly for scalable uncertainty quantification in LLMs and vision models:

• Low-Rank Adaptation (LoRA) with Bayesian Inference: Rather than placing Bayesian priors over all network parameters, inference is conducted in the subspace defined by LoRA parameters (low-rank matrices inserted into specific layers). Gaussian stochastic weight averaging (SWAG) and variance-reduced ensemble sampling provide fast, tractable posterior approximations over these parameters, significantly improving calibration and out-of-distribution (OOD) robustness with minimal computational overhead (Onal et al., 6 May 2024).
• Stochastic Variational Subspace Inference: ScalaBL demonstrates that performing Bayesian inference in the $r$-dimensional subspace of LoRA rank-$r$ parameters (where $r$ is typically $\ll$ full embedding size) allows scaling to the largest LLMs to date, with uncertainty quantification achieved by mapping Gaussian noise in the latent space through learned projection matrices into the full weight space (Samplawski et al., 26 Jun 2025).
• Low-Rank Bayesian Ensembles: Bella approximates the Bayesian posterior for neural networks through low-rank perturbations on a fixed pre-trained network, learning only the low-rank factors via SVGD or deep ensembles. The parameter count is reduced from $O(d_1d_2)$ to $O(r(d_1+d_2))$ per particle, facilitating practical deep Bayesian learning on large-scale tasks (ImageNet, multimodal VQA) (Doan et al., 30 Jul 2024).

6. Statistical Guarantees, Uncertainty Quantification, and Rank Robustness

Low-rank Bayesian inference models, whether based on surrogates, tensor decompositions, or matrix factorization, can enable principled uncertainty quantification and statistical inference. Approaches such as BayeSMG parametrize low-rank matrices via their subspaces and sample directly on manifolds (Stiefel/Grassmann), yielding interpretable and measurable posterior uncertainty over both singular values and subspaces (Yuchi et al., 2021). Regularization using heavy-tailed priors (e.g., spectral scaled Student, log-determinant penalties) ensures that uncertainty conferred on the estimated rank structure is properly propagated (Mai, 2021).

Recent work in inference for low-rank models demonstrates that valid inference for linear functionals of low-rank matrices can be achieved without consistent rank estimation. By using diversified projections and analytic bias correction to account for over-specified rank (leading to implicit ridge-type regularization), the central limit theorem for inference is preserved as long as the working rank exceeds the true rank (Choi et al., 2023). This property is especially valuable in panel data, matrix completion, and causal inference contexts with uncertain or weakly identifiable latent dimension.

7. Impact, Computational Tradeoffs, and Applications

The techniques described achieve significant computational savings and scalability. Surrogate models tame exponential growth to quadratic or linear scaling in problem dimension, while online/offline decomposition strategies and structured priors admit efficient inference even in very high-dimensional or high-order data regimes. Bayesian low-rank inference has seen impactful applications across uncertainty quantification for physical systems (elliptic PDEs, tomography), multitask regression (neuroimaging), recommendation systems, matrix completion, high-order tensor data recovery, high-stake natural language processing, and scalable deep learning with calibrated uncertainty.

Limitations include sensitivity to the validity of the low-rank assumption, potential issues with rank adaptivity in regimes with weak signal, and the challenge of extending efficient inference to non-Gaussian, hierarchical, or highly nonlinear models. Nonetheless, the field continues to make advances in integrating low-rank structure, hierarchical Bayesian priors, and variational approximations, ensuring that state-of-the-art Bayesian inference remains computationally feasible across the spectrum of high-dimensional applications.