Rank-Deficient Prior Covariances

• Rank-deficient prior covariances are positive semi-definite matrices that lack full rank, restricting inferences to a low-dimensional subspace.
• They play a key role in high-dimensional Bayesian inverse problems by allowing stable posterior updates through active subspace projection.
• Model reduction techniques such as oblique projectors and balanced truncation leverage this structure to optimize computation and error bounds.

A rank-deficient prior covariance is a covariance matrix that is positive semi-definite but not full rank, and thus has a nontrivial null space. This structure arises naturally in many high-dimensional Bayesian inverse problems—particularly when the prior is obtained from a limited ensemble, or when the parametrization intentionally restricts possible solutions to a low-dimensional subspace. Rank-deficiency fundamentally alters how information from data and prior combine, shaping both the form of the posterior distribution and the algorithms needed to approximate or reduce its dimensionality.

1. Rank-Deficient Priors in Bayesian Inverse Problems

In the context of linear Bayesian inverse problems, suppose the unknown parameter vector $p \in \mathbb{R}^d$ has a Gaussian prior with mean zero and covariance $\Gamma_{\mathrm{pr}}$, possibly rank-deficient. When $\mathrm{rank}(\Gamma_{\mathrm{pr}}) = s < d$, the prior is supported only on an $s$-dimensional subspace—denoted $\mathrm{Ran}(\Gamma_{\mathrm{pr}})$. The Bayesian update, given linear observations $m = Gp + \epsilon$ (with forward operator $G$ and noise covariance $\Gamma_{\mathrm{obs}}$), yields a posterior that is also supported on $\mathrm{Ran}(\Gamma_{\mathrm{pr}})$. As a consequence, all inferences (means, credible intervals) are confined to this subspace, and directions in $\mathrm{Null}(\Gamma_{\mathrm{pr}})$ are unaffected by the data (2506.23892).

2. Formulation of Dimension and Model Reduction

Dimension reduction exploits the observation that, in high-dimensional linear inverse problems with limited data, the posterior is informed only in a low-dimensional subspace:

1. Oblique Projectors: The paper constructs an oblique projector $\Pi_r = W_r V_r^\top$ using left and right reduced bases ($W_r$, $V_r$), computed from a generalized eigenvalue problem of the form $PQW = W\Sigma^2$, where $P=\Gamma_{\mathrm{pr}}$ and $Q$ is the Fisher information ($G^\top \Gamma_{\mathrm{obs}}^{-1} G$).
2. Restriction to the Active Subspace: For rank-deficient $\Gamma_{\mathrm{pr}}$ with $\mathrm{rank}=s$, only the first $s$ eigenvectors correspond to nonzero eigenvalues, forming a basis $W_s, V_s$ for $\mathrm{Ran}(\Gamma_{\mathrm{pr}})$. The dimension reduction proceeds by projecting both prior and likelihood onto this $s$-dimensional subspace.
3. Reformulated Inverse Problem: The measurement model becomes $m = G\Pi_s p + \epsilon$, and, rewriting $p = W_s q$ with $q \in \mathbb{R}^s$, the effective prior for $q$ is full-rank. Posterior calculations—including means and covariances—are then carried out entirely in $\mathbb{R}^s$ (2506.23892).

A summary table of the main operators:

Notation	Description	Characteristic
$\Gamma_\mathrm{pr}$	Prior covariance	$d \times d$, rank $s < d$
$W_s$	Right reduced basis	$d \times s$
$V_s$	Left reduced basis	$d \times s$
$\Pi_s$	Projector onto $\mathrm{Ran}(\Gamma_\mathrm{pr})$	$W_s V_s^\top$

3. Theoretical Guarantees and Optimality

All approximation guarantees, such as optimality with respect to various risk metrics (e.g., the Förtner distance between the true and reduced posterior covariances, or Mahalanobis Bayes risk on the posterior mean), are established on the restricted $s$-dimensional subspace. Specifically:

• Optimal Low-Rank (OLR) Approximation: The OLR solutions constructed on $\mathbb{R}^s$ minimize both the Förtner metric and a Mahalanobis risk restricted to $\mathrm{Ran}(\Gamma_{\mathrm{pr}})$ (Corollary 3.1, Restricted Optimality).
• LIS Reconstruction Theorem: The reduced bases and approximations computed from the full $d$-dimensional problem coincide with those from the $s$-dimensional restriction (Theorem 3.2).

Thus, all optimality results known in the canonical full-rank case extend, with appropriate restriction, to the rank-deficient setting (2506.23892).

4. Model Reduction Strategies for Dynamical Systems

For Bayesian smoothing problems, the challenge is to efficiently compute posteriors on the initial condition of an evolution system (e.g., $dx/dt = Ax$, $x(0) = p$), given potentially high-dimensional and expensive models and possibly rank-deficient priors. Two reduction strategies are discussed:

• LIS-Balanced Truncation (LIS-BT): Adapts balanced truncation, using the prior covariance $\Gamma_\mathrm{pr}$ and an inference-oriented output Gramian. The state is reduced through a similarity transformation using the leading modes (dominant eigenvectors) arising from $(\Gamma_\mathrm{pr}, Q)$.
• Prior–Driven Balancing (PD-BT): Reinterprets the prior covariance via a square-root factor $L_{\mathrm{pr}}$ ($$\Gamma_{\mathrm{pr}} = L_{\mathrm{pr}} L_{\mathrm{pr}}^\top$$), treating the unknown initial state as an impulse input to a lifted system. Impulse response computations and balanced truncation are then performed in this lower-dimensional subspace.

For both strategies, approximation error bounds are provided in terms of the residual Hankel singular values not retained in the reduction. For example, the expected $L_2$ error in the output impulse response of the reduced model is bounded by $2\sqrt{s} \sum_{k=r+1}^d \sigma_k$ (2506.23892).

5. Numerical and Practical Implications

Experimental results—using benchmark structural dynamics models and both compatible and incompatible sample-based priors—demonstrate:

• OLR dimension reduction achieves the smallest errors for the posterior mean and covariance.
• PD-BT model reduction nearly attains OLR accuracy even for incompatible (rank-deficient and misaligned) priors, outperforming LIS-BT in such settings.
• Approximations and error bounds remain effective, provided computation is confined to the s-dimensional active subspace.

For practical inference, this means that one can:

• Formulate all posterior calculations directly within the range of the prior, avoiding numerical instability due to singular matrices.
• Achieve computational savings: instead of sampling in $d$ dimensions, all critical work happens in $s \ll d$.
• Design reduced-order forward models that provide efficient and accurate online prediction/evaluation for large-scale sequential or real-time data assimilation (2506.23892).

6. Applications and Algorithmic Workflow

The framework is broadly applicable to Bayesian inverse problems with priors estimated from ensembles, PCA/truncated SVD expansions, or other sources of low-rank structure. The general workflow is as follows:

1. Estimate or specify $\Gamma_{\mathrm{pr}}$ (possibly via samples).
2. Compute an $s$-dimensional basis $(W_s, V_s)$ for its range via eigendecomposition.
3. Project the inference problem into the active subspace.
4. Compute posterior mean/covariance using standard formulas, now with computational and storage cost governed by $s$ rather than $d$.
5. In dynamical systems, reduce forward models using LIS-BT or PD-BT, allowing efficient simulation and evaluation in the (typically dominant) informative directions.

The following pseudocode summarizes the dimension reduction step (for a linear observation model):

eigvals, W_s = eigh(Gamma_pr) s = (eigvals > tol).sum() W_s = W_s[:, :s] G_hat = G @ W_s Gamma_post_hat = inv(inv(W_s.T @ Gamma_pr @ W_s) + G_hat.T @ inv(Gamma_obs) @ G_hat) mu_post_hat = Gamma_post_hat @ (G_hat.T @ inv(Gamma_obs) @ m) mu_post = W_s @ mu_post_hat

7. Broader Implications and Limitations

By formulating all reduction, sampling, and model evaluation solely on the informed subspace of rank-deficient priors, this approach enables efficient Bayesian inference in high-dimensional and computationally intensive applications such as geosciences, engineering design, and environmental modeling. Theoretical guarantees ensure that no information is lost in directions where the prior is already uninformative.

Potential limitations include the necessity of explicit basis construction for the active subspace (which may be costly for very large $d$ or when $\Gamma_{\mathrm{pr}}$ has no simple factorization), and the challenge of handling cases where the rank of the prior is not well separated from the full dimension—though the framework accommodates these situations through extensions and the analysis of approximation error.

In summary, dimension and model reduction methods for linear Bayesian inverse problems with rank-deficient prior covariances provide both theoretical foundation and practical strategy for leveraging low-dimensional structure in large-scale inference, allowing for accurate approximation of posterior statistics within the constraints imposed by the prior’s support (2506.23892).