On the design distribution for predictive Bayesian regression

Abstract: The predictive approach to Bayesian inference accesses the posterior distribution via a sequence of one-step-ahead predictives, enabling inference via predictive resampling without Markov chain Monte Carlo. In the random-design regression setting, an explicit specification of the predictive design distribution is required, yet the impact of this choice has received little formal attention. We study the role of this predictive design distribution in parametric martingale posteriors for regression, and identify predictive notions of identifiability and design invariance that are essential for valid inference, particularly in the high-dimensional regression setting. Building on these foundations, we introduce a novel class of parametric martingale posteriors for regression that satisfies a weak form of these desiderata, and naturally accommodates the high-dimensional setting through regularization. We then illustrate our method through a simulation.

• The paper introduces a novel recursive update that constructs parametric martingale posteriors to achieve design invariance in high-dimensional predictive Bayesian regression.
• It formalizes the role of the predictive design distribution, showing that identifiability ensures robust uncertainty quantification even when p > n.
• Empirical evaluations demonstrate that the method drastically reduces computational time while maintaining asymptotic normality and performance compared to MCMC approaches.

On the Design Distribution for Predictive Bayesian Regression

Introduction and Motivation

The paper "On the design distribution for predictive Bayesian regression" (2606.14544) addresses a core aspect of predictive Bayesian inference: the explicit specification and role of the predictive design distribution ($P_X$) in random-design regression. Classical Bayesian inference is invariant to the covariate distribution under standard assumptions, but the predictive approach, especially via predictive resampling and parametric martingale posteriors, requires choosing $P_X$ directly. Despite its fundamental role, the consequences and requirements of this choice have been underexplored, especially in high-dimensional regimes. This work formalizes the statistical implications of $P_X$ for valid inference, introduces predictive notions of identifiability and design invariance, and constructs a new class of parametric martingale posteriors that accommodate high-dimensionality via regularization.

Predictive Resampling and Martingale Posteriors

The predictive approach constructs posterior inference through a sequence of one-step-ahead predictive models, bypassing conventional Markov chain Monte Carlo (MCMC) in favor of predictive resampling. Specifically, given i.i.d. draws of future covariates from a specified $P_X$, responses are imputed from current predictive conditionals; the limiting distribution of a regression estimator forms the martingale posterior. Unlike traditional Bayes, this framework makes the modeling of $P_X$ explicit and operational.

A central insight is that when $P_X$ lacks identifiability (e.g., empirical $P_X$ when $p>n$), the induced martingale posterior can be invalid, even if convergence is achieved. Conversely, with a sufficiently rich $P_X$, the posterior distribution is theoretically invariant to the design (design invariance). This phenomenon is substantiated empirically and then formalized via a regression-specific Doob's theorem.

Figure 1: Posteriors for one component of $\beta$ through predictive resampling under different design distributions: empirical, true, and standard normal.

Identifiability and Design Invariance

To guarantee validity of predictive resampling, the parameter $P_X$0 must be identifiable under $P_X$1. The empirical $P_X$2 is singular when $P_X$3, violating the identifiability condition and resulting in posteriors that systematically underestimate uncertainty. Under mild regularity—a positive-definite expected information matrix for $P_X$4 (Assumption~2.2)—identifiability is restored.

If $P_X$5 is identifiable, the predictive resampling yields a limiting posterior for $P_X$6 that is invariant to the choice of $P_X$7, provided the predictive family $P_X$8 is identifiable via the inner product structure. This key property—design invariance—is desirable in high-dimensional or complex $P_X$9 settings where modeling $P_X$0 is infeasible.

Novel Parametric Martingale Posterior

The paper introduces a new recursive update for martingale posteriors that achieves weak design invariance and robustifies inference in the high-dimensional setting. The update takes the form:

$P_X$1

where $P_X$2 is an accumulated Fisher information-type matrix with regularization, $P_X$3 is the score, and $P_X$4 is a correction factor crafted to ensure (weak) design invariance—the invariance of the posterior mean and covariance of $P_X$5 to $P_X$6.

This update generalizes ordinary recursive least squares to non-Gaussian models and incorporates shrinkage priors—ridge, LASSO, or continuous spike-and-slab—by appropriately structuring the regularization matrix $P_X$7.

Figure 2: Posterior comparison for an active and inactive coefficient. The effect of varying $P_X$8 (standard deviation $P_X$9) is negligible under the proposed method.

Theoretical Guarantees

Weak Design Invariance: Theorems guarantee that under the proposed update, the limiting posterior mean and covariance of $P_X$0 do not depend on $P_X$1 so long as identifiability holds and $P_X$2 (local Fisher information) is bounded and positive. The proof leverages martingale properties, covariate invariance via the Sherman–Morrison formula, and conditional Lyapunov analysis for general score structures.

Asymptotic Normality: Under regularity and appropriate moment conditions on $P_X$3 (with technicalities to handle heavy-tails and non-homogeneous $P_X$4), the martingale posterior is shown to be asymptotically normal. Thus, posterior uncertainty is fully quantified by its mean and covariance asymptotically, supporting fast, scalable inference without MCMC.

High-Dimensional Implementation and Shrinkage Priors

In high dimensions ($P_X$5), identifiability of $P_X$6 is ensured by using continuous distributions (e.g., $P_X$7 with $P_X$8 large) and regularization is intrinsic via the diagonal matrix $P_X$9, which can encode Bayesian LASSO or spike-and-slab shrinkage. The method initializes at the maximum-a-posteriori estimate given the actual data, and subsequent inference is robust (via weak design invariance) to the choice of $P_X$0.

Empirical Evaluation

A simulation study in robust high-dimensional regression demonstrates that the martingale posterior matches the traditional Bayesian posterior both for active (nonzero) and inactive coefficients, and remains invariant under large changes to the scaling of $P_X$1. The approach drastically reduces computation time relative to MCMC (e.g., 0.12s for predictive resampling vs. 59.63s for Gibbs sampling for comparable effective sample size), underscoring the computational advantage in practical work.

Practical implementation benefits from the observation that increasing the scale parameter $P_X$2 in $P_X$3 improves the convergence rate of predictive resampling, allowing early truncation with negligible impact on uncertainty quantification.

Figure 3: Posterior comparison for the Student-t example under the proposed update rule, for active and inactive coefficients.

Figure 4: Additional posterior results for the Student-t example, further illustrating design invariance.

Figure 5: Trajectories of predictive resampling for different scale parameters $P_X$4, demonstrating rapid convergence for larger $P_X$5.

Additional Extensions and Simulations

Further experiments (not shown here) confirm the invariance property for non-Gaussian regression (Gamma, logistic) and under different regularization schemes. The framework also generalizes to fixed-design settings, where the future covariate sequence is deterministic but sufficient richness and moment conditions must be maintained.

Implications and Future Directions

The explicit recognition and formal analysis of $P_X$6 in the predictive Bayesian framework fills a crucial gap in the understanding of predictive resampling, especially for high-dimensional regression where traditional empirical design-based approaches break down. The parametric martingale posterior with weak design invariance allows regression inference with strong theoretical properties, computational tractability, and practical robustness.

This work opens several future directions:

• Extension to nonlinear regression or nonparametric frameworks where identifiability is more subtle.
• The quantification of model selection uncertainty beyond coefficient-level uncertainty when using spike-and-slab priors.
• Further connections with online and federated learning, where the predictive resampling framework may facilitate scalable Bayesian computation.
• Potential application in Bayesian inference for compositional or structured covariate spaces—e.g., in genomics or longitudinal studies—where design modeling is nontrivial.

Conclusion

This paper establishes the central theoretical role of the design distribution in predictive Bayesian regression, introducing a new family of parametric martingale posteriors suited to high-dimensionality and regularization. Under mild identifiability, the approach guarantees robust and computationally efficient inference, independent of $P_X$7, and matches or outperforms traditional Bayesian methods in both accuracy and speed. The framework and results significantly broaden the practical scope of predictive Bayesian inference in modern regression problems.