Bayesian Model Averaging under Predictor Redundancy via Density-Ratio Posterior Compression

Abstract: Bayesian model averaging in support-indexed regression induces a posterior distribution over active predictor supports. Under predictor redundancy, posterior mass can spread across many nearly interchangeable supports, making exact-support summaries unstable or hard to interpret even when prediction is stable. We study how to report an already fitted Bayesian model averaging posterior without changing the Bayesian target. A report uses hard or soft regions of support space, and its compressed reporting law is compared with the reference posterior through an explicit density ratio. This ratio gives computable total-variation and Kullback--Leibler distortion, bounds for bounded predictive summaries, retained-mass diagnostics, and fallback-weight diagnostics. The framework covers fixed hard regions, metric-ball regions, posterior-cluster regions, and pooled-pruned region dictionaries. We prove exact error formulas and validation bounds for these region reports, and give conditions under which a few regions can replace a long list of individual supports. In simulations, our region reports often give shorter and clearer summaries while preserving the main posterior information, and the density-ratio diagnostics show when too much information has been lost.

• The paper introduces a density-ratio-based framework to compress support posteriors in Bayesian Model Averaging under predictor redundancy.
• It formulates explicit TV and KL divergence metrics to quantify compression distortion and balance reporting cost with posterior fidelity.
• The method enables compact, group-based summaries that outperform naive support-atom lists in highly redundant predictor scenarios.

Bayesian Model Averaging under Predictor Redundancy: Density-Ratio Posterior Compression

Introduction

The paper "Bayesian Model Averaging under Predictor Redundancy via Density-Ratio Posterior Compression" (2606.21080) systematically formalizes and addresses the instability of support-based posterior summaries in Bayesian Model Averaging (BMA) when predictor redundancy is present. High posterior mass can be distributed among numerous near-equivalent supports, especially in domains with strongly correlated or structurally redundant predictors (e.g., spectroscopy, sensor networks, -omics modules). The paper advances a model-agnostic framework for compressing arbitrary Bayesian support posteriors into concise region-based reports, equipped with explicit diagnostics quantifying compression distortion via density ratios.

Posterior Redundancy and the Limits of Support-Atom Reporting

BMA over a collection of support-indexed Bayesian regression models induces a posterior $\pi_0(\gamma \mid D)$ over binary support vectors. In settings with redundant predictors—such as collinear channels, spatial neighborhoods, or gene modules—the posterior can be highly multimodal at the support level while maintaining stable and coherent lower-resolution, e.g., group-level, predictive inference (Figure 1).

Figure 1: Predictor redundancy regimes.

Standard reporting methodologies such as top-$M$ support lists, credible support sets, or marginal inclusion probabilities obscure this structure by their inherently atomistic focus. These methods fail to communicate the effective uncertainty when signal is shared across highly overlapping supports. Moreover, under exchangeable or nearly-exchangeable redundancy, enumerating support-atoms on the reporting frontier is infeasible, and the reporting cost becomes prohibitive.

Density-Ratio Posterior Compression: Core Methodology

The main contribution of the paper is recasting the reporting problem as posterior compression via a family of weighted support kernels: indicator or soft-weighted functions over support space. Reporting then proceeds by constructing a mixture

$$\bar{\pi}_q(\gamma \mid D) = \sum_{m=0}^{M} q_m \pi_m(\gamma)$$

where each $\pi_m(\gamma)$ is the kernel-restricted posterior and $q_m$ is the reporting mixture weight. The critical innovation is the explicit use of the density ratio $$h_q(\gamma) = \frac{\bar{\pi}_q(\gamma \mid D)}{\pi_0(\gamma \mid D)}$$, which allows direct and computable quantification of the information loss relative to the reference posterior through total-variation (TV), forward/reverse Kullback–Leibler (KL) divergence, and mass-retention diagnostics. The fallback weight $q_0$ quantifies residual reliance on the unrestricted BMA posterior.

Figure 2: Support-kernel compression in an enumerable redundant example.

This formalism supports hard (set-based) and soft (e.g., exponentially decaying) regions, accommodating arbitrary structures: intervals, group patterns, metric balls, or posterior clusters.

Error Bounds, Rate–Distortion Tradeoff, and Validation

For any support-kernel report, the TV and KL divergences between the compressed and reference posteriors are given explicitly as expectations under $\pi_0$ of functions of the density ratio. The following key results underpin the framework:

• For a hard region $F$, $KL\left(\pi_F \| \pi_0\right) = -\log \alpha(F)$, and $M$0, where $M$1 is retained posterior mass.
• For any $M$2, $M$3 and $M$4.

These metrics directly upper-bound posterior-expected errors in bounded functionals and contract through the posterior-predictive kernel. The reporting protocol optimally trades off between reporting complexity (quantified via the cost of a kernel list and its weights) and posterior distortion, formalized as a rate–distortion curve.

Critically, the framework supports held-out validation via sample splitting, enabling principled use of posterior-adaptive region classes (such as posterior clusters) while safeguarding against overfitting and optimistic mis-calibration, even in MC approximation scenarios and under mass-floor stabilization for rare regions.

Figure 3: Large $M$5 same-target frontier.

Exploiting Redundancy: Region Summaries vs. Support-Atom Lists

A central theoretical result quantifies the compressibility gap between group/region-based and support-atom-based summaries. When posterior mass concentrates on redundancy classes $M$6 (e.g., supports selecting one predictor per group), the minimal TV error achievable by an $M$7-atom list scales as $M$8, and covering TV $M$9 requires covering $$\bar{\pi}_q(\gamma \mid D) = \sum_{m=0}^{M} q_m \pi_m(\gamma)$$0 supports. In contrast, a single group-pattern kernel can yield vanishing distortion at a cost scaling with the (typically much smaller) number of groups.

The consequences are direct. In redundant regimes, support-kernel reporting can yield exponentially shorter and more interpretable summaries at equivalent posterior fidelity—delivering explicit, quantified reductions unattainable via naive support-list truncation.

Figure 4: Semi-synthetic real-$$\bar{\pi}_q(\gamma \mid D) = \sum_{m=0}^{M} q_m \pi_m(\gamma)$$1 Tecator diagnostics; right: $$\bar{\pi}_q(\gamma \mid D) = \sum_{m=0}^{M} q_m \pi_m(\gamma)$$2-weighted channel coverage and simulated active band.

Algorithms and Dictionary Construction

The reporting algorithm proceeds via:

1. Construction of region dictionaries: fixed (prior knowledge, e.g., known groups), metric-based balls (e.g., Hamming or group-Hamming), or posterior-adaptive (clusters, pooled/pruned candidates).
2. Retained-mass evaluation on validation splits.
3. Mixture weight optimization under penalized distortion (including fallback constraints and optional smoothing).
4. Pruned display of the final report, listing kernels, weights, costs, and retention metrics.

The framework is robust to the quality of the reference posterior and holds for any support-indexed Bayesian regression equipped with support-posterior draws (exact enumeration or MCMC/SMC sample approximations).

Empirical Evidence

Benchmarks on both exact and large-scale support spaces ($$\bar{\pi}_q(\gamma \mid D) = \sum_{m=0}^{M} q_m \pi_m(\gamma)$$3) exhibit that region-based kernel reporting (e.g., group-Hamming balls, pooled-pruned adaptive dictionaries) dominate top-$$\bar{\pi}_q(\gamma \mid D) = \sum_{m=0}^{M} q_m \pi_m(\gamma)$$4 or credible-set atom lists in both distortion metrics and reporting cost, especially in structured redundancy settings. Structured kernel summaries attain TV and FKL one to two orders of magnitude below support-atom lists, with shorter, interpretable reports.

Pooled-pruned summaries induce major cost savings while maintaining low reliance on fallback. Posterior cluster-based methods can reduce raw distortion but may over-rely on fallback if not properly validated. Semi-synthetic and reduced real-response checks (e.g., Tecator spectroscopy) demonstrate that the approach is effective in realistic, highly collinear scenarios, and the $$\bar{\pi}_q(\gamma \mid D) = \sum_{m=0}^{M} q_m \pi_m(\gamma)$$5-weighted region coverage closely matches the true active bands.

Figure 5: Full exact reporting-distortion frontiers.

Implications and Outlook

This work rigorously advances posterior summarization under redundancy from atom-level enumeration to rate–distortion quantification at the region level. The formal linkage between compression reports and explicit diagnostics via density ratios provides auditability and enables rigorous calibration of posterior uncertainty at scientifically meaningful resolutions.

Practically, reducing report length and ambiguity is critical for scientific interpretability and communication in high-dimensional applications with complex feature structures (genomics, spectroscopy, environmental sensing). The framework generalizes to any support-indexed Bayesian analysis with a computable or sample-able posterior, suggesting applicability to generalized regression, graphical models, and other hierarchical selection tasks.

Theoretically, the results imply an exponential separation between atomistic and region-based reporting costs in redundant regimes, and the rate–distortion perspective sharpens understanding of the ultimate limits of posterior reporting under information constraints.

Future possible directions include real-data applications with intricate groupings, further exploration of model-agnostic adaptive kernel discovery, and deeper integration with reference-posterior validation methodologies, including hybrid MC and variational inference diagnostics.

Conclusion

This paper establishes a general, fully auditable protocol for posterior compression in BMA under predictor redundancy, leveraging explicit density-ratio diagnostics to anchor region-based uncertainty reporting. The framework provides both theoretical justification and practical methodology for transforming unstable, high-cost support atom lists into compact, reliable summaries, preserving the lower-resolution scientific signal and quantifying residual uncertainty, all without reweighting or re-computation of the core Bayesian posterior (2606.21080).