Bayesian Predictive Synthesis

• Bayesian Predictive Synthesis is a framework that integrates predictive distributions from multiple sources using empirical Bayes factors to assign weights.
• It employs disjoint data partitioning, pairwise Bayes factor computation, and geometric mean weighting to avoid double data use and maintain model coherence.
• Applications, such as ozone prediction, show that this synthesis approach improves prediction error and uncertainty calibration compared to standard automated methods.

Bayesian Predictive Synthesis is a rigorous statistical framework for integrating predictive distributions—originating from diverse sources such as models, human analysts, or expert forecasters—into a coherent, unified Bayesian prediction. Rather than relying strictly on the assumption that one model is “correct,” this paradigm is centered on the synthesis of subjective or objective predictive information, leveraging formal probabilistic weighting that reflects performance, subjective beliefs, or both. This approach addresses both the theoretical concerns of Bayesian data analysis and practical challenges in areas such as environmental statistics, macroeconomic forecasting, and industrial process optimization.

1. Core Methodology

Bayesian Predictive Synthesis (BPS) proceeds by partitioning the available data among multiple independent analysts or models, each of whom constructs a Bayesian model using their assigned subset. Each analyst produces a Bayesian summary—encapsulating their posterior over parameters, an updating mechanism for including new data, and the marginal likelihood. The procedure then unfolds as follows:

• Model Update: Each analyst/model updates its constructed posterior with data not used in the initial model-building phase. This ensures no data are “double used” for fitting and assessment, thereby maintaining the Bayesian principle of a single update per data item.
• Bayes Factor Computation: For each pair of analysts’ updated models, the marginal likelihoods on the “out-of-sample” data are evaluated, and pairwise Bayes factors are computed. If posteriors are available as Monte Carlo samples, the marginal likelihood calculations become straightforward via Monte Carlo averages of the likelihoods under posterior draws.
• Weight Assignment: Letting $k$ be the number of analysts, the weight $b_i$ assigned to analyst $i$ is the geometric mean over all pairwise Bayes factors comparing $i$ to the remaining analysts:

$$b_i = \left\{ \prod_{l=1}^k \tilde{B}_{il} \right\}^{1/k}$$

where $\tilde{B}_{ij}$ is the (estimated) Bayes factor comparing analyst $i$ to $j$ based on the marginal likelihoods evaluated on “new” data.

• Synthesis: The overall posterior is formed as a weighted sum (with normalization) of the individual analysts’ posteriors:

$$f(\theta \mid Y) = \frac{ \sum_{i=1}^k b_i f_i(\theta_i \mid Y_1, \ldots, Y_k) } { \sum_{j=1}^k b_j }$$

This synthesis respects the ordering of Bayes factors in model support, and the method’s “uniqueness” property guarantees that the ratio of weights $b_i / b_j$ equals the Bayes factor $B_{ij}$ provided all Bayes factors are compatible.

This approach is directly applicable to scenarios with unambiguous subjective divides (e.g., multiple skilled human analysts) and is extendable to any context where model or analyst diversity is central to predictive heterogeneity.

2. Comparative Predictive Performance

The principal empirical demonstration centers on the ozone dataset from the Los Angeles basin. Here, three human analysts each built distinct Bayesian models upon different data thirds, with approaches ranging from variable selection with nonlinearity, CAR models with trend removal, to modified LARS algorithms. Predictive accuracy was compared across a suite of classical and modern automated procedures: AIC, BIC, Smoothing Splines, CART, Bagged CART, Bayes CART, Bayesian Model Averaging (BMA), LARS, and BART.

Several performance metrics were evaluated:

• Sum of squared errors (log-scale ozone concentration).
• Mean squared prediction error (original ozone scale).
• Classification error (regulatory ozone exceedance).
• Calibration metrics: 90% predictive interval coverage and optimism ratio (mean squared error/predictive variance).

The key results:

• Human-generated models, even individually, outperformed most automatic methods (excluding BART).
• Bayesian Synthesis further improved predictions, particularly in reducing average prediction errors and maintaining realistic calibration of predictive intervals.
• Convex Synthesis (simple equal-weighted averaging of analysts’ predictions) also yielded gains, sometimes modestly exceeding Bayesian Synthesis performance—especially in sequentially updated scenarios. This suggests convex averaging may recover combinations that expand the envelope of plausible predictions, especially when no single model completely captures the true data generative mechanism.

3. Convex versus Bayesian Synthesis

Convex Synthesis is a reference (null) method where the amalgamation of analyst/model predictions occurs via a fixed, typically equal, weighting:

$$f_{\text{convex}}(\theta \mid Y) = \frac{1}{k} \sum_{i=1}^k f_i(\theta_i \mid Y)$$

Unlike Bayesian Synthesis, where the combination weights reflect the empirical support of each model on unseen data, Convex Synthesis remains agnostic, treating all analysts equally regardless of their marginal likelihoods. Empirical results indicate that convex approaches can be robust, yielding marginally better predictions than synthesis by Bayes factor weighting, particularly when models are equidistant from the “truth.” In situations where predictive diversity is beneficial and none of the available models is fully adequate, convex averages can outperform more sophisticated weightings.

4. Theoretical and Practical Considerations

Bayesian Synthesis addresses several foundational and operational issues in statistical modeling:

• Avoidance of Double Data Use: By using disjoint data for model fitting and performance weighting, the procedure maintains Bayesian updating’s integrity.
• Uniqueness and Coherence: The geometric mean weighting satisfies mathematical constraints (the “Consistency” or “Uniqueness” property) under exchangeability of analysts and compatibility of Bayes factors.
• Implementation Requirements: All analysts’ models must use compatible probability structures—likelihoods must be absolutely continuous with respect to a common reference; marginal likelihoods and updating rules must be unambiguously specified.
• Operational Costs: The methodology demands increased analyst/modeling effort compared to fully automated approaches, as it requires independent modeling and common conventions for posterior evaluation.
• Model Calibration: Synthesized models generally display more realistic uncertainty quantification compared to those produced by automatic methods.

5. Application Insights: Ozone Prediction Case Study

The division of data and subsequent synthesis in the ozone prediction problem led to superior empirical performance. Each analyst’s approach exploited different modeling philosophies, and the eventual Bayesian Synthesis:

• Achieved lower predictive error and improved calibration compared to most automated competitors.
• Outperformed predictive intervals from automatic methods in both coverage and calibration/optimism.
• Demonstrated the benefit of leveraging modeling creativity and domain expertise, particularly when models are challenged by complex data-generating mechanisms (such as ozone formation, where meteorological and pollutant interactions are highly nonlinear and subject to discontinuities).

Notably, BART performed best among automated methods, indicating that flexible, nonparametric approaches can rival or even surpass human/subjective synthesis in sufficiently expressive applications.

6. Broader Implications and Extensions

Bayesian Predictive Synthesis represents a paradigm for merging subjective analysis with formal Bayesian updating, with the following implications:

• Bridges the Bayesian theory–practice gap: Offers a principled device for shifting from idealized Bayesian inference with a single “true” model to a pragmatic setting where subjectivity, analyst diversity, and model uncertainty dominate.
• Generalizes to high-stakes problems: Recommends the deliberate synthesis of independent expert analyses in fields such as clinical trial data assessment, industrial process control, and complex survey analysis, offering robustness to model misspecification and realistic uncertainty assessment.
• Lays groundwork for decision-analytic extension: The basic synthesis can be further generalized to cases where the predictive combination is tuned to downstream decision or utility functions—connecting with contemporary developments in predictive decision synthesis.

7. Summary Table of Syntheses and Methods

Approach	Weighting Strategy	Empirical Property
Bayesian Synthesis	Geometric mean of Bayes factors	Adapts to relative model support
Convex Synthesis	Equal (fixed) weights	Expands predictive convex hull
Automatic Procedures	(various: AIC, BIC, BMA, BART, …)	Lacks subjective domain knowledge
Single Human Model	N/A	May outperform most automatic methods

Bayesian Predictive Synthesis thus offers a technically sound and practically robust method for integrating disparate predictive models or subjective analyses. By formalizing both the weighting and combination, it advances the practice of Bayesian model averaging to a framework capable of exploiting both human expertise and data-driven model comparison, enhancing predictive accuracy and uncertainty calibration while respecting fundamental Bayesian principles.