Jointly Conjugate Prior Distributions

• Jointly conjugate prior distributions are a class of priors that maintain posterior form consistency by coupling parameter and model priors during Bayesian updating.
• They stabilize model selection by neutralizing the effect of diffuse parameter priors, thereby mitigating issues such as Lindley’s paradox in complex models.
• Their joint structure facilitates computational efficiency and analytical tractability, yielding closed-form or approximated marginal likelihoods for robust inference.

A jointly conjugate prior distribution is a class of prior distributions constructed to ensure that, under Bayesian updating, the posterior distribution remains in the same class as the prior for a collection of models or hierarchical structures. This concept generalizes the familiar notion of conjugacy, providing a unified framework for robust Bayesian inference, particularly in contexts involving model selection, variable-order processes, or high-dimensional parameter spaces. In regression-type models and in variable-order Markov models, jointly conjugate priors are leveraged to resolve key pathologies—such as Lindley’s paradox—by coupling the specification of parameter priors with the model space prior.

1. Definition and Motivation

The jointly conjugate prior can be formally characterized as a family of priors $\{\pi_{\lambda}(\theta_m)\}$ indexed by hyperparameters $\lambda$ that, when combined with a likelihood family for models across a model space $\mathcal{M}$, ensures closed-form or analytically tractable posteriors jointly across $(m, \theta_m)$. The main motivation for joint specification arises in Bayesian model comparison, where the marginal likelihood (integrated over parameter priors) interacts with the prior over models, potentially leading to undesirable sensitivity of posterior model probabilities to arbitrary choices of prior dispersions for the $\theta_m$. Such sensitivity is most acute in regression-like models with multivariate normal parameter priors.

Key Principle: By calibrating the model space prior $f(m)$ as a function of the dispersion of the parameter prior $f(\beta_m|m)$—for instance, setting $f(m) \propto c_m^{d_m}$ where $c_m^2$ controls the prior covariance and $d_m = \text{dim}(\beta_m)$—the accidental penalization of higher-dimensional models due to diffuse priors is neutralized, and the posterior model probability becomes stable under prior scale choices (Dellaportas et al., 2012).

2. Formal Construction and Mathematical Properties

Given a collection of models indexed by $m \in \mathcal{M}$, suppose the likelihood under $m$ is $f(y|m, \beta_m)$ and the prior on $\beta_m$ is multivariate normal: $$f(\beta_m|m) = (2\pi)^{-d_m/2} |V_m|^{-1/2}\exp\left(-\tfrac{1}{2}(\beta_m - \mu_m)^\top V_m^{-1}(\beta_m - \mu_m)\right)$$ with $V_m = c_m^2 \Sigma_m$. The standard Bayesian model comparison formula is: $$f(m|y) \propto f(m)\int f(y|m, \beta_m)f(\beta_m|m) d\beta_m$$ For large $c_m$, the marginal likelihood behaves as $$f(y|m) \propto c_m^{-d_m}\times(\text{likelihood portion})$$. The joint specification sets: $f(m) \propto p(m) c_m^{d_m}$ or, more generally,

$f(m)\propto p(m) |V_m|^{1/2} |i(\beta_m)|^{1/2}$

where $i(\beta_m)$ is the unit information matrix (e.g., Fisher information per observation).

With this specification, the detrimental dependence on $c_m$ cancels: $$f(m|y)\propto f(m) f(y|m) \propto p(m) \cdot (\text{likelihood part})$$ making model selection robust to arbitrary scaling of the parameter prior.

3. Resolution of Lindley’s Paradox in Model Selection

A central issue addressed by the jointly conjugate prior construction is the sensitivity of posterior model probabilities to parameter prior dispersion—a phenomenon known as Lindley’s paradox. When a uniform prior $f(m)$ is combined with highly diffuse parameter priors, the marginal likelihood for high-dimensional models is severely penalized (due to the $c_m^{-d_m}$ term), even when the data favor complex models.

Joint Specification Solution: By coupling the model prior and parameter prior dispersions,

$f(m) \propto c_m^{d_m},$

the $c_m$-scale effect is canceled in the posterior odds, effectively preventing the "automatic" selection of the simplest model as $c_m \rightarrow \infty$.

In practical examples—such as linear regression, log-linear regression for contingency tables, and variable-order Markov modeling—posterior model probabilities and predictive performance are stabilized against arbitrary prior scale choices. For instance, in a normal linear model, with this adjustment the log-posterior odds approximates the Bayesian Information Criterion (BIC) penalty: $$\log f(m|y) \approx C + \log f(y|m, \hat{\beta}_m) + \log p(m) - \frac{d_m}{2}\log n + o(1)$$ demonstrating that model selection aligns with BIC-like complexity corrections (Dellaportas et al., 2012).

4. Analysis in Variable-Order, Reversible Markov Chains

For variable-order, reversible Markov chains, the definition and usage of conjugate priors are underpinned by reinforced random walks analogous to the Bayesian use of Beta priors in exchangeable urn models (Bacallado, 2011). The key technical ingredient is the application of Levy’s extension of the Borel–Cantelli lemma. This extension establishes that, for a reinforced random walk with conditional transition probabilities $$p_m = w'_{\tau_m}(\overline{v u}) / w''_{\tau_m}(v)$$ at transition time $\tau_m$, the divergence $\sum_m p_m = \infty$ almost surely implies that the empirical frequency of transitions converges almost surely to the predicted frequency.

This result ensures:

• Every transition with positive prior weight is observed infinitely often (recurrence).
• The mixing properties of the process hold.
• The conjugacy structure necessary for consistent Bayesian updating across model orders is preserved.

Consequently, the conjugate prior for the reversible Markov chain, arising from the partially exchangeable reinforced random walk, supports inference for order testing and parameter estimation robustly throughout the model space.

5. Implications for Inferential and Predictive Quantities

By adopting jointly specified conjugate priors:

• Model posterior probabilities are robust to the choice of parameter prior scaling.
• Predictive distributions derived via model averaging are more stable and data-reflective, immune to over-penalization of complex models due to diffuse priors.
• In practice, the inferential apparatus for regression, high-dimensional variable selection, and Markov model order estimation becomes less arbitrary and more reliable, as predictions and posterior inferences are governed by joint coherence rather than ad hoc prior choices.

The computational advantage is significant: the joint conjugate structure leads to closed-form updates or Laplace-approximated marginal likelihoods, efficient algorithms for parameter and model selection, and tractable expressions for cross-model predictive scores.

6. Examples and Model Classes

Applications elucidated in (Dellaportas et al., 2012) and (Bacallado, 2011) span:

• Simple Linear Regression: Posterior model probabilities and in-sample predictions for different prior dispersions, demonstrating the stability conferred by the joint specification.
• Log–Linear Models: Bayesian inference for contingency tables; Fisher information adaptation in the model prior allows for respect of prior domain knowledge (e.g., negligible higher-order interactions).
• Variable-Order Markov Chains: Reinforced random walks ensure the conjugate prior yields recurrent, ergodic transition behavior across candidate orders.

The table below summarizes the key specifications:

Model Setting	Parameter Prior	Model Prior	Joint Conjugacy/Robustness
Linear regression (Gaussian)	$\mathcal{N}(\mu_m, c_m^2\Sigma_m)$	$f(m) \propto c_m^{d_m}$	Posterior model probabilities robust to $c_m$; BIC-like complexity penalty
Log-linear regression	$\mathcal{N}(\mu_m, c_m^2\Sigma_m)$	$f(m) \propto |V_m|^{1/2}|i(\beta_m)|^{1/2}$	Adjusts for Fisher info; effective for high-dimensional, hierarchical structures
Var.-order reversible MC	Conjugate reinforced random walk distribution	Implied by the urn process	Recurrence and mixing properties ensure consistent parameter/model inference

7. Significance and Limitations

The jointly conjugate prior construction provides a systematic procedure for robust Bayesian model selection, inference, and prediction in structured statistical settings. By explicitly coupling parameter and model space priors, it directly addresses the instability (Lindley’s paradox) inherent in high-dimensional or variationally-well-separated model regimes, and ensures inferential results are governed by substantive information rather than arbitrary scale assignments.

A plausible implication is that similar joint prior design principles may be extended to other hierarchical Bayesian models, possibly by calibrating the prior on the model index (or on a latent structure space) as a function of marginal likelihood sensitivity to parameter prior dispersion.

A limitation is that such joint specification presumes an explicit mathematical relationship between marginal likelihood and parameter prior scale, which is most transparent in regular exponential family models; further research may be needed to extend the paradigm to highly nonregular or non-exponential family model classes where closed-form marginal likelihood expansion is unavailable.

---

In sum, the theory and practice of jointly conjugate prior distributions (Dellaportas et al., 2012, Bacallado, 2011) provide a rigorous foundation for both model selection and parameter inference in hierarchical and structured Bayesian analyses, ensuring stable, scientifically interpretable inference even as prior parameter dispersions vary. This approach is particularly effective in regression-type contexts and in modeling complex stochastic processes, where the avoidance of sensitivity to arbitrary prior hyperparameter scaling is crucial for credible and replicable Bayesian model selection.