Stochastic Search BIC in Model Selection

• Stochastic Search BIC is a model selection criterion that refines standard BIC by incorporating algebraic-geometric penalties like the RLCT to address singularities in models.
• It leverages asymptotic analysis, resolution of singularities, and Newton polyhedra to compute penalties that more accurately reflect complexity beyond mere parameter counts.
• This method improves consistency in stochastic search algorithms, proving useful in applications such as phylogenetics, causal inference, and mixture modeling by reducing overpenalization.

Stochastic Search BIC (Bayesian Information Criterion) refers to a class of model selection methodologies that integrate asymptotic approximations of marginal likelihood, particularly via BIC-type formulas, into stochastic or randomized search strategies over model spaces. This paradigm is especially relevant when standard BIC penalization fails—such as in singular statistical models, graphical models with hidden variables, finite mixtures, and context where exhaustive search is computationally infeasible—as it enables accurate and efficient model comparison using refined, often geometrically- or algebraically-informed, penalty structures.

1. Asymptotic Foundations and Generalization Beyond Standard BIC

Classical BIC relies on Laplace approximation of the marginal likelihood under regularity conditions: the log-likelihood achieves a unique, interior maximum and the Fisher information is invertible. The canonical form is

$$\log Z(N) \approx \ell^\wedge_N - \tfrac{d}{2} \log N + O(1)$$

where $Z(N)$ is the marginal likelihood, $\ell^\wedge_N$ is the maximized log-likelihood, and $d$ is the model dimension.

In statistical models exhibiting nonidentifiability or singularities—such as Bayesian networks with hidden variables, stratified exponential families, mixture models, and tree graphical models—these assumptions fail. The correct asymptotics for the marginal likelihood become

$$\log Z(N) = \ell^\wedge_N - \lambda\log N + (m-1)\log \log N + O(1)$$

where $\lambda$ is the real log-canonical threshold (RLCT) and $m$ its multiplicity, both determined by the geometry of the model at the maximum likelihood locus. Standard BIC over-penalizes model complexity in these regimes, necessitating revised criteria—termed here “stochastic search BIC”—that accommodate the singular behavior and which may include additional $\log\log N$ components (Zwiernik, 2010, Yamazaki et al., 2012, Rusakov et al., 2012, Drton et al., 2013).

The RLCT and related algebraic-geometric invariants can, for structured models (e.g., tree models, mixture models), be computed in closed form or tightly bounded using combinatorial descriptors of the model (such as degrees of nodes in trees or dimensions of model fibers).

2. Mathematical and Algebraic Tools Underpinning Stochastic Search BIC

The core methodology requires techniques from asymptotic analysis of Laplace-type integrals with singularities:

• Resolution of Singularities: Using methods from algebraic geometry (e.g., Hironaka’s theorem), the parameter space is reparameterized so that in local charts the normalized log-likelihood has a monomial structure, reducing the asymptotic to integrals whose dominant behavior is prescribed by the smallest pole of an associated zeta function (Zwiernik, 2010, Yamazaki et al., 2012).
• Newton Polyhedra and RLCT: The Newton diagram method allows calculation of RLCTs for analytic functions whose principal parts are nondegenerate.
• Real Log-Canonical Threshold: Given a nonnegative analytic function $f(\theta)$, the RLCT is defined via

$$\zeta(z) = \int_\Theta f(\theta)^{-z} \varphi(\theta)d\theta$$

and the smallest pole determines the penalty in the modified BIC.

• Combinatorial Parameter Transformations: For models on trees, raw probabilities are mapped to “tree cumulants” adapted to the topology, and parameters are re-expressed to linearize algebraic constraints, facilitating the RLCT computation.

3. Formulas and Special Cases: Tree Models and Stratified Exponential Families

Closed-form stochastic search BIC formulas emerge for several structural model classes:

Model Structure	Marginal Likelihood Expansion	Penalty Structure
Regular Exponential Family	$\ell^\wedge_N - (d/2)\log N + O(1)$	$(d/2)\log N$
Tree Markov Model (Regular)	$\ell^\wedge_N - ((n_v + n_e -2l_2)/2)\log N + O(1)$	Combinatorial; $n_v, n_e, l_2$
Tree Markov (Trivalent root, degenerate)	$$\ell^\wedge_N - ((3n + l_2 + 5l_3 -1)/4)\log N + O(1)$$	Involves number/type of degenerate nodes
Naive Bayes (Stratified EF)	$Nf_y-\lambda\log N + (m-1)\log\log N + O(1)$	$\lambda$ depends on singularity structure

For stratified exponential families, such as Bayesian networks with hidden variables, the dimensionality-dependent penalty of BIC is replaced by a singularity- and geometry-adapted term: $\lambda\log N$ and, possibly, subdominant terms in $\log\log N$ (Rusakov et al., 2012, Drton et al., 2013). Correct classification of singularity types is required to apply the correct penalty, underscoring the necessity of geometric analysis.

4. Implications for Model Selection and Stochastic Search

Stochastic search BIC supplies a theoretically sound criterion for model exploration and comparison even in nonregular settings. In practical stochastic search algorithms—MCMC, genetic algorithms, evolutionary heuristics—the scoring function used to evaluate candidate models or structures should use the refined BIC penalty derived from the RLCT or geometry-specific calculation, rather than relying on the dimension-based classical BIC.

In model selection, using these formulas confers several advantages:

• Reduced Overpenalization: Standard BIC will tend to underfit (select overly simple models) due to inflated penalties when the true model is singular; stochastic search BIC mitigates this effect.
• Consistency: The selection procedures using RLCT-based penalties (for example, in mixture models, tree graphical models, or networks with latent variables) are provably consistent even when classic moment and differentiability conditions are not met.
• Structural Insight: The form of the penalty reveals subtle distinctions in model complexity not captured by parameter count alone—e.g., for tree models, the number and configuration of latent nodes affect the penalty nontrivially (Zwiernik, 2010).

For high-dimensional spaces or multimodal likelihoods, the practical integration with stochastic search algorithms enables effective traversal of the model space without exhaustive enumeration, as the scoring function faithfully reflects both the fit and the algebraic complexity.

5. Applications: Phylogenetics, Causal Inference, and Beyond

Stochastic search BIC has direct applications wherever model classes are algebraically defined but singular, typical of:

• Phylogenetic tree selection: Hidden (ancestral) nodes induce singularities, and combinatorial properties of the evolutionary tree feed directly into the BIC penalty (Zwiernik, 2010).
• Causal structure discovery: Bayesian networks inferred from observational data may possess hidden confounding variables, leading to nonidentifiable parameterizations (Yamazaki et al., 2012, Drton et al., 2013).
• Mixture modeling: The number of components, as well as redundant or unidentifiable parameterizations, result in singularities requiring geometry-aware scoring (Rusakov et al., 2012, Drton et al., 2013).
• Clustering and factor models: Rank-deficient matrices, latent factors, and blockmodel structures all induce singular behavior requiring corrected penalties.

By deploying the refined BIC formulas, researchers can perform stochastic search across highly structured model spaces, maintaining both statistical validity and computational tractability.

6. Limitations, Challenges, and Future Directions

Stochastic search BIC, while a substantial generalization, is not without caveats:

• Asymptotic Nature: The expansions hold for large $N$; for moderate sample sizes, the approximation may be inadequate, particularly when subdominant terms are difficult to estimate or when the sample is far from the asymptotic regime.
• Complexity of RLCT Calculation: Computing RLCTs (or learning coefficients) may be straightforward in certain structured settings, but can become intractable for arbitrary models without further theoretical or computational advances.
• Finite-Sample Performance: Without regularity, the O(1) terms in the expansion may be nonnegligible, and more uniform approximations are an area of ongoing research (Drton et al., 2013).
• Requirement for Closed-Form or Algorithmic RLCTs: When only bounds or estimates of RLCTs are available, the penalty term may be estimated conservatively, potentially impacting selection accuracy.
• Model Space Definition: The "averaging over submodels" implicit in some stochastic search BIC formulas necessitates clear specification of the universe of candidate models, as the penalty can depend subtly on this set (Drton et al., 2013).

7. Summary

Stochastic search BIC provides an extension of the Bayesian Information Criterion that preserves its asymptotic approximation to the (log) marginal likelihood, but crucially adapts the penalty term to the singularity and algebraic-geometric properties of the statistical model at hand. The approach uses tools from analytic geometry and singularity theory—especially the real log-canonical threshold—to derive model- and data-dependent penalties that accurately reflect effective model complexity. These advances enable more reliable model selection in stochastic search algorithms, improving consistency and interpretability in wide-ranging applications from graphical model selection to mixture models, particularly in the presence of hidden variables, multimodal likelihoods, and parameter nonidentifiability (Zwiernik, 2010, Yamazaki et al., 2012, Rusakov et al., 2012, Drton et al., 2013).