Schwarz Information Criterion (SIC) Overview

• SIC is a model selection tool that quantifies the trade-off between goodness-of-fit and complexity using a logarithmic penalty based on the sample size.
• It incorporates additional penalties for model misspecification and singularities, ensuring robust performance in challenging, high-dimensional settings.
• Generalizations like sBIC and OC-BIC extend the SIC framework to handle singular models and order-constrained selections, offering improved empirical and theoretical outcomes.

The Schwarz Information Criterion (SIC), more commonly referred to as the Bayesian Information Criterion (BIC), is a fundamental asymptotic criterion for model selection that quantifies the trade-off between model fit and model complexity on the basis of large-sample approximations to the Bayesian marginal likelihood. The canonical SIC formula balances maximal likelihood attainment with a logarithmic complexity penalty, and has been extended to accommodate situations involving model misspecification, singular parameterizations, and order-constrained hypotheses. These developments have resulted in several generalized forms—each maintaining strong theoretical guarantees under relaxed regularity conditions, and each supported by rigorous asymptotic expansion techniques, including Laplace's method and modern algebraic geometry.

1. Derivation and Structure of the SIC

In regular parametric models, suppose one observes data $y_1,\dots,y_n$ and posits a candidate model $M$ parameterized by $\theta \in \mathbb{R}^d$ and prior density $\pi(\theta)$. The logarithm of the marginal likelihood is asymptotically expanded as

$$\log\bigl[\alpha_M\int e^{\ell_n(\theta)}\,\pi(\theta)\,d\theta\bigr] = \ell_n(\widehat\theta_n) - \tfrac{1}{2} d \log n + \tfrac{1}{2} \log|\tfrac{1}{n} H_n(\widehat\theta_n)| + \log\alpha_M + \tfrac{d}{2}\log(2\pi) + O(1),$$

where $\ell_n(\theta)$ is the quasi-log-likelihood, $\alpha_M$ is a model prior, and $H_n(\theta)$ is the observed information matrix, given by the Hessian of $-\ell_n$. Under typical conditions—full rank design, exponential family structure, smooth prior—the Laplace approximation delivers this result with probability tending to one as $n \to \infty$ (Lv et al., 2010).

The SIC is then formally given by

$$\mathrm{SIC}(M) = -2\,\ell_n(\widehat\theta_n) + d\,\log n - \log|\widehat H_n(\widehat\theta_n)|,$$

where $\widehat H_n$ consistently estimates the covariance-contrast matrix $A_n = H_n^{-1} V_n$, with $V_n(\theta)$ the outer-product of score vectors under the true data-generating law. The components decompose into a goodness-of-fit term, a complexity penalty, and a misspecification penalty.

2. SIC under Model Misspecification and Generalizations

The classical BIC penalty assumes correct specification: $H_n(\widehat\theta_n) = V_n(\widehat\theta_n)$, so $A_n = I_d$ and $-\log|A_n| = 0$. Under model misspecification or when the data-generating distribution is not contained in the parametric family, $A_n \neq I_d$ and the additional penalty $-\log|A_n|$ directly quantifies the Kullback-Leibler discrepancy between the model's Fisher-information and the actual variability. This ensures that SIC penalizes models that fit the data less well even if they are flexible (Lv et al., 2010).

The estimation of $A_n$ involves computing

• $H_n(\widehat\theta_n)$: Observed information (Hessian of negative log-likelihood).
• $V_n(\widehat\theta_n)$: Sample covariance of score vectors. The determinant ratio provides an interpretable deviation from the "ideal" information structure, with practical calculations requiring consistent estimators in high-dimensional regimes.

3. SIC and BIC for Singular Models

For singular models—where the Fisher information is not everywhere invertible—classical BIC fails to capture the correct penalty structure. Recent work by Watanabe establishes that the logarithm of the marginal likelihood in such models takes the form

$$\log L(M) = \log p(y_n|\hat\theta, M) - \lambda(\pi_0) \log n + (m(\pi_0)-1) \log\log n + O_p(1),$$

where $\lambda(\pi_0)$ (the real log canonical threshold, RLCT) and $m(\pi_0)$ (multiplicity) are birational invariants depending on the true distribution $\pi_0$ (Drton et al., 2013, Watanabe, 2012). In regular cases, $\lambda = d/2, m=1$ recovers the traditional BIC. Failure to account for the correct penalty leads to mis-selection and inconsistent model choice, especially for mixture, factor, or hidden Markov models.

The singular BIC (sBIC) addresses circular reasoning by averaging over submodels—using local learning coefficients and fixed-point equations—to obtain marginal likelihood approximations that retain consistency and Bayesian justification. Notably, sBIC rapidly focuses on the true model in simulations where BIC is inconsistent or overly concentrated, especially in complex latent-variable scenarios (Drton et al., 2013).

4. SIC Extensions for Order-Constrained Model Selection

BIC's penalty structure treats order-constrained models (e.g., $\theta_1>\theta_2>\theta_3>0$) as equivalent in complexity to unconstrained alternatives, neglecting their reduced parameter space. The Laplace approximation is invalid when the MLE lies on the boundary of the constraint region (Mulder et al., 2018). Two prominent extensions have been developed:

• Truncated Unit-Information Prior: Centered at the unconstrained MLE. The penalty fails to increase with the strength of evidence for the constraints, violating Occam's razor.
• Truncated Local Unit-Information Prior: Centered at the boundary (null value). The order-constrained BIC based on this prior displays proper Occam behavior and lower error probabilities.

The order-constrained BIC formula for local priors is

$$\mathrm{OC\text{-}BIC}_{LUI}(M_1) = -2\,\ell(\hat\theta_u) + d\,\log n - 2\,\log\Pr(R_1\theta>r_1|D,M_u) + 2\,\log\Pr^{LUI}(R_1\theta>r_1|M_u),$$

where the prior constraint probability penalizes the model according to how restrictive the constraints are, and the posterior constraint probability rewards strong empirical evidence for the inequalities (Mulder et al., 2018).

5. Theoretical Foundations and Asymptotics

The SIC is grounded in large-sample theory utilizing Taylor expansions, change-of-variable arguments, and Gaussian approximations to integrate the likelihood with respect to the prior. Essential assumptions include:

• Exponential family structure with known link function and sufficient smoothness.
• Full rank or diverging eigenvalues in the design matrix ensuring identifiability and concentration.
• Control of third derivatives and bounded prior densities in shrinking neighborhoods around the QMLE.

For singular models, algebraic geometric techniques quantify the breakdown of Laplace's method via the RLCT. The widely applicable Bayesian information criterion (WBIC) estimates the marginal likelihood with tempered posteriors ($\beta = 1/\log n$), recovering the correct penalty asymptotically regardless of regularity assumptions (Watanabe, 2012).

6. Practical Implications and Empirical Results

Empirical investigations demonstrate that the SIC maintains frequentist consistency and robust Bayesian performance under both correct specification and misspecified scenarios. In model selection for high-dimensional GLMs, SIC yields a decomposable criterion that explicitly penalizes misspecification, resulting in improved selection performance (Lv et al., 2010).

For complex models with singularities—such as mixtures, factor analysis, and latent class analysis—sBIC and WBIC offer practical solutions. Simulation studies confirm superior concentration on the true model and improved ranking characteristics relative to classical BIC, especially when the true data-generating process induces information singularities or lies on lower-dimensional subspaces (Drton et al., 2013, Watanabe, 2012).

Extensions to order-constrained selection using local unit-information priors are validated in both synthetic and real-world scenarios. Posterior probabilities computed with OC-BIC track full Bayesian calculations closely, and compared with unconstrained alternatives, they yield greater statistical power and discrimination among hypotheses. These methodologies are implemented in packages such as BICpack for direct application (Mulder et al., 2018).

7. Connections, Limitations, and Outlook

The SIC unifies Bayesian and frequentist model selection principles via asymptotic approximations, and its tractability has facilitated widespread application in high-dimensional statistics. Its generalizations to misspecified, singular, and order-constrained cases preserve consistency and interpretability but require additional information—such as learning coefficients, prior specification, or covariance contrast estimation.

A plausible implication is that continued refinement of these penalties, through algebraic geometrical invariants or order-specific priors, will further improve model selection performance in nonregular and nonparametric regimes. Limitations include the need for accurate estimation of misspecification penalties and RLCTs, which may not be available for all models. Theoretical developments suggest that SIC-type criteria will remain central to principled model choice and regularization in contemporary statistical methodology.