Singular Learning Theory Overview

• Singular Learning Theory is a statistical framework that addresses non-injective parameter mappings by quantifying singularities using invariants such as the real log canonical threshold (RLCT).
• It employs algebraic and toric geometry techniques to resolve singularities, enabling robust estimation of Bayesian generalization error and improved model selection criteria like WAIC.
• The framework extends classical asymptotics to singular and quasi-regular models, offering practical insights for complex models including neural networks and mixture models.

Singular learning theory is a statistical framework designed to analyze learning machines whose parameter-to-distribution mappings are non-injective, resulting in non-isolated sets of optimal parameters, degenerate Fisher information matrices, and analytic or algebraic singularities in the loss landscape. Classical statistical asymptotics, such as maximum likelihood theory and criteria like AIC and BIC, are inapplicable or misleading in such settings. Singular learning theory incorporates techniques from algebraic geometry—most prominently, resolution of singularities—and computes birational invariants such as the real log canonical threshold (RLCT) and singular fluctuation, which strictly govern the asymptotic behavior of Bayesian predictors, generalization error, and minimum description length in practice.

1. Distinction Between Regular, Singular, and Quasi-Regular Statistical Models

In regular statistical models, the mapping from parameters $w$ to probability distributions $p(x|w)$ is one-to-one, and the Fisher information matrix is strictly positive definite. As a consequence, asymptotic theory applies: the maximum likelihood estimator is asymptotically normal, training and generalization errors exhibit a symmetry, and model selection criteria (AIC/BIC) are valid. The principal birational invariants in this case are $\lambda = \nu = d/2$, with $d$ the parameter dimension.

Singular statistical models—including neural networks, mixture models, Boltzmann machines, and reduced-rank regression—are characterized by non-injective parameterization, parameter redundancy, and loss functions with analytic set-like optimal parameter sets. Here, the likelihood cannot be locally approximated as a normal distribution, the Fisher information matrix is degenerate, and classical information criteria fail. As the mapping’s singularities complicate inference, singular learning theory quantifies these via the RLCT and singular fluctuation, which determine asymptotic learning efficiency and error rates.

Quasi-regular cases (Yamada et al., 2011) bridge the two regimes: although singular, their parameter space admits a decomposition such that the Kullback-Leibler divergence behaves as a sum of squares in grouped variables. In this scenario, $\lambda = \nu = g/2$, where $g$ is the effective number of independent directions witnessing quadratic loss growth, and multiplicity $m = d - g + 1$. This maintains symmetry in training/generalization errors.

2. Core Geometric Invariants: Real Log Canonical Threshold and Singular Fluctuation

The real log canonical threshold $\lambda$ is defined from the largest pole of the zeta function $\zeta(z) = \int K(w)^z \phi(w) dw$, where $K(w)$ denotes the Kullback-Leibler divergence between true and modeled distributions, and $\phi$ is the prior (Yamada et al., 2011). The singular fluctuation $\nu$ is the leading term in the asymptotic expansion of the posterior log-likelihood variance. These invariants, extracted via resolution of singularities (e.g., Hironaka’s theorem), quantify the model's singular structure, with $\lambda$ sometimes interpreted as half the effective number of parameters.

For regular models, $\lambda = \nu = d/2$ yields classical rates. In singular and quasi-regular cases, these can be strictly smaller, corresponding to faster asymptotic convergence in Bayes generalization error and tighter redundancy bounds for MDL-based model selection (Urdshals et al., 14 Oct 2025).

3. Asymptotic Learning Theory and Information Criteria

In singular learning theory, the Bayes generalization error $Bg(n)$ and the cross-validation loss $C_{CV}(n)$ obey the relationship

$$Bg(n) + C_{CV}(n) = \frac{2\lambda}{n} + o\left(\frac{1}{n}\right)$$

where $\lambda$ is the RLCT and $n$ the sample size (Watanabe, 2010). WAIC (Widely Applicable Information Criterion) is defined as Bayes training loss plus the posterior variance term; both WAIC and Bayes leave-one-out cross-validation are theoretically shown to be asymptotically equivalent for model selection and hyperparameter tuning, including singular cases. This equivalence is achieved through matched expansions in functional cumulants $Y_k(n)$: $$C_{CV}(n) = -Y_1(n) + \frac{\beta}{2} Y_2(n) - \frac{\beta^2}{6} Y_3(n) + O_p(n^{-2})$$

$$WAIC(n) = -Y_1(n) + \frac{\beta}{2} Y_2(n) - Y_3(n) + O_p(n^{-2})$$

with discrepancy in higher-order terms only.

Deviance information criteria (DIC), by contrast, do not align asymptotically with Bayes generalization error in singular models, due to an incorrect accounting of effective parameter number and induced correlations (Watanabe, 2010). The relationship of cross-validation/WAIC with RLCT provides a robust foundation for model evaluation, surpassing DIC in singular settings.

4. Algebraic and Toric Geometry: Resolution of Singularities and Learning Curves

Singular learning theory utilizes toric and broader algebraic geometric techniques to analyze and “regularize” singularities in loss landscapes (Castillo-Villalba et al., 2017). The Kullback distance function $H(\omega)$ can be encoded as a polynomial in parameters, with exponents generating a lattice cone. The Hilbert basis of this cone provides minimal generators for toric reparameterizations, yielding monomial forms via morphisms: $H(g(u)) = u_1^{k_1} u_2^{k_2} \cdots u_d^{k_d}$ Resolution of singularities is formalized by Theorem 5.4, which asserts non-singularity if and only if the lattice cone’s determinant is unimodular (Det$=1$). This process exposes the effective parameter directions, learning coefficients, and pole multiplicities that appear in the asymptotic expansion of the learning curve: $$K(n) = \lambda_1 \log n + (m_1 - 1) \log \log n + C + o(1)$$ where $m_1$ is the pole multiplicity and $\lambda_1$ the leading RLCT. This machinery produces explicit methods for computing learning curves in complex high-dimensional models, extending theory pioneered by Watanabe et al.

5. Extensions to Deep Learning: Degeneracy and Practical Implications

Deep neural networks, due to parameter redundancy and scaling symmetries, manifest pronounced singularities (Murfet et al., 2020). The optimal set of parameters forms a non-manifold analytic variety, and both the Hessian of the KL divergence and Fisher information matrices are degenerate. Classical Laplace approximations, BIC, and quadratic local modeling are invalid. Singular learning theory instead predicts generalization error rates governed by RLCT: $$\mathbb{E}_n G(n) = \frac{\lambda}{n} + o\left(\frac{1}{n}\right)$$ Empirical work confirms that Bayesian predictive distributions (approximated by MCMC) yield consistently lower learning coefficients than traditional MAP estimators or Laplace approximations, particularly in high-capacity, overparameterized models. The RLCT quantifies the model’s generalization efficiency, and lower values correspond to superior performance.

The scope of singular theory now includes classification models using the Softmax function, where analysis of logit differences substitutes for raw outputs (Aoyagi, 22 Jan 2025).

6. Singular Learning Coefficients, Marginal Likelihood, and MDL-Based Model Compression

Learning coefficients ($\lambda$ for RLCT, $\theta$ for order/multiplicity) rigorously determine the efficiency of Bayesian inference and minimum description length redundancy in singular models (Aoyagi, 22 Jan 2025, Urdshals et al., 14 Oct 2025). With the marginal likelihood $p((x,y)^n)$ asymptotically expressed as

$$p((x,y)^n) \approx \prod_{i=1}^n p_0(x_i, y_i) \frac{(\log n)^{\theta(w_0)-1}}{n^{\lambda(w_0)}} (1 + o_p(1))$$

model selection criteria such as WBIC and sBIC, derived from singular theory, are appropriate replacements for conventional BIC.

Compressibility of neural networks is tightly determined by the local learning coefficient (LLC), with the volume of parameter basins scaling as $$V(\epsilon) \sim c \, \epsilon^{\lambda} (-\log \epsilon)^{m-1}$$ for a loss tolerance $\epsilon$. This leads to an asymptotic redundancy

$R_n = \lambda \log n - (m-1) \log \log n + O_p(1)$

Models with lower LLC are empirically shown to be more compressible under quantization and factorization schemes, aligning theoretical complexity with observed compression ratios (Urdshals et al., 14 Oct 2025).

7. Novel Singular Control and Computational Applications

Recent work extends singular learning theory to continuous-time reinforcement learning in singular stochastic control, where the control process is nondecreasing and causes state jumps (Liang et al., 27 Jun 2025). Optimal control laws are encoded as regions in the augmented state-time space, with optimality conditions formulated via variational inequalities and martingale characterizations. Q-learning algorithms are adapted to estimate value functions and control laws in both finite and infinite horizon cases, demonstrating convergence in simulated experiments.

The geometry of singularities arising in program codes (Turing machines) is now connected to statistical properties via the influence functions concept, bringing new perspectives to inductive simplicity and Bayesian inference on algorithmic structure (Murfet et al., 10 Apr 2025).

---

Singular learning theory comprehensively systematizes statistical inference and model selection in the presence of parameter nonidentifiability, degenerate Fisher information, and algebraic singularities, providing robust criteria and methods for modern learning machines including deep neural networks, mixture models, and control systems. Its geometric invariants dictate learning efficiency, compressibility, and proper evaluation across a range of complex, high-capacity models.