Local Learning Coefficient in Singular Models

• Local Learning Coefficient is defined as the measure of effective parameter volume near a loss minimizer in singular learning theory.
• It replaces traditional heuristics by quantifying degeneracy and yielding refined local corrections in Bayesian free energy and generalization error.
• Empirical methods like SGLD-based estimators enable practical measurement of LLC in deep networks despite complex singular structures.

The local learning coefficient (LLC) is the local counterpart of the learning coefficient, or real log canonical threshold (RLCT), in singular learning theory. It quantifies the singularity-driven effective complexity of a statistical model in a neighborhood of a particular minimizer, rather than at the level of the model class as a whole. In singular models such as neural networks, where identifiability fails and Fisher information can be singular, the LLC replaces regular-model heuristics based on parameter count or quadratic curvature. It enters the local asymptotics of free energy, marginal likelihood, and Bayesian generalization error, and recent work has turned it from a largely algebro-geometric object into an estimable quantity for modern deep networks, while also extending its exact or upper-bound computation in several analytically tractable model classes (Lau et al., 2023, Furman et al., 2024, Kurumadani, 2024, Kurumadani, 13 Mar 2026, Chen et al., 25 Apr 2025, Wang et al., 2024, Cai, 21 Jun 2026).

1. Definition within singular learning theory

For a population loss $L(w)$ with minimizer $w^*$, the LLC is defined from the scaling of the local sublevel-set volume

$$V_{w^*}(\epsilon)=\mathrm{Vol}\{w\in N(w^*)\mid L(w)-L(w^*)<\epsilon\}\propto \epsilon^{\lambda(w^*)}(-\log\epsilon)^{m(w^*)-1},$$

as $\epsilon\to 0$. Here $\lambda(w^*)$ is the local learning coefficient and $m(w^*)$ is its multiplicity. Equivalently, under resolution of singularities, the corresponding local zeta structure yields the same rational $\lambda(w^*)$, with multiplicity given by the order of the dominant pole. The global learning coefficient is the infimum of local values over the realizable set, so LLC is the local analytic invariant from which the model-level RLCT is assembled (Kurumadani, 2024, Chen et al., 25 Apr 2025).

Its statistical importance comes from local free-energy and generalization expansions. In local form,

$$F_n(B_\gamma(w^*))=nL_n(w^*)+\lambda(w^*)\log n-(m(w^*)-1)\log\log n+O_P(1),$$

while in the Bayesian setting the leading correction to local test loss is of order $\lambda(w^*)/n$. In regular identifiable models, $\lambda=d/2$ and $w^*$0, recovering the familiar BIC-type penalty. In singular models, $w^*$1 can be strictly smaller than $w^*$2, so it acts as an effective complexity that reflects degeneracy of the loss landscape rather than ambient parameter dimension (Chen et al., 25 Apr 2025, Wang et al., 2024).

Semi-regular analyses make this interpretation explicit. When the Fisher information has rank $w^*$3 and the remaining directions first appear at order $w^*$4, the local RLCT can take the form

$w^*$5

so each regular direction contributes $w^*$6, while each higher-order singular direction contributes $w^*$7. This gives a precise local notion of “effective dimension” that is unavailable from Hessian rank alone (Kurumadani, 2024).

2. Local geometry, degeneracy, and invariance

The LLC is often interpreted as a fractional dimension of a basin of good parameters. When $w^*$8, the scaling reduces to $w^*$9, so increasing the loss tolerance by a factor $$V_{w^*}(\epsilon)=\mathrm{Vol}\{w\in N(w^*)\mid L(w)-L(w^*)<\epsilon\}\propto \epsilon^{\lambda(w^*)}(-\log\epsilon)^{m(w^*)-1},$$0 increases the allowable parameter volume by $$V_{w^*}(\epsilon)=\mathrm{Vol}\{w\in N(w^*)\mid L(w)-L(w^*)<\epsilon\}\propto \epsilon^{\lambda(w^*)}(-\log\epsilon)^{m(w^*)-1},$$1. An information-theoretic restatement is

$$V_{w^*}(\epsilon)=\mathrm{Vol}\{w\in N(w^*)\mid L(w)-L(w^*)<\epsilon\}\propto \epsilon^{\lambda(w^*)}(-\log\epsilon)^{m(w^*)-1},$$2

which treats $$V_{w^*}(\epsilon)=\mathrm{Vol}\{w\in N(w^*)\mid L(w)-L(w^*)<\epsilon\}\propto \epsilon^{\lambda(w^*)}(-\log\epsilon)^{m(w^*)-1},$$3 as the marginal number of bits required to halve a small error tolerance. Lower $$V_{w^*}(\epsilon)=\mathrm{Vol}\{w\in N(w^*)\mid L(w)-L(w^*)<\epsilon\}\propto \epsilon^{\lambda(w^*)}(-\log\epsilon)^{m(w^*)-1},$$4 therefore means more degeneracy, or a larger near-optimal volume (Furman et al., 2024).

This geometric content is not reducible to counting flat directions. A standard counterexample uses

$$V_{w^*}(\epsilon)=\mathrm{Vol}\{w\in N(w^*)\mid L(w)-L(w^*)<\epsilon\}\propto \epsilon^{\lambda(w^*)}(-\log\epsilon)^{m(w^*)-1},$$5

whose minimizer set contains both a line and a highly degenerate point. The line contributes a local free energy of $$V_{w^*}(\epsilon)=\mathrm{Vol}\{w\in N(w^*)\mid L(w)-L(w^*)<\epsilon\}\propto \epsilon^{\lambda(w^*)}(-\log\epsilon)^{m(w^*)-1},$$6, while the degenerate point contributes $$V_{w^*}(\epsilon)=\mathrm{Vol}\{w\in N(w^*)\mid L(w)-L(w^*)<\epsilon\}\propto \epsilon^{\lambda(w^*)}(-\log\epsilon)^{m(w^*)-1},$$7. The zero-dimensional point therefore dominates asymptotically despite having fewer obvious “flat directions,” showing that local degeneracy is an algebro-geometric property rather than a naive dimension count (Lau et al., 2023).

The quantity is also an invariant in a precise sense. In singular learning theory, $$V_{w^*}(\epsilon)=\mathrm{Vol}\{w\in N(w^*)\mid L(w)-L(w^*)<\epsilon\}\propto \epsilon^{\lambda(w^*)}(-\log\epsilon)^{m(w^*)-1},$$8 is a birational invariant of the analytic variety defined by the local KL surplus. For deep networks, this connects LLC to parameter symmetries such as adjacent-layer ReLU rescaling,

$$V_{w^*}(\epsilon)=\mathrm{Vol}\{w\in N(w^*)\mid L(w)-L(w^*)<\epsilon\}\propto \epsilon^{\lambda(w^*)}(-\log\epsilon)^{m(w^*)-1},$$9

Empirically, stochastic LLC estimators with appropriate preconditioning preserve this invariance over rescalings spanning 8 orders of magnitude, which distinguishes LLC from curvature surrogates that are sensitive to parameterization (Furman et al., 2024).

3. Estimation methodologies

The modern empirical literature operationalizes LLC through tempered local posteriors. Starting from Watanabe’s WBIC relation at inverse temperature $\epsilon\to 0$0, one estimates the learning coefficient from an energy gap under a Gibbs posterior. The localized form introduced for scalable use adds a quadratic confinement around a chosen minimizer:

$\epsilon\to 0$1

with estimator

$\epsilon\to 0$2

Replacing full-gradient MCMC with SGLD makes this practical for large datasets and models, using updates of the form

$\epsilon\to 0$3

where $\epsilon\to 0$4 is an optional preconditioner. In deep linear networks, this approach was shown to recover theoretical LLC values up to 100M parameters and dataset size $\epsilon\to 0$5, with step size $\epsilon\to 0$6 identified as the critical hyperparameter and a MALA-style acceptance diagnostic recommended in the 0.9–0.95 range (Lau et al., 2023, Furman et al., 2024).

A distinct difficulty appears off equilibrium. Mean-energy estimators require an additive baseline $\epsilon\to 0$7, typically interpreted as a local minimum. During transient training phases this baseline is unknown, and replacing it with the lowest noisy mini-batch loss induces a systematic minimization bias. The Shift-Invariant Variance Estimator (SIVE) removes the baseline structurally by exploiting

$\epsilon\to 0$8

and then subtracts evaluation noise by the Law of Total Variance. With grouped noisy evaluations $\epsilon\to 0$9 and within-group variances $\lambda(w^*)$0, the estimator is

$\lambda(w^*)$1

On analytically tractable toy models, SIVE recovers the expected finite-temperature geometric signal in regimes where anchored mean estimators fail, and in MNIST MLP experiments it yields a non-monotonic online geometric diagnostic over training (Cai, 21 Jun 2026).

Finite-$\lambda(w^*)$2 estimation remains subtle. Large-scale studies generally target $\lambda(w^*)$3 alone rather than jointly estimating multiplicity $\lambda(w^*)$4, because the subleading term $\lambda(w^*)$5 can bias single-temperature estimators. Burn-in, local stationarity, and localization strength $\lambda(w^*)$6 are therefore not implementation details but part of what defines the measured local geometry (Furman et al., 2024, Cai, 21 Jun 2026).

4. Exact formulas and upper bounds in analyzable models

Deep linear networks provide a rare large-scale setting where exact theory and numerical estimation can be compared directly. Under realizability and relatively finite variance, the learning coefficient admits a closed-form expression in terms of the rank $\lambda(w^*)$7 of the realized linear map and a combinatorial subset of layers. This formula, due to Aoyagi and used as ground truth in subsequent experiments, enabled empirical validation of SGLD-based LLC estimation across thousands of randomly generated DLNs from 1k to 100M parameters. A notable qualitative finding is that $\lambda(w^*)$8 tends to decrease with depth even as parameter count increases, while multiplicities $\lambda(w^*)$9 are common (Furman et al., 2024).

For semi-regular models, a general exact formula is available under a differential-structure assumption on the log-likelihood ratio. If the Fisher information rank at a realizable point is $m(w^*)$0 and the remaining coordinates vanish up to order $m(w^*)$1, then the local RLCT is

$m(w^*)$2

with multiplicity 1 when the relevant independence condition holds. Two-parameter semi-regular models therefore fall into a complete classification: the regular case with $m(w^*)$3, an order-$m(w^*)$4 singular case with $m(w^*)$5, and a locally one-dimensional realizable manifold with $m(w^*)$6 (Kurumadani, 2024).

A related extension addresses many non-singular points of the realizable set by separating parameters into active coordinates $m(w^*)$7 and “constant” realizable coordinates $m(w^*)$8. Under this straightened local structure, the same form

$m(w^*)$9

yields a computable local RLCT and therefore an upper bound on the model’s global learning coefficient. The method recovers known reduced-rank regression values in specific cases and gives upper bounds for mixed binomial models (Kurumadani, 2024).

Three-layer neural networks require a different approach because singular realization points are dominated by redundant hidden units, duplicated or zero weights, and reduced-rank configurations. For these models, recent work derives an upper-bound formula for $\lambda(w^*)$0 at singular points that can be interpreted as a counting rule under budget constraints and demand–supply constraints. The theorem applies to general analytic activation functions, including swish and polynomial activations, and becomes exact when the input dimension is one. In that setting it partially resolves discrepancies between earlier nonsingular-point formulas and known exact RLCT values (Kurumadani, 13 Mar 2026).

5. Modern extensions: sequence models and refined LLCs

In transformer sequence models, LLC has been linked to a modal description of the data distribution. Conditional sequence distributions can be embedded in a Hilbert-space framework and decomposed into modes by singular value decomposition of the conditional operator. The resulting theory shows that SGLD-based LLC estimates are insensitive to sufficiently small-amplitude modes of the data distribution. Consequently, the estimated LLC characterizes the geometry not of the full true distribution $\lambda(w^*)$1, but of an effective coarse-grained distribution $\lambda(w^*)$2 obtained by truncating modes below a data-dependent threshold. In this picture, inverse temperature $\lambda(w^*)$3 acts as a resolution dial: higher $\lambda(w^*)$4 resolves finer modal structure, while lower $\lambda(w^*)$5 measures coarser effective geometry (Chen et al., 25 Apr 2025).

This modal account also clarifies why empirical LLC estimates can remain stable even when the probed parameter is not a strict minimizer of the full population loss. If training has captured the dominant modes, a parameter may already be a local minimizer of the effective loss $\lambda(w^*)$6 even though it is not a minimizer of the full $\lambda(w^*)$7. The estimator then returns the LLC of that effective potential rather than failing outright. This is a reinterpretation of practical LLC estimation rather than a rejection of the original SLT object (Chen et al., 25 Apr 2025).

A second major extension is the refined LLC (rLLC), which restricts the local geometry to parameter subspaces and/or data subdistributions. Weight-refined LLCs probe a component such as a single attention head while freezing all other parameters; data-refined LLCs change the data distribution to a subset such as GitHub code; combined weight-and-data-refined LLCs do both simultaneously. In a two-layer attention-only transformer, these refinements were used to track differentiation and specialization of attention heads across training. Head-level wrLLC trajectories cluster by functional type, final wrLLC correlates with the number of memorized multigrams for multigram heads, drLLCs on code separate induction heads from multigram heads, and combined rLLC analyses supported the discovery of a previously unidentified multigram circuit associated with nested Dyck-like structure (Wang et al., 2024).

6. Limitations, misconceptions, and terminological scope

Several limitations are intrinsic. Exact LLC values are known only for restricted model classes, large-scale estimators usually do not estimate multiplicity $\lambda(w^*)$8 directly, and practical accuracy depends on localization, temperature, preconditioning, burn-in, and chain stability. In nonlinear deep networks without a theoretical ground truth, empirical validation must rely on self-consistency checks such as seed stability, agreement across samplers, or invariance under known symmetries. Existing exact and upper-bound results also assume analyticity, realizability near the relevant stratum, and generic linear independence conditions that can fail in nongeneric configurations (Furman et al., 2024, Kurumadani, 13 Mar 2026).

A common misconception is to treat LLC as a synonym for Hessian flatness or for parameter count. SLT results make neither identification. Local free energy depends on $\lambda(w^*)$9, not on the raw dimension, and the toy singular examples show that lower-dimensional strata can dominate higher-dimensional ones when their singularity structure is stronger. Nor does a smaller LLC automatically imply a smaller test loss, since the local Bayesian loss has the form $$F_n(B_\gamma(w^*))=nL_n(w^*)+\lambda(w^*)\log n-(m(w^*)-1)\log\log n+O_P(1),$$0, so the bias term $$F_n(B_\gamma(w^*))=nL_n(w^*)+\lambda(w^*)\log n-(m(w^*)-1)\log\log n+O_P(1),$$1 remains decisive when minima differ in population loss (Lau et al., 2023).

The phrase “local learning coefficient” also has unrelated meanings outside the SLT literature. It has been used for scalar coefficients multiplying monomials in polynomial local learning rules (Baldi et al., 2015), for the client learning rate in local update methods for federated optimization (Charles et al., 2020), and for local excess-risk rate exponents in localized SVMs (Eberts et al., 2015). In contemporary singular-learning usage, however, LLC denotes the local RLCT governing the asymptotic Bayesian geometry of a neighborhood in parameter space.