Features at Convergence Theorem

• FACT is a theoretical framework that specifies self-consistency conditions ensuring that key features converge in neural networks, operator algebra, and optimization.
• It employs a precise equation linking weight matrices, activations, and gradients to guarantee structural integrity at convergence.
• FACT principles inform algorithm design and analysis by translating local feature accuracy into global convergence guarantees across various practical and mathematical applications.

The Features at Convergence Theorem (FACT) designates a class of results across mathematics and theoretical machine learning that identify structural properties—“features”—which must be satisfied by objects or iterates at a point of convergence under certain conditions. This principle is exemplified in recent neural network theory, where FACT describes a precise self-consistency equation governing neural network feature representation at convergence with weight decay, as well as in operator theory, analysis, and optimization, where analogues establish which convergence behaviors or object properties are guaranteed once select “feature” conditions are verified.

1. Formal Statement and Theoretical Foundations

The archetypal FACT, as introduced for neural networks, asserts that for any weight matrix $W$ within a network trained with nonzero $\ell_2$ weight decay, the following holds at convergence: $$W^\top W = -\frac{1}{n\lambda}\sum_{i=1}^{n} (\nabla_h \ell_i) h(x_i)^\top$$ Here, $n$ is the number of training points, $\lambda$ is the weight decay coefficient, $h(x_i)$ represents the layer’s activation on $x_i$, and $\nabla_h \ell_i$ is the gradient of the loss with respect to the same layer’s input. This self-consistency equation characterizes the feature matrix $W^\top W$ in terms of both forward activations and backward error signals. The result is derived by setting the gradient of the weight-decayed empirical risk loss to zero—capturing the critical point condition at convergence (2507.05644).

Beyond neural networks, related FACT principles surface in functional analysis, operator algebras, and stochastic optimization:

• In operator theory, if a net of self-adjoint operators $(H_i)$ converges weakly, and so does its image under a single strictly convex function $f$, then $H_i$ must converge strongly—a transfer of “feature” convergence to global convergence (1410.6800).
• In stochastic optimization, verifying a vanishing sequence of “stationarity measures” is enough to guarantee convergence for broad algorithm families (2206.03907). These all instantiate the general FACT philosophy: that convergence (weak or local) of certain features inexorably implies convergence (strong, global, or in value) of the object as a whole.

2. Mathematical Characterization and Self-Consistency

In FACT for neural networks, the fundamental identity ties together three matrix- or tensor-valued objects:

• The feature matrix $W^\top W$.
• The set of input activation vectors $\{h(x_i)\}$.
• The set of backpropagated gradient vectors $\{\nabla_h \ell_i\}$.

The self-consistency arises as follows. Let $L = \sum_i \ell_i + \frac{\lambda n}{2} \|W\|_F^2$ denote the empirical loss plus weight decay. Taking the derivative $\nabla_W L$ and setting it to zero at the critical point yields: $$\sum_i \nabla_h \ell_i h(x_i)^\top + \lambda n W^\top W = 0 \implies W^\top W = -\frac{1}{n\lambda} \sum_i \nabla_h \ell_i h(x_i)^\top$$ This formula is universal for any matrix $W$ that appears only through matrix multiplication in the forward pass.

In operator theory, the corresponding “feature” is the action of a strictly convex continuous function on operators: if $f(H_i) \to f(H)$ weakly in operator topology, then all bounded continuous functions $y(H_i) \to y(H)$ strongly; thus, the “test function” $f$ encodes features diagnostic of the whole net (1410.6800).

3. Empirical and Analytical Validation

FACT has been validated empirically in neural networks:

• Experiments with deep fully connected networks on datasets such as MNIST and CIFAR-10 show near-perfect agreement (Pearson correlation $\approx$ 0.999 on MNIST) between the left-hand side ($W^\top W$) and the FACT right-hand side (2507.05644).
• The FACT remains robust across architectures and learning tasks, even in cases (synthetic or adversarial) where alternative predictions (e.g., the Neural Feature Ansatz, NFA) diverge.

In the analysis domain, comparison theorems similarly validate that checking the “feature” convergence for a function $f$ is sufficient to guarantee convergence of broader function classes (1410.6800). In stochastic optimization, plug-in convergence analyses confirm that a collection of “stationarity” quantities, when driven to zero, ensures convergence to optimality across various algorithmic schemas (2206.03907).

4. Algorithmic Implementations and Practical Consequences

The FACT has informed the design of learning algorithms that enforce or exploit feature self-consistency:

• The FACT-RFM (Recursive Feature Machine) algorithm replaces previous fixed-point updates (based on the NFA) with ones derived from the FACT self-consistency. An example update:

$$W_{t+1} \leftarrow \left(\text{FACT}_t\,\text{FACT}_t^\top\right)^{1/4}$$

where $$\text{FACT}_t = -\frac{1}{n\lambda}\sum_{i} \nabla_h \ell_i\, h(x_i)^\top$$, symmetrized to ensure positive semi-definiteness. Geometric averaging variants (using previous $W_t^\top W_t$) are introduced for stability.

• Empirical results on tabular datasets (UCI repository) and synthetic benchmark tasks (sparse parity learning, modular arithmetic) demonstrate that FACT-RFM achieves competitive or superior performance relative to classical matrix methods or prior feature-learning algorithms (2507.05644).
• In operator algebras, convergence theorems underpin computational approaches in numerical analysis, quantum computing, and quantum probability—providing diagnostics for when weak convergence “upgrades” to strong convergence simply by verifying feature convergence (1410.6800, 1504.03829).

5. Broader Context and Interpretations in Analysis and Optimization

The FACT paradigm appears in several forms across mathematics:

• In functional analysis, convergence criteria based on critical functionals (e.g., for almost surely convergent sequences of measurable functions) serve as necessary and sufficient “feature” conditions that dictate global convergence (1507.04020).
• In stochastic optimization, the unified convergence theorem reduces the analysis for a spectrum of algorithms (SGD, random reshuffling, stochastic proximal gradient) to verifying a set of plugin-type conditions on the stationarity measure—directly paralleling the FACT spirit (2206.03907).
• In the paper of extreme values of dependent sequences, “complete convergence” results specify the limiting point process in terms of clustering and temporal structure—one can interpret the limiting structure (timing and shape of clusters) as features at convergence (1508.03520).

These exemplify the transfer principle: feature convergence for selected test quantities enforces stronger global properties.

6. Implications for Theory and Applications

The FACT implies several consequences for machine learning, mathematics, and optimization:

• Understanding feature matrices at convergence aids in interpreting deep networks, quantifying “implicit bias” introduced by weight decay, and developing diagnostic and analytic tools for model selection and training behavior.
• Algorithms explicitly harnessing FACT—by directly enforcing the self-consistency equation—can yield superior or more stable performance on problems where adaptive feature learning is crucial, including tabular data, sparse combinatorial problems, and tasks exhibiting grokking or learning phase transitions (2507.05644).
• In abstract mathematical settings, recognizing which features “control” convergence allows for simplification and unification of many convergence arguments, promoting modular analysis of complex algorithms.

A plausible implication is that FACT-like theorems will continue to play a central explanatory role across domains requiring the transfer of convergence from features to objects, and that explicit use of such self-consistency relations may inspire new algorithms and analytical tools in both deep learning and broader areas of applied mathematics.

7. Summary Table: Representative FACT Instances

Domain	Feature at Convergence	Consequence
Neural Network Learning	$W^\top W$ self-consistency (FACT)	Characterizes learned features, drives RFM
Operator Theory	Weak convergence under $f$	Implies strong convergence of operators
Stochastic Optimization	Stationarity measure $\to 0$	Guarantees convergence of algorithmic iterates
Harmonic Analysis	Tail functional vanishes	Almost sure convergence of function sequence

FACT thus crystallizes the modern convergence principle: convergence of select structural “features”—often accessible, interpretable, or empirically verifiable—ensures, and can be equated to, convergence of entire functional or algorithmic objects. This perspective bridges classical analytic results, modern learning theory, and the design of new algorithmic frameworks.