The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning

Abstract: In training a neural network with gradient descent (GD), each iteration induces a linear operator that governs first-order updates to a model's internal state variables. We define this operator as the Global Empirical Neural Tangent Kernel (NTK). In finite-width networks, the NTK is typically intractable to form, leading prior work to focus on restrictive settings such as tracking outputs only or taking infinite-width limits. Here, we study the structure of the NTK for a range of models. Formulating the model state as the solution to a single global implicit constraint, we derive the NTK as a product of two operators: K, accounting for immediate parameter-to-state interactions, and P, describing internal state-to-state dependencies. For a broad class of weight-based models, including RNNs and transformers, we prove a universal Kronecker-core theorem showing that K admits an exact, computable form given by the Gram matrix of weight-site variables. This core structure reveals that the NTK is structurally bottlenecked, constraining its effective rank and giving rise to a self-referential bias whereby GD preferentially learns within dominant modes of joint hidden and input activity. For recurrent models, we examine the spectrum of the NTK and show when it is biased and low-rank in space or time under the proposed decomposition. We further demonstrate that model dynamics at initialization bias the NTK, restricting learning and preventing task components from being learned effectively. Finally, we show that the NTK associated with a self-attention transformer is likewise structurally constrained to be low-rank. Overall, we show that the NTK possesses tractable structure that explains GD bias toward task solutions and the emergence of low-rank representations. To enable use of the NTK as a practical metric, we build kpflow, a library relying on randomized matrix-free numerical linear algebra.

• The paper introduces a novel operator-theoretic framework using an empirical Global-State NTK that quantifies self-referential bias in gradient descent learning.
• It decomposes NTKs into propagation and parameter sensitivity components, unveiling intrinsic bottlenecks that restrict learning to pre-initialized state axes.
• Empirical validations across RNNs, GRUs, and transformers demonstrate how input embedding and initialization strategies can alleviate low-rank expressivity issues.

The Global Empirical NTK: Structural Bottlenecks, Self-Referential Bias, and the Geometry of Gradient Descent

Introduction and Context

This work introduces a comprehensive operator-theoretic framework for analyzing how gradient descent (GD) reshapes internal model states by way of the empirical Neural Tangent Kernel (NTK) acting on the global state tensor. A central contribution is the generalization from the output-centric and infinite-width NTK perspective toward a finite-width, state-centric characterization—specifically the empirical Global-State NTK (NTKs). This operator models, at each iteration, how parameter updates induce corrections throughout all internal variables, providing a means to analyze learning bias and representational bottlenecks in arbitrary weight-based architectures, including RNNs and transformers.

Formal Decomposition of the Global-State NTK

The paper establishes that for any differentiable, parameterized model, internal variables (collectively the global state) are governed implicitly by a constraint $F(h, x, \theta) = 0$. Utilizing the implicit function theorem, the authors prove that the empirical NTK acting on the entire state (NTKs) admits a universal decomposition:

$\mathrm{NTKs} = P K P^{*}$

where $P$ is a state-to-state propagation operator (i.e., the resolvent or causal Green's operator for recurrent models), and $K$ links immediate parameter-to-state sensitivity. This operator-level view unifies first-order learning dynamics across discrete, continuous, and implicit architectures. Key results include:

• Proposition 1: Provides the explicit form of NTKs as above, establishing the generality of the decomposition.
• Corollary 1: For recurrent settings, $P$ is lower-triangular in time, enabling precise characterization of how state-space corrections integrate over time.

Universal Kronecker-Core Theorem and Gram Matrix Structure

The central theoretical result is the Kronecker-core theorem (Theorem 1), demonstrating that for a broad class of weight-based neural models, $K$ is an explicit Gram matrix over "weight-sites"—i.e., those internal variables directly multiplied by parameters (the functional loci of parameterization). Formally, $K = VV^{*} \otimes I$, with $V$ collecting all such sites over all data and time. This has fundamental implications:

• The global NTK's expressivity is bottlenecked by the span of $V$. Regardless of network depth or complexity, GD correction vectors are restricted to the linear span of the current model's weight-site activity.
• For RNNs, $V$ typically concatenates hidden activations and input trajectories, so task structure, initialization, and model dynamics jointly determine the accessible rank.
• For transformer self-attention blocks, the bottleneck emerges from (typically low-rank) input and intermediate representations ($\mathrm{NTKs} = P K P^{*}$0 for input and attention output).

Consequently, gradient descent updates are self-referentially biased towards currently occupied modes of the joint input-hidden state manifold, introducing a major limitation for learning directions absent at initialization.

Empirical and Theoretical Analysis of Bottlenecks

Self-Referential Bias

Corollary 2 rigorously formalizes self-referential bias: the only error directions accessible to GD at each step are those lying within the image of $\mathrm{NTKs} = P K P^{*}$1. Any component of the adjoint error orthogonal to this subspace is projected out by NTKs, causing slow or stalled learning for tasks whose target activity lives outside the initial dynamic span. Experimental results with RNN/GRU architectures show that unless task-relevant temporal/latent modes are present in $\mathrm{NTKs} = P K P^{*}$2 at initialization, learning proceeds only along the existent axes; others remain unlearned or require curvature-aware optimization (e.g., Shampoo) to escape associated plateaus.

Space-Time Decomposition

A rigorous treatment of rank decomposition establishes that the effective rank of state corrections is bounded by the minimal rank of NTK reductions over temporal (batch × time) or spatial (hidden unit) axes. This leads to:

$\mathrm{NTKs} = P K P^{*}$3

Thus, architectural and input-driven low-rank structure in either domain results in expressive bottlenecks along both.

Controlled Experimental Validation

The theoretical constructs are substantiated across settings:

• In GRUs trained on delayed-response tasks, alignment of the initial dynamic core $\mathrm{NTKs} = P K P^{*}$4 to task targets predicts learnability via SGD.
• RNN student-teacher setups show that increases in recurrent gain enrich temporal rank (until instability), while input rank enrichment ameliorates temporal bottlenecks when training only input weights.
• For transformers, the temporal rank of the NTK is sharply limited by input dimension and batch size, but not by attention width or head count; widening the input embedding relieves this, in precise mathematical accordance with the Kronecker-core theorem.

Practical Utility and Computational Tools

The analysis is operationalized in kpflow—a matrix-free operator library that enables scalable spectral and rank analysis via randomized SVD and trace estimation, sidestepping the intractability of explicit kernel construction in state spaces of practical size.

Implications and Future Research Directions

Theoretical Implications: The framework provides a concrete foundation for studying (and quantifying) inductive bias in GD-based learning, relating model architecture, input encoding, and initialization directly to the geometry of state corrections. The self-referential bottleneck elucidates why learned solutions are often aligned with dominant latent directions present at initialization or induced by prevalent input structure, explaining empirical observations such as spectral/temporal bias, neural collapse, and representational reuse across tasks.

Practical Implications: The Kronecker-core characterization provides diagnostic tools for predicting when models are likely to suffer from poor conditioning for a given task and points to practical interventions—e.g., enriching input representations, modifying initialization protocols, or employing second-order optimizers—that can ameliorate bottlenecks.

Speculative Future Directions:

• Extending the local-operator perspective to characterize how bottlenecks in the state-correcting NTK accumulate globally over the course of training, yielding persistent low-rank structure.
• Integrating nontrivial state-space phenomena such as bifurcations or attractor formation to better understand transitions in model expressivity.
• Generalizing to non-standard learning rules, restriction to subspaces (parameter freezing), or studying the impact of non-backpropagation-compatible plasticity mechanisms.

Conclusion

This work provides a mathematically rigorous, general, and computable framework for understanding the structure and limitations of gradient descent in finite-width neural models. Through the explicit decomposition of the empirical global-state NTK and demonstration of a universal Kronecker-core bottleneck, the study clarifies why learning is preferentially aligned with modes present in the model’s initial or current dynamics. The implications extend to model analysis, diagnostics of learning bottlenecks, and architectural/algorithmic design for improved expressivity and learnability.

Citation: "The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning" (2605.08746).