---
title: Weight-Induced Gram Operators in Deep Learning
url: https://www.emergentmind.com/topics/weight-induced-gram-operators
type: topic
---

# Weight-Induced Gram Operators in Deep Learning

Searching arXiv for the cited papers to ground the article in the relevant literature.
Weight-induced Gram operators are operator-valued summaries of training geometry that arise when gradient descent in deep ReLU networks is rewritten in example space rather than weight space. For feed-forward ReLU networks with fixed readout and quadratic loss, the learning dynamics can be expressed as a residual evolution driven by a finite hierarchy of layerwise Gram matrices and pullback operators determined by activation overlaps, conjugate-field correlators, and the weights themselves. In this formulation, the residual vector evolves under a closed collective dynamics in the training-set space, and from depth three onward closure requires a hierarchy of weight-induced Gram operators that mediate information transport across layers [2606.09744].

## 1. Network setting and operator-valued reformulation

The construction is stated for an \(L\)-hidden-layer feed-forward network with ReLU activations,
\[
u_0^\alpha \to v_1^\alpha=W^{(1)}u_0^\alpha \to u_1^\alpha=\mathrm{ReLU}(v_1^\alpha)\to \cdots \to u_L^\alpha=\mathrm{ReLU}(W^{(L)}u_{L-1}^\alpha),
\]
with fixed readout
\[
\hat Y^\alpha(t)=\frac{1}{N}\sum_{i=1}^N u_{L,i}^\alpha(t),
\]
targets \(Y^\alpha\), quadratic loss
\[
L=\frac{1}{M}\sum_\alpha (\hat Y^\alpha-Y^\alpha)^2,
\]
gradient descent on each \(W^{(\ell)}\) with step \(\eta\), and residuals \(r^\alpha=\hat Y^\alpha-Y^\alpha\) [2606.09744].

The central shift is to eliminate, as far as possible, the explicit weight variables from the learning dynamics and to rewrite gradient descent as a collective dynamics on quantities defined over example indices \(\alpha,\beta=1,\dots,M\). In this representation, the residual update is no longer described primarily as motion in the full parameter space, but as a coupled evolution of finitely many \(M\times M\) matrices. The key objects are layerwise Gram matrices in example-index space and, for deeper networks, a hierarchy of weight-induced pullback operators acting across layers.

The following objects organize the construction:

| Object | Definition | Role |
|---|---|---|
| \(Q^{(\ell-1)}_{\alpha\beta}\) | \(\frac{1}{N}u_{\ell-1}^\alpha\cdot u_{\ell-1}^\beta\) | activation overlap |
| \(S^{(\ell)}_{\alpha\beta}\) | \(\frac{1}{N}\sum_n [a_{\ell,n}^\alpha b_{\ell,n}^\alpha][a_{\ell,n}^\beta b_{\ell,n}^\beta]\) | conjugate-field correlator |
| \(G^{(\ell)}_{\alpha\beta}\) | \(Q^{(\ell-1)}_{\alpha\beta}\cdot S^{(\ell)}_{\alpha\beta}\) | layerwise Gram matrix |
| \(G_\ell^{\alpha\beta}\) | \((W^{(\ell+1)})^T D_\ell^{\alpha\beta} W^{(\ell+1)}\) | weight-induced pullback operator |

Here \(a_{\ell,n}^\alpha=1\{v_{\ell,n}^\alpha>0\}=1\{u_{\ell,n}^\alpha>0\}\), the backward fields \(b_\ell^\alpha\) are defined by backward propagation through ReLU masks, and \(b_L^\alpha\equiv 1\) [2606.09744].

## 2. Single-hidden-layer factorization

For a single hidden layer, the dynamics closes directly on the residuals. The fixed input geometry is encoded by the input Gram matrix
\[
K_{\mathrm{in}}(\alpha,\beta)=Q^{(0)}_{\alpha\beta}:=\frac{1}{N}u_0^\alpha\cdot u_0^\beta,
\]
while the dynamical gating structure is encoded by the co-activation overlap
\[
C_{\alpha\beta}(t)=A^{(1)}_{\alpha\beta}(t):=\frac{1}{N}\sum_i a_{1,i}^\alpha(t)a_{1,i}^\beta(t).
\]
The residuals satisfy
\[
r(t+1)=\left[I-\frac{2\eta}{M}G(t)\right]r(t),
\]
with collective kernel
\[
G_{\alpha\beta}(t)=Q^{(0)}_{\alpha\beta}\cdot A^{(1)}_{\alpha\beta}(t),
\qquad
G(t)=K_{\mathrm{in}}\circ C(t),
\]
where \(\circ\) denotes the Hadamard product [2606.09744].

This factorization isolates two distinct contributions. The matrix \(K_{\mathrm{in}}\) is fixed by the training data, whereas \(C(t)\) changes with the ReLU activation pattern. The result is a kernel that is neither purely data-geometric nor purely activation-driven, but a pointwise product of the two. In this sense, the single-layer case already exhibits the essential theme of the theory: learning is controlled by an example-space metric induced jointly by the data geometry and the current weight-dependent activation configuration.

A direct derivation follows from the chain rule. For \(L=1\),
\[
\frac{\partial \mathcal L}{\partial W_{ij}}=\frac{2}{MN}\sum_\beta r^\beta a_i^\beta u_{0,j}^\beta,
\qquad
\Delta W_{ij}= -\frac{2\eta}{MN}\sum_\beta r^\beta a_i^\beta u_{0,j}^\beta,
\]
so that
\[
\Delta u_i^\alpha
=
a_i^\alpha\sum_j \Delta W_{ij}u_{0,j}^\alpha
=
-\frac{2\eta}{M}a_i^\alpha\sum_\beta Q^{(0)}_{\alpha\beta}a_i^\beta r^\beta.
\]
Averaging over \(i\) yields
\[
\Delta \hat Y^\alpha
=
-\frac{2\eta}{M}\sum_\beta
\left[Q^{(0)}_{\alpha\beta}A^{(1)}_{\alpha\beta}\right]r^\beta,
\]
which is the residual update in closed form [2606.09744].

## 3. Layerwise Gram metrics in deep networks

For \(L\) hidden layers, the residual update decomposes into \(L\) layerwise contributions. Define
\[
Q^{(\ell-1)}_{\alpha\beta}=\frac{1}{N}u_{\ell-1}^\alpha\cdot u_{\ell-1}^\beta
\]
and
\[
S^{(\ell)}_{\alpha\beta}
=
\frac{1}{N}\sum_n
[a_{\ell,n}^\alpha b_{\ell,n}^\alpha]
[a_{\ell,n}^\beta b_{\ell,n}^\beta].
\]
Then
\[
\Delta r^\alpha
=
r^\alpha(t+1)-r^\alpha(t)
=
-\frac{2\eta}{M}\sum_{\ell=1}^L \sum_\beta
\left[Q^{(\ell-1)}_{\alpha\beta}S^{(\ell)}_{\alpha\beta}\right]r^\beta(t).
\]
Equivalently, with
\[
G^{(\ell)}_{\alpha\beta}(t):=
Q^{(\ell-1)}_{\alpha\beta}(t)\cdot S^{(\ell)}_{\alpha\beta}(t),
\]
one obtains
\[
r(t+1)=\left[I-\frac{2\eta}{M}\sum_{\ell=1}^L G^{(\ell)}(t)\right]r(t),
\]
or in the continuous-time limit,
\[
\dot r(t)=-\frac{2}{M}\sum_{\ell=1}^L G^{(\ell)}(t)\,r(t).
\]
Thus the single-layer factorization \(G=K_{\mathrm{in}}\circ C\) generalizes to deep networks by summing \(L\) Hadamard-factorized terms \(G^{(\ell)}=Q^{(\ell-1)}\circ S^{(\ell)}\) [2606.09744].

This decomposition has two immediate consequences. First, each layer contributes its own example-space metric rather than all training effects being compressed into a single aggregate kernel. Second, the dynamics remains closed in a finite set of \(M\times M\) matrices \(\{G^{(\ell)}(t)\}_{\ell=1}^L\). No higher-order statistics of the residuals are needed. The gradient flow in function space is therefore fully determined by the current Gram operators \(G^{(\ell)}(t)\).

The role of the conjugate-field correlator \(S^{(\ell)}\) is especially significant in depth. In the single-layer case, gating information is carried by the co-activation matrix \(A^{(1)}\). In deeper networks, this generalizes to a combined co-activation/backpropagated object, since each neuron’s contribution to the residual update depends not only on whether it is active, but also on how downstream structure propagates backward through the network.

## 4. Backward pullback recursion and the hierarchy of operators

From depth three onward, closure requires explicit operator-valued quantities that transport geometry backward through the network. For every layer \(\ell=1,\dots,L-1\), define the projector
\[
D_\ell^{\alpha\beta}:=\mathrm{diag}(a_{\ell+1}^\alpha\circ a_{\ell+1}^\beta),
\]
which projects onto neurons simultaneously active for examples \(\alpha\) and \(\beta\) at layer \(\ell+1\). The simplest nontrivial pullback operator at layer \(\ell\) is then
\[
G_\ell^{\alpha\beta}:=(W^{(\ell+1)})^T D_\ell^{\alpha\beta} W^{(\ell+1)}.
\]
Interpreting the top layer as \(G_L^{\alpha\beta}=I\), these operators satisfy the backward recursion
\[
G_L^{\alpha\beta}=I,
\qquad
G_\ell^{\alpha\beta}=(W^{(\ell+1)})^T G_{\ell+1}^{\alpha\beta} W^{(\ell+1)},
\]
for \(\ell=L-1,\dots,1\) [2606.09744].

At fixed ReLU masks and neglecting threshold crossings, this recursion is obtained by substituting successive next-layer expressions into the projector form. The result shows that each \(G_\ell^{\alpha\beta}\) is symmetric, positive semidefinite, and encodes how the co-activation geometry at layer \(\ell+1\) is pulled back to layer \(\ell\) via the weight map.

The significance of the hierarchy is structural rather than merely notational. The conjugate-field dynamics is governed by operators satisfying a backward pullback recursion, of which the weight-induced Gram operators are the first nontrivial instances [2606.09744]. In practical terms, deep layers do not affect earlier layers only through scalar summary statistics; they transmit a recursively transformed geometry determined jointly by activation masks and weights. This is the operator-theoretic core of the deep-network generalization.

## 5. Spectral interpretation, finite-width adaptivity, and relation to feature evolution

Because
\[
\dot r = -\sum_\ell G^{(\ell)} r
\]
up to the overall factor \(2/M\), the spectrum of the layerwise Gram matrices controls the decay of residual modes. The decomposition exposes which modes of the residual vector decay fastest: those in the top eigendirections of the \(G^{(\ell)}\). It also shows that deep layers communicate their co-activation geometry backward through the chain of \(W\)’s, so that earlier layers see an effective kernel shaped by all deeper layers. In the infinite-width NTK regime each \(G^{(\ell)}\) converges to a fixed limit, recovering the well-known fixed kernel; at finite width they evolve during training, giving a richer, adaptive metric [2606.09744].

A related but distinct 2026 framework studies the weight Gram matrix
\[
G^\ell=(W_\ell)^T W_\ell
\]
as the key object capturing feature dynamics in deep networks. There the Feature Learning Equation
\[
\nabla_h f\cdot h^\top = W^\top \nabla_W f
\]
translates weight-space updates into feature-space updates, and the leading-order shift in \((W_\ell^+)^T W_\ell^+\) matches the Virtual Covariance Shift up to \(O(\gamma^2)\). That framework also introduces Target Linearity and argues that deep networks sequentially transform representations toward target-linear structure [2605.06258].

The two viewpoints operate on different spaces. In the residual-dynamics formulation, the principal objects are \(M\times M\) matrices indexed by training examples and closed under the learning dynamics. In the feature-centric formulation, the principal object is the layerwise weight Gram matrix \(W_\ell^T W_\ell\), interpreted as encoding the covariance-type update that hypothetical direct feature optimization would produce. A plausible implication is that the two approaches describe complementary closures: one in example space through residual modes, the other in feature space through virtual covariance and target alignment.

## 6. Related operator-theoretic usages and terminological ambiguities

Outside deep-learning theory, closely related terminology appears in frame theory. For Bessel sequences \(\Phi=\{\phi_i\}\) and \(\Psi=\{\psi_i\}\), and any bounded operator \(U\), the \(U\)-cross Gram operator is
\[
G_{U,\Phi,\Psi}=T_\Phi^* U T_\Psi,
\]
with entries
\[
(G_{U,\Phi,\Psi})_{i,j}=\langle U\psi_j,\phi_i\rangle.
\]
Specializing to the diagonal weight operator \(U=\mathrm{diag}(w_n)\) on \(\ell^2\) gives
\[
(G_{U,\Phi,\Psi})_{i,j}=w_j\langle \psi_j,\phi_i\rangle,
\]
and for \(\Phi=\Psi\) one obtains
\[
G_{w,\Phi}=\{\,w_j\langle \phi_j,\phi_i\rangle\,\}_{i,j\in I},
\]
described as the standard “weight-induced Gram operator” in frame theory [1804.00203].

In that setting, the main questions are Schatten \(p\)-class membership, invertibility, Moore–Penrose pseudoinverses, and perturbation stability. For example, if \(\Phi,\Psi\) are Riesz bases and \(w_n\neq 0\) for all \(n\), then
\[
G_{w,\Phi,\Psi}^{-1}=G_{1/w,\widetilde\Psi,\widetilde\Phi},
\]
while small perturbations of the weights or frame sequences preserve invertibility under explicit norm conditions [1804.00203]. A fusion-frame analogue replaces vectors by weighted subspaces and defines the \(U\)-fusion cross Gram matrix
\[
\mathcal G_{U,W,V}=T_W^* U T_V,
\]
with corresponding results on invertibility, pseudo-invertibility, and stability [1711.00091].

These usages are mathematically adjacent but conceptually different. In frame and fusion-frame theory, a weighted Gram operator is an operator representation problem in Hilbert space. In deep-network learning dynamics, weight-induced Gram operators are dynamical objects built from ReLU masks, activation overlaps, conjugate fields, and weight pullbacks, and they serve to close gradient descent in function space. A common misconception is to identify all such objects with a static matrix \(W^T W\) or with a fixed kernel. The deep-network theory instead emphasizes a time-dependent hierarchy of operators whose evolution is itself part of the learning process [2606.09744].

Source: https://www.emergentmind.com/topics/weight-induced-gram-operators