---
title: Feature Learning Equation Overview
url: https://www.emergentmind.com/topics/feature-learning-equation
type: topic
---

# Feature Learning Equation Overview

Several papers explicitly note that they do not introduce a single formula literally called the “feature learning equation,” while others attach the term to a paper-specific identity. A prominent instance is the higher-order score-function relation
\[
\mathcal S_m(x)=(-1)^m\frac{\nabla^{(m)}p(x)}{p(x)},
\qquad
\mathbb E[y\cdot \mathcal S_m(x)]=\mathbb E[\nabla_x^{(m)}G(x)],
\qquad
G(x)=\mathbb E[y\mid x],
\]
which turns unlabeled structure in \(p(x)\) into discriminative derivative information [1412.6514]. This suggests that “Feature Learning Equation” is best understood as a family of formal devices—identity, objective, fixed-point system, or dynamical law—used to specify what feature learning is, what information it recovers, and how that information changes during training [1504.00083].

## 1. Score-function features as a discriminative identity

In "Score Function Features for Discriminative Learning" the central learned quantity is the \(m\)-th order score-function feature
\[
\mathcal{S}_m(x) = (-1)^m \frac{\nabla^{(m)} p(x)}{p(x)},
\]
for \(x\sim p(\cdot)\), together with the conditional mean
\[
\mathbb E[y\mid x]:=G(x).
\]
The paper’s main discriminative quantity is
\[
\mathbb E\big[\nabla_x^{(m)} G(x)\big], \qquad m\ge 1,
\]
and the bridge from unlabeled to labeled learning is the exact identity
\[
\mathbb E\big[y\cdot \mathcal S_m(x)\big]=\mathbb E\big[\nabla^{(m)}G(x)\big].
\]
This is the paper’s main “feature learning equation” [1412.6514].

The definition recovers the classical score function at first order:
\[
\mathcal S_1(x)=-\frac{\nabla p(x)}{p(x)}=-\nabla\log p(x),
\]
while second order yields
\[
\mathcal S_2(x)=\frac{\nabla^{(2)}p(x)}{p(x)}
=\nabla^{(2)}\log p(x)+\nabla\log p(x)\otimes \nabla\log p(x)
=\nabla^{(2)}\log p(x)+\mathcal S_1(x)\otimes \mathcal S_1(x).
\]
More generally, \(\mathcal S_m(x)\) is a function of the higher-order log-density derivatives \(\nabla_x^{(n)}\log p(x)\) for \(n\le m\). The representation is therefore vector-valued for \(m=1\), matrix-valued for \(m=2\), and tensor-valued for \(m\ge 3\).

The paper interprets these objects as capturing local variations in the probability density function of the input. In this formulation, feature learning is not the direct optimization of a discriminative predictor; it is the construction of a hierarchy of differential descriptors of \(p(x)\) whose cross-moments with labels recover expected derivatives of \(G\). This gives a precise sense in which unlabeled generative structure becomes supervised information.

## 2. Two-stage generative-to-discriminative pipeline

The score-function framework is explicitly two-stage. First, unlabeled samples \(\{x_i\}\) are used to estimate the input density \(p(x)\) or a generative latent-variable model for \(x\), from which one computes
\[
\mathcal S_m(x)=(-1)^m\frac{\nabla^{(m)}p(x)}{p(x)}.
\]
These are called general-purpose features because they are learned without labels and can be reused for multiple tasks. Second, labeled data \(\{(x_i,y_i)\}\) are used only through empirical cross-moments
\[
\mathbb E[y\cdot \mathcal S_m(x)],
\]
which equal \(\mathbb E[\nabla^{(m)}G(x)]\) and therefore recover task-specific discriminative information [1412.6514].

A major point of the method is that higher-order score features are naturally matrix- or tensor-valued. For \(m=2\),
\[
\mathbb E[y\cdot \mathcal S_2(x)] = \mathbb E[\nabla^{(2)}G(x)],
\]
and for \(m\ge 3\),
\[
\mathbb E[y\cdot \mathcal S_m(x)] = \mathbb E[\nabla^{(m)}G(x)].
\]
The resulting derivative matrix or tensor is then decomposed spectrally into rank-1 symmetric components,
\[
\mathbb E[\nabla^{(m)}G(x)] = \sum_{j=1}^k u_j^{\otimes m}.
\]
For \(m=2\) this reduces to
\[
\mathbb E[\nabla^{(2)}G(x)] = \sum_{j=1}^k u_j u_j^\top.
\]

The vectors \(u_j\) are interpreted as discriminative directions or components. The paper suggests either using them directly as parameters in a discriminative model or constructing nonlinear features such as
\[
\sigma(u_j^\top x).
\]
A key advantage stressed by the paper is that tensor decomposition allows overcomplete representations, namely \(k>d\), so that
\[
\mathbb E[\nabla^{(m)}G(x)] = \sum_{j=1}^k u_j^{\otimes m}, \qquad k \text{ possibly } > d.
\]
This suggests a broader reading of the phrase “feature learning equation”: not just a feature definition, but an end-to-end analytic chain linking \(p(x)\), \(\mathcal S_m(x)\), cross-moments, derivative tensors, and spectral extraction.

The same paper also extends the construction to parametric models \(p(x;\theta)\). In that setting one analogously recovers derivatives with respect to parameters,
\[
\mathbb E[\nabla_\theta^{(m)}G(x;\theta)],
\]
using parametric score-function features \(\mathcal S_m(x;\theta)\) described as functions of higher-order Fisher score functions \(\nabla_\theta^{(n)}\log p(x;\theta)\) for \(n\le m\). This makes the framework a higher-order generalization of classical Fisher-score ideas.

## 3. Risk, dependence, and function-space formulations

A different line of work defines feature learning through preservation or approximation of task-relevant information rather than through density derivatives. In "A Theory of Feature Learning" the central identity is the feature gap
\[
\Delta R_L(P_{XY},P_{Z|X})
=
R_L(P_{ZY})-R_L(P_{XY})
=
\mathbb E_{(x,z)\sim P_{XZ}}D_L\!\big(P_{Y|x},P_{Y|z}\big),
\]
which exactly quantifies the loss of predictive information due to feature extraction [1504.00083]. In that decision-theoretic language, a feature map \(P_{Z|X}\) is good if replacing \(X\) by \(Z\) causes only a small increase in Bayes-optimal risk. The same paper gives an unsupervised characterization:
\[
\Delta R_L(P_{XY},P_{Z|X}) \le \epsilon\|L\|
\quad\text{for all }P_{XY},Y,L
\]
if and only if there exists a reconstruction kernel \(\hat P_{X|Z}\) such that
\[
\mathbb E_{x\sim P_X}\mathbb E_{x'\sim \hat P_{X|Z}\circ P_{Z|x}}\mathbf 1(x'\neq x)\le \epsilon.
\]
This makes reconstructability the paper’s main unsupervised feature-learning equation.

In "Neural Feature Learning in Function Space" feature learning is formulated geometrically. The canonical dependence kernel is
\[
i_{X;Y}(x,y)=\frac{P_{X,Y}(x,y)-P_X(x)P_Y(y)}{P_X(x)P_Y(y)},
\]
and the low-rank feature learning problem is
\[
\min_{f\in \mathcal F_X^{(k)},\,g\in \mathcal F_Y^{(k)}} \|i_{X;Y}-f\otimes g\|^2.
\]
The trainable objective is the H-score
\[
\mathcal H(f,g)
=
\mathbb E[f^\top(X)g(Y)]
-
\mathbb E[f(X)]^\top\mathbb E[g(Y)]
-
\frac12\operatorname{tr}(A_fA_g),
\]
which equals
\[
\frac12\Big(\|i_{X;Y}\|^2-\|i_{X;Y}-f\otimes g\|^2\Big)
\]
and is maximized when \(f\otimes g\) is the best low-rank approximation of the target dependence component [2309.10140].

A related but distinct criterion appears in "Fourier Preconditioning for Neural Feature Learning," where the exact training objective is
\[
H(f,g)
=
\mathbb E[f(X)^\top g(Y)]
-
\mathbb E[f(X)]^\top\mathbb E[g(Y)]
-
\operatorname{tr}(A_fA_g),
\]
and networks are trained by
\[
\max_{f_\theta,g_\phi} H\!\left(f_\theta(X),g_\phi(Y)\right).
\]
The paper presents this objective as a computable proxy for captured Hilbert-Schmidt dependence energy, and under Fourier preconditioning as
\[
\max_{\theta,\phi}H\!\left(f_\theta(FX),g_\phi(Y)\right)
\]
when spectral concentration is favorable [2607.02199].

Taken together, these formulations suggest that the phrase “feature learning equation” can denote at least three mathematically distinct objects: an exact discriminative identity, a Bayes-risk preservation law, or a dependence-approximation objective.

## 4. Finite-width kernel adaptation and tangent-feature transport

In deep-network theory, feature learning is often expressed as an equation for how kernels or tangent features become label dependent at finite width. In "Critical feature learning in deep neural networks" the exact prior of a deep finite-width network is written as a superposition of Gaussian processes,
\[
p(Y|X)=\int \mathcal D C\,\mathcal N(Y\vert 0,C^{(L)}+\kappa I)\,p(C),
\]
and posterior kernel adaptation is governed by a coupled forward–backward system
\[
C^{(l+1)} = F(C^{(l)},\tilde C^{(l+1)}),
\qquad
\tilde C^{(l)} = G(C^{(l)},C^{(l+1)})\,\tilde C^{(l+1)}.
\]
More explicitly,
\[
C_{\alpha\beta}^{(l+1)}
=
g_{l+1}\,\big\langle\phi_\alpha^{(l)}\phi_\beta^{(l)}\big\rangle_{\mathcal P^{(l)}}+g_b,
\]
with a label-dependent tilted measure \(\mathcal P^{(l)}\), and
\[
\tilde C^{(L)}
=
\frac12(C^{(L)}+\kappa I)^{-1}YY^{\mathsf T}(C^{(L)}+\kappa I)^{-1}
-
\frac12(C^{(L)}+\kappa I)^{-1}.
\]
At leading order,
\[
C_{\alpha\beta}^{(l+1)}
=
C_{\alpha\beta}^{(l+1),\mathrm{NNGP}}
+
\frac{g_{l+1}^2}{N}\sum_{\gamma,\delta}V_{\alpha\beta,\gamma\delta}^{(l)}\tilde C_{\gamma\delta}^{(l+1)}
+
\mathcal O(N^{-2}),
\]
so finite-width kernel fluctuations convert backpropagated label error into forward kernel adaptation [2405.10761].

A complementary tangent-space formulation appears in "Feature Learning and Signal Propagation in Deep Neural Networks." There the layerwise tangent kernel factorizes as
\[
\bar{\mathbf K}_l=\overrightarrow{\mathbf K}_l\circ \overleftarrow{\mathbf K}_l,
\]
with
\[
\overrightarrow{K}_l(x,x')=\frac1N\phi(z_{l-1}(x))\cdot\phi(z_{l-1}(x')),
\qquad
\overleftarrow{K}_l(x,x')=\frac1N
\frac{\partial f_{l:L}}{\partial z}(z_l(x))
\cdot
\frac{\partial f_{l:L}}{\partial z}(z_l(x')).
\]
Layerwise feature learning is measured by
\[
A_l
=
\frac{\tilde Y^\top \mathbf\Psi_l^\top \mathbf\Psi_l\tilde Y}
{\|\mathbf C\mathbf\Psi_l^\top \mathbf\Psi_l\mathbf C\|_F\|\tilde Y\|^2},
\]
and the Equilibrium Hypothesis states that layers with highest alignments satisfy
\[
IL^f_{l,N}=\Theta(IL^b_{l,N}).
\]
For deep ReLU feed-forward networks the paper derives the scaling law
\[
l=\Theta(L^{3/5})
\]
for the layer at which alignment peaks [2110.11749].

A Bayesian one-hidden-layer proportional-limit analysis adds collective and microscopic observables. In "Microscopic and collective signatures of feature learning in neural networks" the posterior feature covariance is
\[
\langle \sigma(h_i^\mu)\sigma(h_i^\nu)\rangle
=
K_{\mu\nu}-\Delta_1^{\mu\nu}+\Delta_2^{\mu\nu},
\]
while the collective class-manifold distance satisfies
\[
\langle D^2\rangle
=
\frac{4}{\alpha P}y^T K y
-
\frac{4}{\alpha P}y^T\Delta_1 y
+
\frac{4}{\alpha P}y^T\Delta_2 y
\]
and approximately
\[
\langle D^2\rangle
\simeq
\frac{4}{\alpha}\frac{1}{P}\sum_{\mu,\nu=1}^P y_\mu K_{\mu\nu}y_\nu
+
\frac{\lambda_1}{\bar Q}(\bar f_+-\bar f_-)^2.
\]
At the microscopic level,
\[
\langle w_{1h}w_{1k}\rangle
=
\frac{\delta_{hk}}{\lambda_0}
+
\frac{(\Lambda_1)_{hk}}{\lambda_0}
+
\frac{(\Lambda_2)_{hk}}{\lambda_0},
\]
so first-layer parameters develop finite data-dependent correlations even though the posterior predictive distribution is that of Gaussian process regression with a trivially rescaled prior [2508.20989].

## 5. Spectral spike and mean-field theories of early feature learning

A large early gradient step in a two-layer network can be analyzed as a low-rank deformation of a random-feature matrix. In "A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks" the post-update feature matrix satisfies
\[
F = F_\ell + \Xi,
\qquad
F_\ell := F_0 + \sum_{k=1}^{\ell} c_1^k c_k \eta^k (\tilde X \beta)^{\circ k}(a^{\circ k})^\top,
\]
with
\[
\|\Xi\|_{\mathrm{op}}=o(\sqrt n),
\]
whenever
\[
\eta\asymp n^\alpha,
\qquad
\frac{\ell-1}{2\ell}<\alpha<\frac{\ell}{2\ell+2}.
\]
The top-\(\ell\) left singular subspace then converges to
\[
\operatorname{span}\{(\tilde X\beta)^{\circ k}\}_{k\in[\ell]},
\]
so one gradient step inserts polynomial spikes aligned with degree-\(k\) features [2310.07891].

"Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent" extends this to two gradient steps. The second-step update yields
\[
\mathbf W_2
=
\mathbf W_0
+
c_1\mathbf W_0(\eta_1\hat{\mathbf w}_1+\eta_2\hat{\mathbf w}_{2;0})^\top
+
\sum_{k=1}^{\Lambda(\alpha_1,\alpha_2)}
(k+1)c_{k+1}c_1^k\eta_2\eta_1^k
\mathbf W_0^{\odot(k+1)}\hat{\mathbf w}_{2;k}^\top
+
\mathbf E_2,
\]
where
\[
\Lambda(\alpha_1,\alpha_2)
=
\min\left\{L-1,\left\lfloor \frac{\alpha_2}{1/2-\alpha_1}\right\rfloor\right\}.
\]
The paper shows that the updated weights behave as a spiked random matrix with multiple outliers, each corresponding to a learned direction, and distinguishes reused-batch from fresh-batch regimes through the asymptotic alignment formulas for \(\hat{\mathbf w}_{2;q}\) [2605.17767].

A function-space reformulation appears in "How does feature learning reshape the function space?" There the learned first-layer weight distribution is approximated by a target-dependent spiked Gaussian covariance
\[
T=\mathbb E[zz^\top]=A I_d + B\,w^\star (w^\star)^\top,
\]
and the induced kernel becomes
\[
k_1(x,x')
=
\mathbb E_{w\sim \mathcal N(0,T)}
\big[\sigma(\langle w,x\rangle)\sigma(\langle w,x'\rangle)\big]
=
k_0(T^{1/2}x,T^{1/2}x').
\]
The leading perturbation depends explicitly on \(\langle x,w^\star\rangle\) and \(\langle x',w^\star\rangle\), so feature learning is described as a data-adaptive deformation of the kernel rather than a scalar rescaling [2605.17718].

A statistical-physics alternative is given by "A simple mean field model of feature learning." There the central fixed-point system is
\[
-\ln p_{\mathrm{MF}}(w,a\mid \{m_A\},\mathcal D)
=
\frac{a^2}{2\sigma_a^2}
+
\frac{d}{2\sigma_w^2}\|w\|^2
+
\frac{a^2}{2\kappa^2 N^{2\gamma}}\Sigma(w)
-
\frac{a}{\kappa^2 N^\gamma}\left(J_{\mathcal Y}(w)-\sum_A m_AJ_A(w)\right),
\]
\[
m_A = N^{1-\gamma}\langle aJ_A(w)\rangle_{p_{\mathrm{MF}}}.
\]
In the extended MF-ARD theory this is coupled to coordinate-wise precisions
\[
\rho_j
=
\frac{\alpha_0+\tfrac N2}{\frac{\alpha_0}{d}+\tfrac N2\langle w_j^2\rangle_{p_{\mathrm{ARD}}}},
\]
which implement self-reinforcing input feature selection. The phase transition criterion is expressed as \(\omega_0>1\), and the theory distinguishes the infinite-width kernel regime from a finite-width symmetry-breaking feature-learning regime [2510.15174].

## 6. Specialized formulations, applications, and controversies

In some domains the phrase denotes an application-specific architectural equation rather than a generic representation-learning principle. In "Improve the Fitting Accuracy of Deep Learning for the Nonlinear Schrödinger Equation Using Linear Feature Decoupling Method," feature learning is formulated through the operator decomposition
\[
\frac{\partial S(z,t)}{\partial z} = \left(\hat D+\hat N[S(z,t)]\right)S(z,t),
\]
with linear feature decoupling based on
\[
\frac{\partial s}{\partial z}+\frac{\alpha s}{2}+\frac{i\beta}{2}\frac{\partial^2 s}{\partial t^2}=0,
\]
and evaluation through the PDE residual
\[
\text{NLSE loss}
=
\frac{\partial s}{\partial z}
+
\frac{\alpha s}{2}
+
\frac{i\beta}{2}\frac{\partial^2 s}{\partial t^2}
-
\frac{8}{9}i\gamma |s|^2 s.
\]
Here the feature-learning equation is effectively the physics-structured decomposition that moves known linear propagation outside the neural component [2411.04511].

In "Tensor Network Based Feature Learning Model" the term refers to a learnable mixture of tensor-product features,
\[
f(x)=\left[\sum_{p=1}^P \lambda_p\,\phi_{\theta_p}(x)\right]^\top w,
\]
subject to the CPD constraint
\[
w=\sum_{r=1}^{R} w_r^{(D)}\otimes\cdots\otimes w_r^{(1)},
\]
and optimized by alternating least squares through
\[
\min_{\lambda,w}
\frac12\left\|y-\sum_{p=1}^P\lambda_p\Phi_p w\right\|_2^2
+
\frac{\alpha}{2}\|w\|_2^2
+
\beta \operatorname{Reg}(\lambda).
\]
In this setting, feature learning means learning feature-mixture coefficients \(\lambda_p\) jointly with the model tensor [2512.02547].

Other specialized theories attach the phrase to message-passing or dynamical systems. In the RBM analysis of "Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses," the central inference equation is
\[
m_{i\rightarrow a}
=
\tanh\left(\sum_{b\in\partial i\backslash a}u_{b\rightarrow i}\right),
\qquad
u_{b\rightarrow i}
=
\tanh^{-1}\left(\tanh(\beta G_{b\rightarrow i})\tanh(\beta\sigma_i^{b}/\sqrt N)\right),
\]
with
\[
G_{b\rightarrow i}
=
\frac{1}{\sqrt N}\sum_{j\in\partial b\backslash i}\sigma_j^b\,m_{j\rightarrow b},
\]
so unsupervised feature learning is framed as marginal inference of a hidden binary feature vector [1612.01717]. In "Free Dynamics of Feature Learning Processes," by contrast, feature learning in linear regression is an autonomous dynamical system for spectral strengths \(c_a\) and alignments \(f_a\),
\[
\dot c_a
=
\frac{2\Lambda^2 c_a}{(1+\Lambda c_a)^2}
\left(f_a^2+\frac{\mathcal B}{N}\right),
\qquad
\dot f_a
=
\Lambda^2\left[
\frac{f_a}{1+\Lambda c_a}(1-r-\sigma^2)
+
\sum_{b\ne a}
\frac{c_a+c_b}{c_a-c_b}
\frac{f_af_b^2}{(1+\Lambda c_a)(1+\Lambda c_b)}
\right],
\]
linking feature learning directly to generalization through evolving spectral alignment [2210.10702].

The phrase also supports geometric reinterpretations. In "Half-Space Feature Learning in Neural Networks," a DLGN path feature is
\[
f_\pi(x)
=
\prod_{\ell=1}^{L-1}\mathbf 1(\eta_{\ell,i_\ell}(x)\ge 0),
\qquad
A_\pi
=
\bigcap_{\ell=1}^{L-1}\{x:\eta_{\ell,i_\ell}(x)\ge 0\},
\]
so a learned feature is an indicator of an intersection of half-spaces [2404.04312]. A different controversy is raised in "Feature learning is decoupled from generalization in high capacity neural networks," which defines feature quality by the feature learning gap
\[
\Delta_{\operatorname{NTK}}(m)
=
\mathcal E_{\operatorname{gen}}(\mu_{\operatorname{NTK};m})
-
\mathcal E_{\operatorname{gen}}(f_\theta;m),
\]
and argues that current FL definitions characterize FL by measuring FL strength \(S(f_\theta)\), while FL strength is decoupled from feature quality [2507.19680]. This suggests that even when a paper supplies a mathematically precise feature-learning equation, that equation may characterize representation change, task-relevant information, or generalization gain only under additional assumptions.

Overall, the literature supports no single universal “Feature Learning Equation.” What recurs instead is a shared structural role: an equation specifies how latent geometry, density structure, kernel structure, or task dependence is converted into a learned representation. In score-function methods this role is played by an exact cross-moment identity; in decision-theoretic and function-space theories by a risk or dependence approximation law; in finite-width deep-network theory by a forward–backward or spiked-spectral update; and in specialized settings by architecture-specific or inference-specific dynamical equations.

Source: https://www.emergentmind.com/topics/feature-learning-equation