Feature Learning Equation Overview

• Feature Learning Equation is a family of formal devices that define how latent generative structures, via density derivatives and cross-moments, yield discriminative signals.
• Score-function formulations translate unlabeled generative data into actionable derivative features, bridging generative and discriminative learning in a two-stage pipeline.
• Alternative views use risk gap formulations, spectral decompositions, and dynamical systems to capture latent geometry and guide kernel adaptations for improved representation.

Several papers explicitly note that they do not introduce a single formula literally called the “feature learning equation,” while others attach the term to a paper-specific identity. A prominent instance is the higher-order score-function relation

$$\mathcal S_m(x)=(-1)^m\frac{\nabla^{(m)}p(x)}{p(x)}, \qquad \mathbb E[y\cdot \mathcal S_m(x)]=\mathbb E[\nabla_x^{(m)}G(x)], \qquad G(x)=\mathbb E[y\mid x],$$

which turns unlabeled structure in $p(x)$ into discriminative derivative information (Janzamin et al., 2014). This suggests that “Feature Learning Equation” is best understood as a family of formal devices—identity, objective, fixed-point system, or dynamical law—used to specify what feature learning is, what information it recovers, and how that information changes during training (Rooyen et al., 2015).

1. Score-function features as a discriminative identity

In "Score Function Features for Discriminative Learning" the central learned quantity is the $m$-th order score-function feature

$$\mathcal{S}_m(x) = (-1)^m \frac{\nabla^{(m)} p(x)}{p(x)},$$

for $x\sim p(\cdot)$, together with the conditional mean

$\mathbb E[y\mid x]:=G(x).$

The paper’s main discriminative quantity is

$$\mathbb E\big[\nabla_x^{(m)} G(x)\big], \qquad m\ge 1,$$

and the bridge from unlabeled to labeled learning is the exact identity

$$\mathbb E\big[y\cdot \mathcal S_m(x)\big]=\mathbb E\big[\nabla^{(m)}G(x)\big].$$

This is the paper’s main “feature learning equation” (Janzamin et al., 2014).

The definition recovers the classical score function at first order: $$\mathcal S_1(x)=-\frac{\nabla p(x)}{p(x)}=-\nabla\log p(x),$$ while second order yields

$$\mathcal S_2(x)=\frac{\nabla^{(2)}p(x)}{p(x)} =\nabla^{(2)}\log p(x)+\nabla\log p(x)\otimes \nabla\log p(x) =\nabla^{(2)}\log p(x)+\mathcal S_1(x)\otimes \mathcal S_1(x).$$

More generally, $p(x)$0 is a function of the higher-order log-density derivatives $p(x)$1 for $p(x)$2. The representation is therefore vector-valued for $p(x)$3, matrix-valued for $p(x)$4, and tensor-valued for $p(x)$5.

The paper interprets these objects as capturing local variations in the probability density function of the input. In this formulation, feature learning is not the direct optimization of a discriminative predictor; it is the construction of a hierarchy of differential descriptors of $p(x)$6 whose cross-moments with labels recover expected derivatives of $p(x)$7. This gives a precise sense in which unlabeled generative structure becomes supervised information.

2. Two-stage generative-to-discriminative pipeline

The score-function framework is explicitly two-stage. First, unlabeled samples $p(x)$8 are used to estimate the input density $p(x)$9 or a generative latent-variable model for $m$0, from which one computes

$m$1

These are called general-purpose features because they are learned without labels and can be reused for multiple tasks. Second, labeled data $m$2 are used only through empirical cross-moments

$m$3

which equal $m$4 and therefore recover task-specific discriminative information (Janzamin et al., 2014).

A major point of the method is that higher-order score features are naturally matrix- or tensor-valued. For $m$5,

$m$6

and for $m$7,

$m$8

The resulting derivative matrix or tensor is then decomposed spectrally into rank-1 symmetric components,

$m$9

For $$\mathcal{S}_m(x) = (-1)^m \frac{\nabla^{(m)} p(x)}{p(x)},$$0 this reduces to

$$\mathcal{S}_m(x) = (-1)^m \frac{\nabla^{(m)} p(x)}{p(x)},$$1

The vectors $$\mathcal{S}_m(x) = (-1)^m \frac{\nabla^{(m)} p(x)}{p(x)},$$2 are interpreted as discriminative directions or components. The paper suggests either using them directly as parameters in a discriminative model or constructing nonlinear features such as

$$\mathcal{S}_m(x) = (-1)^m \frac{\nabla^{(m)} p(x)}{p(x)},$$3

A key advantage stressed by the paper is that tensor decomposition allows overcomplete representations, namely $$\mathcal{S}_m(x) = (-1)^m \frac{\nabla^{(m)} p(x)}{p(x)},$$4, so that

$$\mathcal{S}_m(x) = (-1)^m \frac{\nabla^{(m)} p(x)}{p(x)},$$5

This suggests a broader reading of the phrase “feature learning equation”: not just a feature definition, but an end-to-end analytic chain linking $$\mathcal{S}_m(x) = (-1)^m \frac{\nabla^{(m)} p(x)}{p(x)},$$6, $$\mathcal{S}_m(x) = (-1)^m \frac{\nabla^{(m)} p(x)}{p(x)},$$7, cross-moments, derivative tensors, and spectral extraction.

The same paper also extends the construction to parametric models $$\mathcal{S}_m(x) = (-1)^m \frac{\nabla^{(m)} p(x)}{p(x)},$$8. In that setting one analogously recovers derivatives with respect to parameters,

$$\mathcal{S}_m(x) = (-1)^m \frac{\nabla^{(m)} p(x)}{p(x)},$$9

using parametric score-function features $x\sim p(\cdot)$0 described as functions of higher-order Fisher score functions $x\sim p(\cdot)$1 for $x\sim p(\cdot)$2. This makes the framework a higher-order generalization of classical Fisher-score ideas.

3. Risk, dependence, and function-space formulations

A different line of work defines feature learning through preservation or approximation of task-relevant information rather than through density derivatives. In "A Theory of Feature Learning" the central identity is the feature gap

$x\sim p(\cdot)$3

which exactly quantifies the loss of predictive information due to feature extraction (Rooyen et al., 2015). In that decision-theoretic language, a feature map $x\sim p(\cdot)$4 is good if replacing $x\sim p(\cdot)$5 by $x\sim p(\cdot)$6 causes only a small increase in Bayes-optimal risk. The same paper gives an unsupervised characterization: $x\sim p(\cdot)$7 if and only if there exists a reconstruction kernel $x\sim p(\cdot)$8 such that

$x\sim p(\cdot)$9

This makes reconstructability the paper’s main unsupervised feature-learning equation.

In "Neural Feature Learning in Function Space" feature learning is formulated geometrically. The canonical dependence kernel is

$\mathbb E[y\mid x]:=G(x).$0

and the low-rank feature learning problem is

$\mathbb E[y\mid x]:=G(x).$1

The trainable objective is the H-score

$\mathbb E[y\mid x]:=G(x).$2

which equals

$\mathbb E[y\mid x]:=G(x).$3

and is maximized when $\mathbb E[y\mid x]:=G(x).$4 is the best low-rank approximation of the target dependence component (Xu et al., 2023).

A related but distinct criterion appears in "Fourier Preconditioning for Neural Feature Learning," where the exact training objective is

$\mathbb E[y\mid x]:=G(x).$5

and networks are trained by

$\mathbb E[y\mid x]:=G(x).$6

The paper presents this objective as a computable proxy for captured Hilbert-Schmidt dependence energy, and under Fourier preconditioning as

$\mathbb E[y\mid x]:=G(x).$7

when spectral concentration is favorable (Pitzer et al., 2 Jul 2026).

Taken together, these formulations suggest that the phrase “feature learning equation” can denote at least three mathematically distinct objects: an exact discriminative identity, a Bayes-risk preservation law, or a dependence-approximation objective.

4. Finite-width kernel adaptation and tangent-feature transport

In deep-network theory, feature learning is often expressed as an equation for how kernels or tangent features become label dependent at finite width. In "Critical feature learning in deep neural networks" the exact prior of a deep finite-width network is written as a superposition of Gaussian processes,

$\mathbb E[y\mid x]:=G(x).$8

and posterior kernel adaptation is governed by a coupled forward–backward system

$\mathbb E[y\mid x]:=G(x).$9

More explicitly,

$$\mathbb E\big[\nabla_x^{(m)} G(x)\big], \qquad m\ge 1,$$0

with a label-dependent tilted measure $$\mathbb E\big[\nabla_x^{(m)} G(x)\big], \qquad m\ge 1,$$1, and

$$\mathbb E\big[\nabla_x^{(m)} G(x)\big], \qquad m\ge 1,$$2

At leading order,

$$\mathbb E\big[\nabla_x^{(m)} G(x)\big], \qquad m\ge 1,$$3

so finite-width kernel fluctuations convert backpropagated label error into forward kernel adaptation (Fischer et al., 2024).

A complementary tangent-space formulation appears in "Feature Learning and Signal Propagation in Deep Neural Networks." There the layerwise tangent kernel factorizes as

$$\mathbb E\big[\nabla_x^{(m)} G(x)\big], \qquad m\ge 1,$$4

with

$$\mathbb E\big[\nabla_x^{(m)} G(x)\big], \qquad m\ge 1,$$5

Layerwise feature learning is measured by

$$\mathbb E\big[\nabla_x^{(m)} G(x)\big], \qquad m\ge 1,$$6

and the Equilibrium Hypothesis states that layers with highest alignments satisfy

$$\mathbb E\big[\nabla_x^{(m)} G(x)\big], \qquad m\ge 1,$$7

For deep ReLU feed-forward networks the paper derives the scaling law

$$\mathbb E\big[\nabla_x^{(m)} G(x)\big], \qquad m\ge 1,$$8

for the layer at which alignment peaks (Lou et al., 2021).

A Bayesian one-hidden-layer proportional-limit analysis adds collective and microscopic observables. In "Microscopic and collective signatures of feature learning in neural networks" the posterior feature covariance is

$$\mathbb E\big[\nabla_x^{(m)} G(x)\big], \qquad m\ge 1,$$9

while the collective class-manifold distance satisfies

$$\mathbb E\big[y\cdot \mathcal S_m(x)\big]=\mathbb E\big[\nabla^{(m)}G(x)\big].$$0

and approximately

$$\mathbb E\big[y\cdot \mathcal S_m(x)\big]=\mathbb E\big[\nabla^{(m)}G(x)\big].$$1

At the microscopic level,

$$\mathbb E\big[y\cdot \mathcal S_m(x)\big]=\mathbb E\big[\nabla^{(m)}G(x)\big].$$2

so first-layer parameters develop finite data-dependent correlations even though the posterior predictive distribution is that of Gaussian process regression with a trivially rescaled prior (Corti et al., 28 Aug 2025).

5. Spectral spike and mean-field theories of early feature learning

A large early gradient step in a two-layer network can be analyzed as a low-rank deformation of a random-feature matrix. In "A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks" the post-update feature matrix satisfies

$$\mathbb E\big[y\cdot \mathcal S_m(x)\big]=\mathbb E\big[\nabla^{(m)}G(x)\big].$$3

with

$$\mathbb E\big[y\cdot \mathcal S_m(x)\big]=\mathbb E\big[\nabla^{(m)}G(x)\big].$$4

whenever

$$\mathbb E\big[y\cdot \mathcal S_m(x)\big]=\mathbb E\big[\nabla^{(m)}G(x)\big].$$5

The top-$$\mathbb E\big[y\cdot \mathcal S_m(x)\big]=\mathbb E\big[\nabla^{(m)}G(x)\big].$$6 left singular subspace then converges to

$$\mathbb E\big[y\cdot \mathcal S_m(x)\big]=\mathbb E\big[\nabla^{(m)}G(x)\big].$$7

so one gradient step inserts polynomial spikes aligned with degree-$$\mathbb E\big[y\cdot \mathcal S_m(x)\big]=\mathbb E\big[\nabla^{(m)}G(x)\big].$$8 features (Moniri et al., 2023).

"Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent" extends this to two gradient steps. The second-step update yields

$$\mathbb E\big[y\cdot \mathcal S_m(x)\big]=\mathbb E\big[\nabla^{(m)}G(x)\big].$$9

where

$$\mathcal S_1(x)=-\frac{\nabla p(x)}{p(x)}=-\nabla\log p(x),$$0

The paper shows that the updated weights behave as a spiked random matrix with multiple outliers, each corresponding to a learned direction, and distinguishes reused-batch from fresh-batch regimes through the asymptotic alignment formulas for $$\mathcal S_1(x)=-\frac{\nabla p(x)}{p(x)}=-\nabla\log p(x),$$1 (Moniri et al., 18 May 2026).

A function-space reformulation appears in "How does feature learning reshape the function space?" There the learned first-layer weight distribution is approximated by a target-dependent spiked Gaussian covariance

$$\mathcal S_1(x)=-\frac{\nabla p(x)}{p(x)}=-\nabla\log p(x),$$2

and the induced kernel becomes

$$\mathcal S_1(x)=-\frac{\nabla p(x)}{p(x)}=-\nabla\log p(x),$$3

The leading perturbation depends explicitly on $$\mathcal S_1(x)=-\frac{\nabla p(x)}{p(x)}=-\nabla\log p(x),$$4 and $$\mathcal S_1(x)=-\frac{\nabla p(x)}{p(x)}=-\nabla\log p(x),$$5, so feature learning is described as a data-adaptive deformation of the kernel rather than a scalar rescaling (Lobo et al., 18 May 2026).

A statistical-physics alternative is given by "A simple mean field model of feature learning." There the central fixed-point system is

$$\mathcal S_1(x)=-\frac{\nabla p(x)}{p(x)}=-\nabla\log p(x),$$6

$$\mathcal S_1(x)=-\frac{\nabla p(x)}{p(x)}=-\nabla\log p(x),$$7

In the extended MF-ARD theory this is coupled to coordinate-wise precisions

$$\mathcal S_1(x)=-\frac{\nabla p(x)}{p(x)}=-\nabla\log p(x),$$8

which implement self-reinforcing input feature selection. The phase transition criterion is expressed as $$\mathcal S_1(x)=-\frac{\nabla p(x)}{p(x)}=-\nabla\log p(x),$$9, and the theory distinguishes the infinite-width kernel regime from a finite-width symmetry-breaking feature-learning regime (Göring et al., 16 Oct 2025).

6. Specialized formulations, applications, and controversies

In some domains the phrase denotes an application-specific architectural equation rather than a generic representation-learning principle. In "Improve the Fitting Accuracy of Deep Learning for the Nonlinear Schrödinger Equation Using Linear Feature Decoupling Method," feature learning is formulated through the operator decomposition

$$\mathcal S_2(x)=\frac{\nabla^{(2)}p(x)}{p(x)} =\nabla^{(2)}\log p(x)+\nabla\log p(x)\otimes \nabla\log p(x) =\nabla^{(2)}\log p(x)+\mathcal S_1(x)\otimes \mathcal S_1(x).$$0

with linear feature decoupling based on

$$\mathcal S_2(x)=\frac{\nabla^{(2)}p(x)}{p(x)} =\nabla^{(2)}\log p(x)+\nabla\log p(x)\otimes \nabla\log p(x) =\nabla^{(2)}\log p(x)+\mathcal S_1(x)\otimes \mathcal S_1(x).$$1

and evaluation through the PDE residual

$$\mathcal S_2(x)=\frac{\nabla^{(2)}p(x)}{p(x)} =\nabla^{(2)}\log p(x)+\nabla\log p(x)\otimes \nabla\log p(x) =\nabla^{(2)}\log p(x)+\mathcal S_1(x)\otimes \mathcal S_1(x).$$2

Here the feature-learning equation is effectively the physics-structured decomposition that moves known linear propagation outside the neural component (Zhang et al., 2024).

In "Tensor Network Based Feature Learning Model" the term refers to a learnable mixture of tensor-product features,

$$\mathcal S_2(x)=\frac{\nabla^{(2)}p(x)}{p(x)} =\nabla^{(2)}\log p(x)+\nabla\log p(x)\otimes \nabla\log p(x) =\nabla^{(2)}\log p(x)+\mathcal S_1(x)\otimes \mathcal S_1(x).$$3

subject to the CPD constraint

$$\mathcal S_2(x)=\frac{\nabla^{(2)}p(x)}{p(x)} =\nabla^{(2)}\log p(x)+\nabla\log p(x)\otimes \nabla\log p(x) =\nabla^{(2)}\log p(x)+\mathcal S_1(x)\otimes \mathcal S_1(x).$$4

and optimized by alternating least squares through

$$\mathcal S_2(x)=\frac{\nabla^{(2)}p(x)}{p(x)} =\nabla^{(2)}\log p(x)+\nabla\log p(x)\otimes \nabla\log p(x) =\nabla^{(2)}\log p(x)+\mathcal S_1(x)\otimes \mathcal S_1(x).$$5

In this setting, feature learning means learning feature-mixture coefficients $$\mathcal S_2(x)=\frac{\nabla^{(2)}p(x)}{p(x)} =\nabla^{(2)}\log p(x)+\nabla\log p(x)\otimes \nabla\log p(x) =\nabla^{(2)}\log p(x)+\mathcal S_1(x)\otimes \mathcal S_1(x).$$6 jointly with the model tensor (Saiapin et al., 2 Dec 2025).

Other specialized theories attach the phrase to message-passing or dynamical systems. In the RBM analysis of "Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses," the central inference equation is

$$\mathcal S_2(x)=\frac{\nabla^{(2)}p(x)}{p(x)} =\nabla^{(2)}\log p(x)+\nabla\log p(x)\otimes \nabla\log p(x) =\nabla^{(2)}\log p(x)+\mathcal S_1(x)\otimes \mathcal S_1(x).$$7

with

$$\mathcal S_2(x)=\frac{\nabla^{(2)}p(x)}{p(x)} =\nabla^{(2)}\log p(x)+\nabla\log p(x)\otimes \nabla\log p(x) =\nabla^{(2)}\log p(x)+\mathcal S_1(x)\otimes \mathcal S_1(x).$$8

so unsupervised feature learning is framed as marginal inference of a hidden binary feature vector (Huang, 2016). In "Free Dynamics of Feature Learning Processes," by contrast, feature learning in linear regression is an autonomous dynamical system for spectral strengths $$\mathcal S_2(x)=\frac{\nabla^{(2)}p(x)}{p(x)} =\nabla^{(2)}\log p(x)+\nabla\log p(x)\otimes \nabla\log p(x) =\nabla^{(2)}\log p(x)+\mathcal S_1(x)\otimes \mathcal S_1(x).$$9 and alignments $p(x)$00,

$p(x)$01

linking feature learning directly to generalization through evolving spectral alignment (Furtlehner, 2022).

The phrase also supports geometric reinterpretations. In "Half-Space Feature Learning in Neural Networks," a DLGN path feature is

$p(x)$02

so a learned feature is an indicator of an intersection of half-spaces (Yadav et al., 2024). A different controversy is raised in "Feature learning is decoupled from generalization in high capacity neural networks," which defines feature quality by the feature learning gap

$p(x)$03

and argues that current FL definitions characterize FL by measuring FL strength $p(x)$04, while FL strength is decoupled from feature quality (Göring et al., 25 Jul 2025). This suggests that even when a paper supplies a mathematically precise feature-learning equation, that equation may characterize representation change, task-relevant information, or generalization gain only under additional assumptions.

Overall, the literature supports no single universal “Feature Learning Equation.” What recurs instead is a shared structural role: an equation specifies how latent geometry, density structure, kernel structure, or task dependence is converted into a learned representation. In score-function methods this role is played by an exact cross-moment identity; in decision-theoretic and function-space theories by a risk or dependence approximation law; in finite-width deep-network theory by a forward–backward or spiked-spectral update; and in specialized settings by architecture-specific or inference-specific dynamical equations.