Local EGOP Learning in High-Dimensional Regression

• Local EGOP learning is a recursively adaptive framework that utilizes the EGOP matrix to guide kernel regression by accurately estimating local function derivatives.
• It leverages recursive metric updates and anisotropic shrinkage to adapt kernel smoothing for functions with localized, low-dimensional variations.
• Empirical benchmarks demonstrate significant MSE reductions and improved subspace recovery compared to standard neural networks and kernel methods.

Local EGOP learning is a recursively adaptive algorithmic framework for high-dimensional nonparametric regression, targeting functions with localized, low-dimensional variation. Centered on the Expected Gradient Outer Product (EGOP) quadratic form, Local EGOP learning steers kernel regression by dynamically estimating and exploiting the metric structure of function derivatives in the vicinity of each query point. This approach provably achieves intrinsic-dimensional learning rates in settings where the target function varies primarily on a lower-dimensional manifold embedded in a high-dimensional ambient space, and its effectiveness exceeds that of standard neural networks and generic kernel methods in continuous index models and noisy manifold data (Kokot et al., 11 Jan 2026).

1. EGOP Quadratic Form: Definition and Interpretation

Local EGOP learning is grounded in the EGOP matrix, defined for a smooth regression function $f:\mathbb{R}^D\to\mathbb{R}$ as

$$\mathcal{L}(\mu) = \int \nabla f(x)\nabla f(x)^T d\mu(x),$$

where $\mu$ is a probability measure localizing the region of interest, commonly instantiated as a Gaussian $N(x^*,\Sigma)$ centered at query point $x^*$ (Kokot et al., 11 Jan 2026). The EGOP encapsulates the local principal directions and magnitudes of variation in $f$, serving as both a metric and (when inverted) a covariance estimate for kernel adaptation. The associated Dirichlet form

$W(\mu) = \mathrm{tr}(\mathcal{L}(\mu)\Sigma)$

quantifies the contribution of convolution bias when estimating $f(x^*)$ via local averaging, controlling bias in nonparametric regression.

2. Recursive Kernel Adaptation via EGOP

Local EGOP learning operates by iteratively adjusting a Mahalanobis metric $M \succeq 0$, interpreted as an inverse covariance for Gaussian kernel smoothing. The continuous kernel smoother at $x^*$ is defined as

$$P_{M}(f) = \frac{1}{C_M}\int k(\|M^{1/2}(y-x^*)\|/\sqrt{2}) f(y) dP(y),$$

where $k(\cdot)$ is a second-order kernel and $C_M$ normalizes the integral. Bias is bounded by $O(W(\mu))$ for $\mu = N(x^*,M^{-1})$; variance is $O(1/(n\sqrt{\det M}))$ (Kokot et al., 11 Jan 2026). The fundamental update rule sets $\Sigma_t = \frac{t}{D}\mathcal{L}(\mu_t)^{-1}$, with $\mu_t = N(x^*,\Sigma_t)$ and $t \to 0$, optimizing the trade-off between bias and variance. Empirical estimation replaces the true EGOP with observed average gradient outer products (AGOPs), steering the local metric adaptively.

3. Local EGOP Learning Algorithm

The algorithm proceeds for $T$ iterations as follows:

1. At iteration $i$, compute Mahalanobis weights $w_j \propto \exp(-(X_j-x^*)^T M_i (X_j-x^*))$, normalized to sum to one.
2. Subsample $m$ points $S$ according to the weight distribution.
3. For each $j\in S$, perform leave-one-out local linear regression at $X_j$ using $M_i$ to obtain $\widehat{\nabla}f_j$ (gradient estimate) and $\widehat{f}_j$ (interpolated value).
4. Aggregate to form $$L_i = \sum_{j\in S} w_j\widehat{\nabla}f_j \widehat{\nabla}f_j^T$$ and empirical MSE $\sum_{j\in S}w_j(Y_j-\widehat{f}_j)^2$.
5. The metric update applies trace normalization and momentum:

$$M_{i+1} \leftarrow \frac{\beta L_i + (1-\beta)L_{i-1}}{t_{i+1} \mathrm{tr}\big(\beta L_i + (1-\beta)L_{i-1}\big)},$$

where $t_{i+1}$ is the kernel bandwidth at iteration $i+1$ and $\beta\in(0,1)$ (Kokot et al., 11 Jan 2026).

The output is the regression estimate at $x^*$ corresponding to the lowest leave-one-out MSE across all iterations.

4. Theoretical Guarantees Under the Noisy Manifold Hypothesis

Local EGOP learning analysis assumes features $X\in\mathbb{R}^D$ reside in a tubular neighborhood of a compact $d$-dimensional $C^4$ manifold $\mathcal{M}$, with $f(x)=g(\pi(x))$ where $\pi(x)$ is the nearest-point projection onto $\mathcal{M}$ and additive, zero-mean noise with finite fourth moment (Kokot et al., 11 Jan 2026).

Key results include:

• For Gaussian localization $\mu_t=N(x^*,\Sigma_t)$, $$\mathbb{E}[\|\widehat P_{M_t}(f)-f\|_{L^2(\mu_t)}^2]= O(1/(n\sqrt{\det\Sigma_t}) + W(\mu_t))$$.
• If $W(\mu_t)=O(t)$ and $\det\Sigma_t=O(t^q)$, optimal bandwidth choice $t\asymp n^{-2/(2+q)}$ yields MSE rate $O(n^{-1/(1+q/2)})$.
• With invertible Hessian, the recursion shrinks the metric covariance anisotropically:

$$\Sigma_i = \mathrm{diag}(\Theta(t_i), \Theta(\sqrt{t_i}),\dots,\Theta(\sqrt{t_i})),$$

accomplishing near-quadratic improvement over standard kernel rates: $O(n^{-4/(D+5)})$ versus $O(n^{-2/(D+2)})$.

• On noisy manifolds where $\mathrm{rank}(\nabla^2 f)\leq 2d$ and $D\geq 2d$, the rate further improves to $O(n^{-4/(2d+5)})$; the ambient dimension $D$ no longer influences the exponent (Kokot et al., 11 Jan 2026).

Convergence and anisotropic shrinkage are established via matrix recursions, Poincaré inequalities, and geometric analysis of manifold projections.

5. Empirical Benchmarks: Regression and Feature Adaptation

Local EGOP learning has demonstrated pronounced empirical advantages:

• On helical synthetic datasets with orthogonal noise, the algorithm matches the theoretical intrinsic rate $-4/(2d+5)$ independent of ambient $D$, while two-layer NNs deteriorate to $-2/(D+2)$ with increasing $D$. In single-index cases, NNs fail to attain oracle performance regardless of width.
• When compared to transformer architectures (FTTransformer) on noisy spherical data, EGOP-based localizations result in influence maps and embeddings similar to those learned by transformers, indicating robust capture of anisotropic neighborhoods.
• On molecular dynamics simulations ($D\approx50$, $d\approx2$), Local EGOP reduces test MSE twenty-eightfold relative to a standard Gaussian Nadaraya–Watson baseline (Kokot et al., 11 Jan 2026).

Common performance metrics include test MSE scaling with $n$, principal angle metrics for subspace recovery, and visualizations of kernel weight distributions.

6. Computational Considerations and Limitations

The computational complexity per iteration scales as $O(T[(m+2)nD^2 + (m+1)D^3])$, making the approach intensive for large-scale and high-dimensional data. Hyper-parameter selection (subsample size $m$, momentum $\beta$, bandwidth schedule $t_i$) critically affects performance and is not addressed by automated tuning strategies.

Theoretical analyses in current studies presume access to an oracle EGOP; quantification of empirical AGOP estimation error remains open. The framework as developed is tailored to $\ell_2$ regression; extension to classification and other loss functions has not been established (Kokot et al., 11 Jan 2026).

7. Relationship to TrIM and Broader EGOP-based Dimension Reduction

Local EGOP learning extends and localizes the expected gradient outer product methodology pioneered by TrIM (Transformed Iterative Mondrian) forests (Baptista et al., 2024), which estimate a global EGOP to identify a dimension-reduced feature subspace for regression forests. TrIM employs Mondrian forest regression, computes empirical EGOP via local finite differences, and iteratively re-weights inputs by linear transformations derived from EGOP estimates. The approach is supported by finite-sample EGOP consistency guarantees ($n^{-3/(4(d+3))}$) and improved prediction rates under low-dimensional ground truth (Baptista et al., 2024).

A plausible implication is that Local EGOP learning provides a flexible, recursive framework for kernel adaptation in continuous-index settings, while TrIM offers computational efficiency for discrete, axis-aligned subspace discovery. Both methods highlight the critical role of local gradient information and EGOP in mitigating the curse of dimensionality and promoting intrinsic-structure recovery in regression tasks.