Local Gaussian Approximation Techniques

• Local Gaussian Approximation is a technique that fits Gaussian distributions to local data subsets, offering precise nonparametric density, entropy, and mutual information estimation with boundary correction.
• It extends to regression and surrogate modeling by leveraging localized kernels and Bayesian inference to capture nonstationarity and reduce prediction errors.
• The approach scales efficiently to large datasets through adaptive neighborhood selection, facilitating enhanced kernel approximations in Gaussian process models and random feature methods.

A local Gaussian approximation refers to a class of methodologies in statistics and machine learning where the local structure of a complex object—such as a probability density, regression function, empirical process, Gaussian process, kernel function, or quantum wavepacket—is modeled or approximated using a (potentially anisotropic) Gaussian distribution or related analytic surrogates, typically in a data-driven, pointwise-adaptive, or subset-specific manner. Local Gaussian approximations constitute foundational tools for scalable inference, density and entropy estimation, large-scale regression/surrogacy, kernel approximation, and dynamical simulation, combining analytic tractability with local adaptivity.

1. Local Gaussian Approximation in Density and Information Estimation

The local Gaussian approximation framework is prominently used for nonparametric density estimation, particularly in scenarios requiring entropy or mutual information estimation. The standard paradigm replaces classical local-uniform (e.g., k-nearest neighbor) density approximations—which are known to perform poorly near boundaries and under strong dependencies—with a Gaussian locally fit around each sample point.

Given i.i.d. data $\{x_j\}_{j=1}^N\subset\mathbb{R}^d$, a point $x_i$ is associated with a local covariance bandwidth $H$ (e.g., the average distance to the $k$th nearest neighbor), and a Gaussian is fit to the neighboring data by maximizing a weighted (kernel-smoothed) local log-likelihood:

$$\mathcal{L}(x_i; \mu, \Sigma) = \frac{1}{N} \sum_{j=1}^N K_H(x_j - x_i) \log\mathcal{N}_d(x_j; \mu, \Sigma) - \int K_H(t - x_i)\mathcal{N}_d(t; \mu, \Sigma)\,dt$$

where $K_H$ is typically a product-type kernel.

The optimal parameters $(\mu_i, \Sigma_i)$ are used to compute local density estimates:

$$\widehat{f}(x_i) = \mathcal{N}_d(x_i;\mu_i,\Sigma_i)$$

and these are plugged into entropy and mutual information estimators using the standard plug-in formulas.

Key properties established for the Local Gaussian Approximation (LGA)-based mutual information estimator include:

• Asymptotic unbiasedness: If bandwidths $h_k \to 0, N h_k \to \infty$, the estimator is asymptotically unbiased for entropy and mutual information.
• Boundary-bias correction: By accurately capturing local curvature of $\log f$, LGA avoids systematic underestimation near support boundaries.
• Alleviation of exponential sample complexity: Unlike k-NN analogs whose sample complexity grows exponentially in the true mutual information for strongly dependent variables, the LGA estimator maintains accuracy across the full range of dependency strengths.

Empirical studies demonstrate that LGA outperforms k-NN, graph-based, and MST-based entropy estimators in accurately measuring mutual information—even in the strongly dependent, low-noise limit—without saturation or severe bias (Gao et al., 2015).

2. Local Gaussian Approximation in Regression and Surrogate Modeling

Local Gaussian regression frameworks generalize classical locally weighted regression (LWR) by combining localized Gaussian basis functions with flexible, hierarchical Bayesian inference. Local Gaussian Regression (LGR) considers data $\{(x_n, y_n)\}_{n=1}^N$ and represents $f(x)$ as a sum over $M$ local experts, each associated with its own location $c_m$, localizing kernel $n_m(x)$, and small set of polynomial features:

$$f(x) = \sum_{m=1}^M \sum_{k=1}^K w_{m, k} [n_m(x) E_{m,k}(x)]$$

Bayesian inference is performed over the local weights, and the overall kernel is approximated by a sum of localized kernels. The variational mean-field approximation enables efficient training—scaling as $O(N M K^2 + M K^3)$ per pass—and prediction is effectively linear in the number of training points for fixed parameter dimension. Each local model may learn its own length-scale, allowing for adaptivity to spatial heterogeneity and automatic handling of nonstationarity.

Empirical evaluations reveal that LGR surpasses classical LWR and mixture-of-experts approaches in mean-squared error and efficiency: in the 2D "cross" function, the normalized MSE improves from ≈0.23 (LWR) and ≈0.17 (ME) to ≈0.0137 (LGR with length-scale learning); for the SARCOS inverse-dynamics task (N ≈44k), nMSE on joint 1 drops from 0.045 (LWPR) to 0.015 (LGR) (Meier et al., 2014).

3. Local Gaussian Approximation in Large-Scale Gaussian Process Models

Local Gaussian approximations underpin several methodologies designed to scale Gaussian process (GP) models to large datasets by exploiting the rapid decay of the kernel or the locality of influence near prediction sites. The main strategies include:

• TwinGP framework (Vakayil et al., 2023): The covariance is modeled as the sum of a "global" (low-rank or inducing-point) kernel and a "local" compactly supported kernel. For each test point, a small subset of $l$ local neighbors is selected, and a standard GP fit is performed using only local kernels. Predictions combine these local fits with a small number of global points, exploiting block structure for efficient inversion.
• Local experts and Distributed GPs (Jalali et al., 2020): Training is divided across $M$ disjoint or overlapping "local" experts (using data partitions), each yielding a GP predictive distribution. Classical aggregation via product-of-experts assumes conditional independence but this often fails in practice. Newer aggregation schemes detect and incorporate statistical dependencies among experts by fitting a Gaussian graphical model to predictive means and estimating a precision matrix, followed by clustering and dependency-aware combination to ensure statistical consistency and calibration.

In full local GP approximation (Gramacy et al., 2013), a small neighborhood is adaptively constructed (e.g., by sequential selection minimizing local predictive variance), and per-test-point kriging is performed using only the local subset. Key features include:

• Substantial computational savings, with per-point costs $O(n^3)$ for local subset size $n \ll N$ as opposed to $O(N^3)$ globally.
• Algorithmic schemes for fast matrix updates (partitioned inverses, Cholesky updates), and capability for full parallelization.
• Adaptivity to nonstationarity by re-estimating local kernel parameters at each prediction site.

4. Local Gaussian Approximation for Kernel and Random Feature Methods

Random feature approximations of the Gaussian kernel are ubiquitous for linearizing kernel methods; however, global approximations (e.g., via Maclaurin or RFF expansions) can fail for high-frequency data. Local Gaussian approximation here refers to:

• Localized Maclaurin / polynomial-sketch features: For a given region or cluster of the input space, random features are "recentered" around a local centroid, exploiting the shift-invariance of the Gaussian kernel. This ensures non-vanishing feature norms and tight kernel approximation in high-frequency, low-length-scale regimes (Wacker et al., 2022).
• Clustering-based localization: The input space is partitioned into regions (using, e.g., farthest-point clustering with a radius proportional to the kernel length-scale). Training data are associated with centroids, and localized feature maps are constructed per region, leading to a substantial reduction in kernel approximation error (KL divergence, RMSE) and avoiding collapse of the predicted variance.

For kernel regression, this localization dramatically improves accuracy compared to global random features, especially when the true kernel length-scale is small; in large-sample settings and smoother kernels, global and local feature schemes become comparable.

5. Local Gaussian Approximation in Empirical Process Theory and Quantum Dynamics

In empirical process theory, local Gaussian approximation refers to powerful coupling results that allow one to approximate suprema (or more general functionals) of empirical processes (e.g., local kernel density estimators, series-sieve estimators) by their Gaussian analogs without requiring classic Donsker conditions. Under minimal moment and entropy integrability assumptions, explicit non-asymptotic error rates are obtained. Applications include constructing confidence bands for nonparametric kernel or series estimators and deriving precise rates for convergence of statistics indexed by growing function classes or shrinking bandwidth (Chernozhukov et al., 2012).

In quantum molecular dynamics, local Gaussian approximation is used to propagate semiclassical (Gaussian wavepacket) solutions: a local Taylor expansion (at cubic order) of the potential yields analytic expressions for expectation values, and high-order geometric integrators preserve symplectic structure, norm, and energy conservation under this locally Gaussian–approximated Hamiltonian dynamics (Fereidani et al., 2023).

6. Computational and Algorithmic Considerations

Across different domains, the practical realization of local Gaussian approximation exhibits recurring computational motifs:

• Local design selection (e.g., for local GPs, variational or active learning-driven)
• Weighted MLE for Gaussian parameter fitting, often using kernel bandwidths set by nearest-neighbor distances
• Efficient blockwise matrix operations (partitioned inverses, Cholesky updates)
• Use of locality-sensitive data structures (kd-trees, cluster centroids) to accelerate neighbor search and partitioning
• Variational and mean-field approximate inference when the local Gaussian is embedded within a larger Bayesian latent representation (e.g., conditional variational schemes, structured approximate posteriors (Tan et al., 2019))
• Explicit recentering/localization of feature maps in kernel methods to maintain accuracy at high frequencies or away from data support

The following table synthesizes core algorithmic patterns:

Domain	Local Gaussian Role	Computation
Density/MI estimation	Local Gaussian fit for plug-in MI	$O(d^3)$ per point, small $k$
GP regression/surrogates	Local kriging/GP on subsets	$O(n^3)$ per test, scalable
Distributed GPs/expert agg.	Local expert posteriors	Partition, combine via graph agg.
Kernel methods/RFF	Localized feature map construction	Clustering, per-region featurize
Empirical process/statistics	Gaussian process coupling	Stein's/entropy-based coupling
Quantum dynamics	Taylor expansion, wavepacket prop.	Analytic expectation computation

Local Gaussian approximation is foundational for scalable nonparametric inference, efficient estimation of dependency information, model uncertainty quantification, and structure-preserving simulation in settings where both global flexibility and local adaptivity are crucial. The approach systematically outperforms local uniform approximations (k-NN, local constant, or plug-in) particularly under strong dependence, nonstationarity, boundary effects, or high-frequency phenomena (Gao et al., 2015, Meier et al., 2014, Vakayil et al., 2023, Wacker et al., 2022, Fereidani et al., 2023).