kNNSampler: Adaptive Nearest-Neighbour Methods

• kNNSampler is a nearest-neighbour based framework that leverages adaptive metrics to perform density estimation and stochastic imputation.
• It employs local sampling strategies using the k closest points to recover empirical distributions and quantify uncertainty in various domains.
• The method is theoretically validated with consistency proofs and is applied in manifold estimation, network sampling, and missing data imputation.

kNNSampler is a class of nearest-neighbour-based methodologies for density estimation, representative object sampling, stochastic imputation, and nonparametric function estimation, characterized by adaptive, local selection of sample points according to neighbourhood structure. While the specific operational definitions and domains vary (density estimation, survey imputation, ensemble classification, network sampling), all kNNSampler-like methods share a core mechanism: leveraging local neighbourhoods formed via a metric or similarity function to approximate data-dependent quantities, often in a data-driven and stochastic manner. These methods are deployed in settings ranging from manifold-supported probability densities through graph and spatial sampling, to empirical distributional recovery under missing data.

1. Foundational Methodologies

kNNSampler approaches select, weigh, and/or sample data points based on their local neighbourhood, typically defined as the k closest (or most similar in terms of an abstract metric or proximity function) objects to a query point. Canonical algorithms from this family include:

• Kernelized k-NN Density Estimator (Manifold Context): Points $x_1, x_2, \ldots, x_n$ on a Riemannian manifold $(M, g)$ are used to construct a density estimate at $p \in M$ via an adaptive, data-dependent bandwidth $\zeta_n(p)$ determined by the geodesic distance to the $k$-th nearest neighbour:

$$\widetilde{f}_n(p) = \frac{1}{n\, \zeta_n^d(p)} \sum_{j=1}^n \frac{1}{\theta_{\mathbf{x}_j}(p)} K\left( \frac{d_g(p, \mathbf{x}_j)}{\zeta_n(p)} \right)$$

where $\theta_{\mathbf{x}_j}(p)$ is the volume density and $K$ is a kernel function with compact support (Henry et al., 2011).

• Neighbourhood Representative Sampling (Network/Data Map Context): Sampling is based on proximity rank (number of times an object is a neighbour of others) and degree, selecting representatives via a representativeness score

$$r(o) = \frac{k(o)}{\log_x d(o)} \text{ or 0/1 for degenerate cases}$$

for scalable subgraph or spatial sampling (Kudelka et al., 2014).

• Stochastic Imputation by kNN Empirical Distribution (Missing Data Context): Instead of imputing with the mean, missing values are imputed by sampling randomly from the observed responses of the $k$ nearest neighbours. This procedure yields a conditional empirical distribution estimate:

$$\widehat{P}(y \mid x) = \frac{1}{k} \sum_{j=1}^k \delta(y - y_j),$$

and theoretically achieves convergence in conditional distribution via RKHS mean-embedding bounds (Pashmchi et al., 10 Sep 2025).

2. Theoretical Properties

kNNSampler methods admit rigorous analysis hinging on local adaptivity and sample-dependent bandwidth or neighbourhood size:

• Consistency & Asymptotics: Under regularity (density bounded away from zero, smoothness, appropriate $k_n$ scaling), uniform consistency and asymptotic normality are established for manifold-based estimators (Henry et al., 2011). For stochastic imputation, the mean-embedding error of the kNN empirical distribution converges as:

$$\|\Phi(P(\cdot | x)) - \Phi(\widehat{P}(\cdot | x))\|^2 \leq C_1 \frac{\ln n}{k} + C_2 \left(\frac{k}{n}\right)^{d/2},$$

with minimax-optimal rate at $k = O(n^{2/(2+d)})$ if $d$ is the intrinsic dimension (Pashmchi et al., 10 Sep 2025).

• Bandwidth Selection: The local bandwidth (distance to $k$-th neighbour) adapts to data density, with the injectivity radius ensuring geometric regularity on manifolds (Henry et al., 2011). In imputation, $k$ is chosen to balance bias and variance via fast cross-validation (Pashmchi et al., 10 Sep 2025).

3. Algorithmic Implementation and Complexity

A generic kNNSampler pipeline involves:

• For each query:
  1. Compute distances/similarities to all data points.
  2. Identify the k nearest neighbours.
  3. Apply local aggregation (density estimate, representative sampling, empirical distribution, etc.).

• Time complexity is $O(N M)$ for $N$ points with average $M$ neighbours, but efficient spatial indexing (KD-tree, ball-tree) or batch processing typically yields nearly linear scaling (Kudelka et al., 2014).

In the imputation context (Pashmchi et al., 10 Sep 2025):

def knn_sampler(X_obs, y_obs, X_miss, k): knn = NearestNeighbors(n_neighbors=k).fit(X_obs) indices = knn.kneighbors(X_miss, return_distance=False) # Stochastic imputation return [np.random.choice(y_obs[idx]) for idx in indices]

4. Applications and Empirical Performance

• Manifold Density Estimation: Accurate density recovery on non-Euclidean domains (spheres, cylinders) with quantification of bias-variance and finite-sample error (Henry et al., 2011).
• Network/Subgraph Sampling: Preserves key topological metrics (degree distribution, clustering coefficients) and local cluster structure, even under aggressive sampling (Kudelka et al., 2014).
• Stochastic Imputation and Uncertainty Quantification: Recovers the full distribution—including multimodality and heteroscedasticity—of missing values in survey, ring, and nonlinear data models. Provides empirical quantile-based prediction intervals (Pashmchi et al., 10 Sep 2025).
• Representative Sample Selection: Retains cluster centers and internal data structure better than naïve random, stratified, or fixed-radius sampling, with robust performance in vector, network, and spatial domains (Kudelka et al., 2014).

5. Comparison with Other Methods

Method	Target Estimate	Uncertainty Quantification	Typical Usage
kNNImputer	Conditional mean	No	Point imputation
kNNSampler	Conditional distribution	Yes	Multiple imputation, distributional recovery
KDE/kNN-KDE	Kernel density	Yes (requires bandwidth)	Smooth density estimation
Representative kNN (Kudelka et al., 2014)	Node/sample selection	Structural density preserved	Data reduction

For example, kNNSampler in imputation outperforms regression mean imputation and kernel-smoothing-based KDE methods, with only a single discrete hyperparameter and fewer tuning complexities (Pashmchi et al., 10 Sep 2025).

6. Future Directions

Areas for further research include:

• Adaptive Selection of k: Systematic or local adaptation of $k$, possibly via intrinsic density or geometric data characteristics (Pashmchi et al., 10 Sep 2025).
• Extension to Complex Missing Data Mechanisms: Beyond missing at random (MAR) to not missing at random (NMAR), incorporating response-dependent or sample-selection bias.
• Integration with Multiple Imputation and Inference Pipelines: Downstream statistical analysis, variance estimation, and principled uncertainty quantification; broader case studies in survey statistics, health, and industrial applications.
• High-dimensional and Non-Euclidean Covariate Spaces: Utilization of dimension-intrinsic scaling to manage curse of dimensionality, especially for functional or manifold-valued predictors.
• Hybrid Combinations: Embedding kNNSampler in ensemble frameworks, network sampling, or density recovery on structured domains, leveraging the method's inherent local adaptivity.

7. Summary

kNNSampler constitutes a versatile, theoretically supported, and empirically robust framework for adaptive inference from local neighbourhoods, applicable to density estimation, data reduction, stochastic imputation, and network sampling. Its principal advantage lies in leveraging the empirical conditional distribution—rather than a deterministic summary—thereby naturally supporting uncertainty quantification and distributional recovery. Its theoretical guarantees (consistency, minimax-optimal convergence) and empirical performance (robust structure preservation, uncertainty intervals) make it well suited for a range of modern statistical and data science applications. Further work will refine its adaptive components, extend its reach to broader data-generating mechanisms, and integrate its stochastic outputs into large-scale inference pipelines (Pashmchi et al., 10 Sep 2025, Henry et al., 2011, Kudelka et al., 2014).