Adaptive k*-Nearest Neighbors

• k*-Nearest Neighbors is a family of adaptive nonparametric methods that determine the optimal neighborhood size per query to balance bias and variance.
• Techniques such as convex optimization, Bayesian inference, geometric rules, and kernelized sparse coding are used to compute the adaptive parameter k*.
• These methods enhance predictive performance by dynamically adjusting local smoothing based on data density, noise levels, and label ambiguity.

A $k^*$-Nearest Neighbors algorithm denotes a general class of approaches in which the neighborhood size $k^*$—previously a fixed user-chosen parameter in standard $k$-NN—is instead selected adaptively, often per query, according to a criterion that expresses local optimality, Bayesian plausibility, or algorithmically-learned tradeoff. These techniques fundamentally address the well-known bias–variance dilemma in nonparametric learning, yielding improved control over local estimation and predictive accuracy. The literature encompasses convex optimization, Bayesian evidence maximization, kernelized sparse coding, and explicit geometric constructions for the discovery or inference of $k^*$. This entry synthesizes major formulations and algorithmic principles for $k^*$-NN, including both statistical and algorithmic perspectives.

1. Foundations of Adaptive Neighborhood Size

The principal motivation for $k^*$-NN is to circumvent the global hyperparameter tuning of $k$, which often incurs costly cross-validation or model selection cycles and is suboptimal in heterogeneous data regimes. Instead, for each query $x_0$, a (potentially different) optimal $k^*$ is computed by optimizing a surrogate risk or following a probabilistic model for the neighborhood. Early approaches to this problem include bias–variance balancing (Anava et al., 2017), change-point detection in the local ordering around $x_0$ (Nuti, 2017), and marginal likelihood maximization in generalized Bayesian models (Kim, 2016).

The general prediction rule in $k^*$0-NN is: $k^*$1 where the support of the optimal weights $k^*$2 is restricted to the $k^*$3 nearest neighbors of $k^*$4; both $k^*$5 and $k^*$6 are defined by an adaptive criterion. The result is an estimator that leverages a locally optimal level of smoothness, adapting to density, noise, or label ambiguity in the vicinity of $k^*$7 (Anava et al., 2017, Jodas et al., 2022).

2. Mathematical Formulations for $k^*$8 Selection

Several distinct approaches to computing $k^*$9 are prominent in the literature:

• Bias–Variance Convex Relaxation (Anava et al., 2017): For regression, $k$0 arises as the maximal set of neighbors for which the combined objective

$k$1

minimizes the finite-sample tradeoff under constraints $k$2 (the simplex). The unique optimizer is thresholded: $k$3, with the implicit cutoff $k$4, where $k$5 encodes scaled distances.

• PL-kNN Geometric Rule (Jodas et al., 2022): $k$6 is the cardinality of the set of training points lying within a semicircle of radius $k$7 (Manhattan distance to the nearest class median centroid), oriented toward the “nearest” class. No parameter tuning is required.
• Bayesian Change-Point Model (Nuti, 2017): The posterior distribution $k$8 is computed recursively as the distribution over run-length (number of same-segment nearest neighbors) using the local likelihood and a hazard function. $k$9 is typically chosen as the MAP (maximum a posteriori) value.
• Bayesian Mutual/Symmetric $k^*$0-NN (Kim, 2016): $k^*$1 is found by maximizing the log-evidence (marginal likelihood) $k^*$2 under a Gaussian process Laplacian covariance induced by $k^*$3-NN graph structure. The optimal $k^*$4 yields the best model fit, trading off complexity and data fit without cross-validation.
• Kernelized Sparse Coding in Graph Models (Li et al., 23 Jan 2026): For each sample $k^*$5, an adaptive Lasso is solved over a composite (feature+class) kernel, yielding a sparse weight vector $k^*$6; the number of nonzero entries $k^*$7 is the learned neighborhood size $k^*$8 for $k^*$9.

The following table summarizes these criteria:

Approach	$k^*$0 Definition	Key Mechanism
Bias–Variance Convex	Threshold in convex surrogate	Stationarity/KKT
Bayesian Change-pt	MAP/posterior run-length	Change-point recursion
Bayesian Laplacian GP	MLE w.r.t. evidence	Laplacian evidence
Geometric (PL-kNN)	Count in semicircle of radius $k^*$1	Nearest centroid
Kernel Sparse Graph	$k^*$2 in adaptive Lasso	Sparse reconstruction

3. Algorithmic Realizations and Implementation

Details vary by formulation:

• k*-NN Convex Optimizer (Anava et al., 2017): For each query, distances are scaled, sorted, and a threshold $k^*$3 is found by explicit algebraic sweep. Final weights and $k^*$4 are directly computed from this value ($k^*$5 per query).
• Bayesian Change-Point (Nuti, 2017): The target’s ordered neighbor sequence is processed recursively using hazard and predictive likelihoods (see on-line Bayesian change-point update equations), yielding the full $k^*$6 posterior.
• PL-kNN (Jodas et al., 2022): The instance-specific neighborhood is determined in geometric space (by median-centric radius and directional filtering), with no tuning cycles; all points in the eligible sector become the $k^*$7.
• Kernel Sparse Graphs ($k^*$8NN-Graph) (Li et al., 23 Jan 2026): Each node’s $k^*$9 is discovered via kernel Lasso (with per-node regularization proportional to density), and the consensus label is precomputed over this neighborhood. Inference is routed through a hierarchical HNSW index.
• Bayesian Laplacian GP (Kim, 2016): Precompute neighbor graphs at all candidate $k^*$0, construct Laplacian, optimize evidence w.r.t. other GP hyperparameters, then select the maximizing $k^*$1. Algorithmic complexity is $k^*$2 per candidate $k^*$3.

In practical large-scale or high-dimensional regimes, additional techniques such as bandit-based Monte Carlo selection (Bagaria et al., 2018) and distributed protocols (Fathi et al., 2020) provide adaptivity for $k^*$4 while maintaining computational feasibility.

4. Bias–Variance Tradeoff and Theoretical Properties

A recurring theme in all $k^*$5-NN schemes is the explicit or implicit control of the local bias–variance tradeoff. Adjusting $k^*$6 dynamically as a function of query location density, label heterogeneity, or expected risk formalizes this tradeoff:

• In convex formulations (Anava et al., 2017), a larger $k^*$7 reduces variance but increases bias; the optimizer selects the cutoff at each point.
• PL-kNN (Jodas et al., 2022) demonstrates that $k^*$8 rises naturally in high-density regions, functioning as a low-variance estimator, while shrinking in sparse regions, acting as a low-bias estimator.
• Bayesian models (Nuti, 2017, Kim, 2016) control $k^*$9 via implicit priors on the run-length or model complexity as manifested in the evidence criterion or the hazard function.
• Empirical analyses routinely show that locally-adaptive $k$0 achieves lower error (regression or classification) than globally-tuned $k$1, with typical error decreases of 5-15% on benchmark datasets.

No approach universally dominates: the relative merits of each formulation depend on computational budget, application (classification vs regression), and available prior information.

5. Empirical Performance and Comparative Analysis

Key evaluation results demonstrate consistent advantages of $k$2-NN classifiers:

• k*-NN (Bias–Variance) (Anava et al., 2017): On eight UCI datasets, outperformed fixed-$k$3 and kernel regressors on 7/8 benchmarks with statistically significant gains.
• PL-kNN (Jodas et al., 2022): Achieved the highest F1-score on 7/11 datasets, globally outperforming both fixed-$k$4 and parameterless competitors (Wilcoxon and Nemenyi tests, $k$5).
• Bayesian mutual/symmetric $k$6-NN (Kim, 2016): Evidence-based $k$7 selection often chooses $k$8 very different from LOOCV, yielding substantially lower test error (up to $k$9 reduction in artificial datasets).
• kNN-Graph (Li et al., 23 Jan 2026): Adaptive sparsification and per-node $x_0$0 in HNSW structure yields classification accuracy surpassing all fixed-$x_0$1 and “one-size-fits-all” competitors, e.g., mean accuracy of 73.76% versus 72.22% (O$x_0$2NN baseline), alongside order-of-magnitude speedups.
• Bandit-Based Approaches (Bagaria et al., 2018): In extreme high-dimensional settings (e.g., Tiny ImageNet: $x_0$3), coordinate-wise adaptive nearest neighbor identification achieves 80× reduction in total coordinate computations relative to exact brute-force, matching or exceeding recall of approximate graph-based heuristics.

6. Algorithmic and Computational Considerations

While $x_0$4-NN methods provide compelling statistical guarantees, some practical tradeoffs are significant:

• Per-query computational complexity can be higher than fixed-$x_0$5 if each new instance requires custom optimization, e.g., full O($x_0$6) Laplacian inversions in Bayesian GP models (Kim, 2016).
• PL-kNN and convex $x_0$7-NN (bias–variance) method costs are dominated by distance calculations and sorting, remaining linearithmic in $x_0$8 per query (Jodas et al., 2022, Anava et al., 2017).
• Methods employing offline adaptive neighbor selection (e.g., kNN-Graph with HNSW (Li et al., 23 Jan 2026)) can transfer all expensive neighbor computations to the training phase, achieving $x_0$9 inference time at query—unlike all classical local $k^*$0-NN schemes.

Hyperparameter burden is generally modest: often a single ratio or density-scale in bias–variance or kernel-sparse models, while PL-kNN and Bayesian algorithms have negligible or data-driven parameter dependence.

7. Extensions, Limitations, and Future Directions

The versatility of $k^*$1-NN methods extends to:

• Graph and manifold domains: via multi-source Dijkstra (Har-Peled, 2016) or dynamic planar data structures (Berg et al., 2021).
• Distributed and parallel computation: randomized pruning and selection protocols allow efficient $k^*$2-NN in distributed settings, with round-optimal algorithms in the communication-centric $k^*$3-machine model (Fathi et al., 2020).
• Large-scale, high-dimensional, and dynamic data: coordinate sampling with stochastic filtering (Bagaria et al., 2018), as well as fully dynamic data structures (Berg et al., 2021), offer scalable $k^*$4-nearest neighbor retrieval.

Limitations typically arise in computational cost for exhaustive per-query optimization and the scalability of evidence maximization, as well as the requirement for density estimation or kernel selection in some frameworks (Li et al., 23 Jan 2026). In highly dynamic scenarios, adaptive graph or kernel representations may require costly retraining.

The $k^*$5-NN paradigm continues to influence the design of nonparametric estimators, guiding both the construction of new adaptive, parameterless algorithms and the analysis of sample complexity and generalization in locally weighted inference (Anava et al., 2017, Jodas et al., 2022, Li et al., 23 Jan 2026).