K-Nearest Neighbors (KNN) Algorithm

• K-Nearest Neighbors (KNN) is a nonparametric, instance-based learning method that classifies or predicts instances based on the k-nearest similar observations using a specified distance or similarity metric.
• KNN can perform both classification tasks (by assigning the class with the highest number of votes) and regression tasks (by averaging the target values), allowing it to handle various types of datasets with different characteristics and types of relationships.
• In practice, KNN's performance can be influenced by the choice of distance metric, feature scaling, and the number of nearest neighbors k; however, it is particularly suited for applications where local, nonlinear relationships are important, despite being computationally intensive when dealing with large datasets, especially in high-dimensional spaces.

K-nearest neighbors (KNN) is a nonparametric, instance-based learning method that predicts the label or response of a query from the observations most similar to it under a specified distance or similarity rule. It stores the training data rather than fitting a compact parametric decision function, and performs most computation at prediction time. In classification, the conventional rule assigns the class receiving the largest number of votes among the $k$ nearest observations; in regression, it commonly averages their target values. KNN has been extended through distance weighting, adaptive neighborhood selection, graph and manifold geometry, spatial dual-neighborhood construction, feature scaling, anomaly scoring, open-set rejection, neural imitation, approximate search, and quantum fidelity comparison.

1. Definition, notation, and basic decision rules

Let the training set be

$$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$

where $X_i\in\mathbb R^d$ is a feature vector and $L_i$ is its class label. For a query $Y$, KNN evaluates a distance such as the Euclidean distance,

$$\operatorname{dist}(X,Y)=\|X-Y\|_2 =\left(\sum_{j=1}^{d}(x_j-y_j)^2\right)^{1/2}.$$

The $k$ training points with the smallest distances are ordered from closest to farthest. Ordinary KNN predicts

$$\widehat{\omega}(Y) =\arg\max_{\omega_r\in\Omega} \sum_{j=1}^{k}\mathbf 1\{L_j=\omega_r\},$$

where $\Omega$ is the class set. Every selected neighbor contributes one vote, independently of its absolute distance. Ties may be broken arbitrarily, although particular applications may use the closest neighbor, random selection, or distance-weighted voting.

For regression, the standard estimator is

$\widehat f(x_0)=\frac{1}{k}\sum_{i=1}^{k}y_i,$

after ordering observations by distance from $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$0. The parameter $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$1 controls the bias–variance trade-off: small values produce highly local, low-bias and high-variance estimates, whereas large values reduce variance but may average across heterogeneous regions and increase bias. The limiting case $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$2 is especially sensitive to noise and isolated observations; as $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$3 grows toward the training-set size, the estimator approaches a highly biased global average (Sarma et al., 2013, Anava et al., 2017).

KNN is commonly described as a lazy-learning algorithm. Training primarily consists of storing observations, while prediction requires distance computation, neighbor selection, and aggregation. A brute-force query over $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$4 $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$5-dimensional observations requires approximately $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$6 distance work and typically $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$7 distance evaluations. Storage is generally $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$8, or $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$9 when the feature dimension is included. Without indexing or approximation, the neighborhood-search cost remains linear in the number of stored observations.

Classical asymptotic results recalled in the literature state that, under the usual conditions and sufficiently many training examples, $X_i\in\mathbb R^d$0-NN approaches Bayes classification. The asymptotic error of 1-NN is bounded above by twice the Bayes error in the classical result cited by the authors of “An improvement to k-nearest neighbor classifier” (Sarma et al., 2013). These properties do not imply that a particular finite-sample value of $X_i\in\mathbb R^d$1, metric, feature representation, or weighting rule is optimal.

2. Geometry, scaling, and weighting

Distance metrics and feature scale

KNN depends fundamentally on the geometry induced by its distance function. Common choices include Minkowski distances,

$X_i\in\mathbb R^d$2

with Manhattan distance for $X_i\in\mathbb R^d$3 and Euclidean distance for $X_i\in\mathbb R^d$4. Mahalanobis distance incorporates feature covariance,

$X_i\in\mathbb R^d$5

All distance-based variants remain sensitive to feature representation. A feature with a large numerical range can dominate Euclidean or Minkowski distances, even when it is weakly predictive. Standard preprocessing includes Z-score normalization,

$X_i\in\mathbb R^d$6

and Min-Max scaling. These methods equalize numerical scale but do not necessarily make every feature equally relevant to the prediction task.

The Feature Importance based Dynamic Scaling method, or FIDS, first normalizes features, then trains a Random Forest-style ensemble, estimates feature-related out-of-bag errors, and multiplies each normalized feature by an importance value. The resulting distance can be written as

$X_i\in\mathbb R^d$7

FIDS therefore acts as a supervised diagonal metric-learning heuristic: it modifies individual feature scales but does not learn a full covariance metric or explicitly model all feature interactions (Bhardwaj et al., 2018). In the reported experiments, FIDS improved accuracy over uniformly Z-score-scaled KNN on five of six UCI datasets, but performed worse on Breast Cancer. The supplied study does not provide a complete specification of the out-of-bag implementation, feature-weight normalization, or train-test protocol.

Distance-weighted voting

Ordinary KNN assigns equal weight to all selected observations. Weighted KNN instead uses a distance-dependent vote. A linear rule attributed to Dudani assigns

$X_i\in\mathbb R^d$8

where $X_i\in\mathbb R^d$9 is the distance to the $L_i$0-th neighbor. This gives the closest neighbor weight one and the farthest selected neighbor weight zero.

Gaussian-weighted KNN replaces linear decay with

$L_i$1

and predicts the class maximizing

$L_i$2

The method is formally related to a truncated Gaussian-kernel or Parzen-window density classifier: it uses only the $L_i$3 nearest observations rather than all training points. In the reported experiments, features are normalized to zero mean and unit variance and $L_i$4 is used. The method is orthogonal to prototype selection, feature reduction, editing, compact data structures, indexing, and bootstrap-based training-set transformations. Its weighting rule does not itself reduce the cost of neighbor retrieval (Sarma et al., 2013).

Adaptive neighborhood size

A globally fixed $L_i$5 may be inappropriate when query points occupy regions with different density, smoothness, or noise levels. The $L_i$6-NN method derives both the support and the weights locally from an explicit bias–variance objective. Under a Lipschitz regression model and bounded independent noise, it uses weights

$L_i$7

where $L_i$8 is the Lipschitz parameter and $L_i$9 is determined by the noise bound and confidence level. Thus, $Y$0 varies by query, and the positive weights form a truncated linear profile over the nearest observations. The method obtained lower average absolute error than standard KNN and Gaussian Nadaraya–Watson regression on seven of eight reported datasets, while losing to Nadaraya–Watson on PopFailures (Anava et al., 2017).

A Bayesian alternative treats the neighborhood size as a target-specific random variable. After ordering training observations by distance from the query, it interprets the appropriate $Y$1 as a change point separating observations generated by the same local distribution from observations generated by another local regime. Bayesian online change-point recursion produces a posterior $Y$2, avoiding MCMC for exponential-family models with conjugate priors. The reported Ripley classification error was $Y$3, compared with $Y$4 for manually tuned global KNN; on the Nuclear Power Plant regression data, average absolute error was $Y$5, compared with $Y$6 for global KNN (Nuti, 2017).

3. Manifold, graph, spatial, and temporal extensions

Graph-based semi-supervised KNN

Euclidean proximity can be misleading when observations lie near a nonlinear manifold. A graph-based semi-supervised method, called manifold $Y$7NN or $Y$8NN, constructs a graph from labeled and unlabeled observations, incorporates must-link relations between same-class labeled points and cannot-link relations between differently labeled points, and computes a constrained tired random walk (TRW) similarity.

For a graph transition matrix $Y$9 and decay parameter $$\operatorname{dist}(X,Y)=\|X-Y\|_2 =\left(\sum_{j=1}^{d}(x_j-y_j)^2\right)^{1/2}.$$0,

$$\operatorname{dist}(X,Y)=\|X-Y\|_2 =\left(\sum_{j=1}^{d}(x_j-y_j)^2\right)^{1/2}.$$1

A symmetric similarity is obtained from the directed walk matrix by

$$\operatorname{dist}(X,Y)=\|X-Y\|_2 =\left(\sum_{j=1}^{d}(x_j-y_j)^2\right)^{1/2}.$$2

The classifier selects labeled neighbors according to TRW similarity and predicts the class with the largest cumulative similarity,

$$\operatorname{dist}(X,Y)=\|X-Y\|_2 =\left(\sum_{j=1}^{d}(x_j-y_j)^2\right)^{1/2}.$$3

Unlike ordinary geodesic KNN, which may be dominated by a single short circuit, TRW aggregates all paths while discounting longer paths. The batch method has approximately $$\operatorname{dist}(X,Y)=\|X-Y\|_2 =\left(\sum_{j=1}^{d}(x_j-y_j)^2\right)^{1/2}.$$4 complexity for direct matrix inversion and potentially $$\operatorname{dist}(X,Y)=\|X-Y\|_2 =\left(\sum_{j=1}^{d}(x_j-y_j)^2\right)^{1/2}.$$5 memory. A sequential version reconstructs a new sample from local neighbors using a nonnegative affine reconstruction and estimates its TRW weights without recomputing the full inverse. On synthetic and UCI experiments with very few labeled examples, $$\operatorname{dist}(X,Y)=\|X-Y\|_2 =\left(\sum_{j=1}^{d}(x_j-y_j)^2\right)^{1/2}.$$6NN generally had lower error than ordinary, weighted, and geodesic KNN, although its advantages depend on sufficient sampling of the underlying manifold and reliable labels (Tu et al., 2016).

Spatial double-neighborhood KNN

Spatial KNN incorporates two distinct notions of proximity: covariate similarity and geographic proximity. The method first selects $$\operatorname{dist}(X,Y)=\|X-Y\|_2 =\left(\sum_{j=1}^{d}(x_j-y_j)^2\right)^{1/2}.$$7 spatially nearby observations, then selects $$\operatorname{dist}(X,Y)=\|X-Y\|_2 =\left(\sum_{j=1}^{d}(x_j-y_j)^2\right)^{1/2}.$$8 covariate neighbors within that spatial neighborhood. The combined weight is

$$\operatorname{dist}(X,Y)=\|X-Y\|_2 =\left(\sum_{j=1}^{d}(x_j-y_j)^2\right)^{1/2}.$$9

where $k$0 is a random covariate bandwidth determined by the $k$1-th covariate neighbor and $k$2 is the spatial bandwidth determined by the $k$3-th spatial neighbor.

The regression predictor is the normalized weighted average of observed responses. For categorical responses, the same weights produce weighted class frequencies. Under smoothness, density, kernel, mixing, and local-identical-distribution assumptions, the paper proves almost-complete convergence of the predictor and almost-sure convergence of the classifier to Bayes risk. The stated regression rate is

$k$4

Applications to Swiss Jura soil prediction and Senegal fisheries classification show that spatial KNN can outperform cokriging and several nonspatial baselines in some settings, but fixed-bandwidth kernels, SVMs, logistic regression, and ordinary KNN with coordinates are best on particular responses or evaluation metrics (Ahmed et al., 2018).

Time-series forecasting

For time series, a query is a recent lagged subsequence and its neighbors are historical subsequences followed by known future blocks. Classical Parameters Tuning in Weighted Nearest Neighbors, CPTO-WNN, searches all eligible historical patterns. Fast Parameters Tuning in Weighted Nearest Neighbors, FPTO-WNN, searches a reduced set of horizon-aligned historical patterns.

Both methods select the window length $k$5 and neighbor count $k$6 by rolling, time-series cross-validation and use Rank Order Centroid weights,

$k$7

Forecasts are generated recursively, often after Box–Cox transformation and first differencing. CPTO-WNN was more accurate than FPTO-WNN on the reported U.S. retail and food-services series, but FPTO-WNN required less computation. On the U.K. milk-production series, both methods generally outperformed SARIMA, Holt–Winters, and exponential smoothing state-space models, with FPTO-WNN having lower average computation time (Tajmouati et al., 2021).

4. Approximate search, graph construction, and computational acceleration

Exact KNN search compares a query with every stored observation. For large or high-dimensional datasets, approximate indexing can reduce the number of candidate distances.

Random projection forests, or rpForests, construct an ensemble of recursively partitioned random-projection trees. At each node, a projection direction is chosen, optionally from $k$8 candidates by selecting the direction with the largest projected standard deviation. A query reaches one leaf in each tree, and the union of those leaves forms a candidate set. Exact distances are then computed only within that union. A single tree has expected construction complexity $k$9 and average query traversal cost $$\widehat{\omega}(Y) =\arg\max_{\omega_r\in\Omega} \sum_{j=1}^{k}\mathbf 1\{L_j=\omega_r\},$$0; the forest is naturally parallel across trees and queries (Yan et al., 2018).

The probability that two close points are separated in every tree decreases exponentially with the number of trees under the paper’s geometric assumptions involving neck size, tree depth, and controlled shrinkage. Experiments on nine datasets showed rapidly decreasing missing rates and distance discrepancies as the number of trees increased, often approaching zero with approximately 20–40 trees. The method sacrifices unconditional exactness: a true neighbor may be absent from all candidate leaves, and increasing the number of trees increases index cost, memory use, and candidate-set size.

K-Nearest Neighbor Descent, or NND, uses the friend-of-a-friend principle: the neighbors of a current neighbor become candidates for the next approximation. With approximately $$\widehat{\omega}(Y) =\arg\max_{\omega_r\in\Omega} \sum_{j=1}^{k}\mathbf 1\{L_j=\omega_r\},$$1 candidates per point per round and favorable neighborhood overlap, the empirical work pattern is $$\widehat{\omega}(Y) =\arg\max_{\omega_r\in\Omega} \sum_{j=1}^{k}\mathbf 1\{L_j=\omega_r\},$$2. The paper rigorously justifies a related second-neighbor range-query algorithm under homogeneous Poisson and geometric assumptions, but not the general ranking-based NND algorithm. It also constructs metrizable ranking systems in which friend-of-friend information is essentially uninformative and the method requires quadratic work. Thus, subquadratic KNN descent depends on exploitable locality and correlated neighborhoods, not merely on the existence of a metric (Baron et al., 2019).

KNN can also be combined with learned or neural representations. The sequence-to-sequence and memory-network models in “k-Nearest Neighbors by Means of Sequence to Sequence Deep Neural Networks and Memory Networks” use KNN-generated neighbor labels and feature vectors as structured supervision. V2LS predicts ordered neighbor labels; V2VS predicts ordered neighbor feature vectors; V2VSLS predicts both and a final class. The models can generate synthetic feature vectors for oversampling and have out-of-core variants that use random batches when the complete dataset cannot fit in memory. V2VSLS achieved the best reported F-1 scores on the four structured datasets and improved upon SMOTE and ADASYN in several oversampling comparisons, although it requires KNN preprocessing and multiple neural outputs (Xu et al., 2018).

5. Specialized decision rules and applications

Open-set and abstaining KNN

Ordinary KNN is a closed-set classifier: it must return one of the labels observed during training. Advanced KNN, or A-kNN, computes a Training Class Area,

$$\widehat{\omega}(Y) =\arg\max_{\omega_r\in\Omega} \sum_{j=1}^{k}\mathbf 1\{L_j=\omega_r\},$$3

and defines a class acceptance area approximately as

$$\widehat{\omega}(Y) =\arg\max_{\omega_r\in\Omega} \sum_{j=1}^{k}\mathbf 1\{L_j=\omega_r\},$$4

where $$\widehat{\omega}(Y) =\arg\max_{\omega_r\in\Omega} \sum_{j=1}^{k}\mathbf 1\{L_j=\omega_r\},$$5 is a user-selected gap constant. After identifying the majority-vote class $$\widehat{\omega}(Y) =\arg\max_{\omega_r\in\Omega} \sum_{j=1}^{k}\mathbf 1\{L_j=\omega_r\},$$6, the query is rejected as unknown if its minimum distance to the training set exceeds $$\widehat{\omega}(Y) =\arg\max_{\omega_r\in\Omega} \sum_{j=1}^{k}\mathbf 1\{L_j=\omega_r\},$$7. On Iris and BUPA experiments with artificially introduced far-away unknown observations, A-kNN with small gap constants rejected all 20 unknown cases in some configurations. Its performance deteriorated as $$\widehat{\omega}(Y) =\arg\max_{\omega_r\in\Omega} \sum_{j=1}^{k}\mathbf 1\{L_j=\omega_r\},$$8 increased, and the unknown instances were not all rejected in the Alpha Beta Detection experiment (Asim et al., 2020).

Flexible KNN, or Flex$$\widehat{\omega}(Y) =\arg\max_{\omega_r\in\Omega} \sum_{j=1}^{k}\mathbf 1\{L_j=\omega_r\},$$9NN, replaces fixed $\Omega$0 with a maximum acceptable distance $\Omega$1. It uses every training sample within the radius,

$\Omega$2

and sets $\Omega$3. If $\Omega$4, it abstains rather than returning a forced label. In IMS indoor localization, Flex$\Omega$5NN converted some impossible cross-condition classifications into explicit no-label decisions. With training rooms absent from the training data, it returned no label for 713 of 4,361 test samples and achieved an overall correct-or-no-label rate of $\Omega$6, compared with $\Omega$7 accuracy for standard $\Omega$8 KNN. Radius selection remains difficult, and nearby unknown-class samples may still be misclassified (Müller, 2023).

Anomaly detection

In anomaly detection, the training set contains only normal examples. A conventional KNN anomaly score is typically the sum or average of distances from a test embedding to its nearest normal training embeddings. The k-NNN operator augments this with local geometry around each direct neighbor. For every normal training feature, it estimates local eigenvectors $\Omega$9 and eigenvalues $\widehat f(x_0)=\frac{1}{k}\sum_{i=1}^{k}y_i,$0. The anomaly score is

$\widehat f(x_0)=\frac{1}{k}\sum_{i=1}^{k}y_i,$1

Low-variance directions receive larger weights, while high-variance directions are treated as more normal. The operator is modular and can replace the ordinary KNN component in systems such as SPADE and PANDA without changing the remaining detector. On MVTec, the reported mean AUROC increased from $\widehat f(x_0)=\frac{1}{k}\sum_{i=1}^{k}y_i,$2 to $\widehat f(x_0)=\frac{1}{k}\sum_{i=1}^{k}y_i,$3 for the plain feature detector, from $\widehat f(x_0)=\frac{1}{k}\sum_{i=1}^{k}y_i,$4 to $\widehat f(x_0)=\frac{1}{k}\sum_{i=1}^{k}y_i,$5 for SPADE, and from $\widehat f(x_0)=\frac{1}{k}\sum_{i=1}^{k}y_i,$6 to $\widehat f(x_0)=\frac{1}{k}\sum_{i=1}^{k}y_i,$7 for PANDA. The method requires additional preprocessing, eigenvector storage, feature partitioning, and hyperparameter selection (Nizan et al., 2023).

Radar detection

KNN can be used as a radar detector by treating raw whitened observations or conventional radar statistics as feature vectors. The test feature is compared with labeled training features under $\widehat f(x_0)=\frac{1}{k}\sum_{i=1}^{k}y_i,$8 and $\widehat f(x_0)=\frac{1}{k}\sum_{i=1}^{k}y_i,$9, and the fraction of $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$00-labeled neighbors is compared with a threshold $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$01. For a statistics-based feature family built from Kelly-, AMF-, and ACE-related quantities, the authors derive expressions for false-alarm and detection probabilities and prove a constant false-alarm-rate property with respect to the unknown disturbance covariance. The raw-data feature has high matched detection performance but is not theoretically CFAR. Feature selection and weighting allow interpolation between matched performance and steering-vector-mismatch robustness (Coluccia et al., 2019).

Quantum KNN

Quantum KNN, or Q$$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$02NN, replaces classical distance with quantum-state fidelity. For pure states,

$$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$03

The algorithm encodes fidelities using a coherent swap-test construction, converts amplitude information into digital fidelity estimates, and reduces neighbor selection to quantum $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$04-maxima finding. Its stated query complexity is

$$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$05

where $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$06 is the number of labeled training states. This is a query-complexity result conditional on efficient state-preparation, fidelity-comparison, phase-estimation, and analog-to-digital oracles. The method is intended for quantum data that can be prepared directly, avoiding explicit quantum-state tomography. Reported applications include entanglement classification and promised quantum-state identification (Basheer et al., 2020).

6. Extensions, limitations, and methodological considerations

Local differential regression

Differential Nearest Neighbors Regression, or DNNR, modifies both neighborhood construction and prediction. It estimates a local gradient around each selected neighbor and transports that neighbor’s target to the query through a Taylor approximation:

$$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$07

The gradient is estimated from normalized finite differences by least squares,

$$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$08

DNNR may also learn a diagonal feature-scaling matrix using a correlation objective relating distance to local prediction error. It retains KNN’s case-based interpretability while replacing zero-order target averaging with local Taylor transport. On more than 250 regression datasets, it was reported as competitive with gradient boosting and MLPs, while its advantages depended on smooth continuous structure, adequate local geometry, and stable gradient estimation (Nader et al., 2022).

Class-wise aggregation

PMM-KNN replaces global majority voting with class-wise aggregation based on the Power Muirhead Mean. It selects up to $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$09 nearby points separately for each class, computes a support-weighted PMM centroid for each class, and assigns the query to the nearest class centroid. The method is intended to reduce the influence of isolated observations and mitigate direct class-frequency effects. On five reported datasets, PMM-KNN had higher mean accuracy than ordinary KNN, with statistically significant improvements on Landsat, EEG Eye State, and Digits. The supplied paper does not fully specify a tractable vector-valued implementation of the permutation-based Muirhead Mean or provide a formal complexity analysis (Shahnazari et al., 2022).

Sequential neighborhoods and selective ensembles

The OExNRule ensemble uses an extended sequential neighborhood. It selects the point closest to the query, then repeatedly selects the closest unused point to the most recently selected point. Each base learner is trained on a bootstrap sample and a random feature subset. After evaluating out-of-bag error, only the best-performing models are retained for final voting. The principal reported configuration uses $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$10 base models, $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$11, $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$12, and approximately 25% of the models selected. Across 17 benchmark datasets, OExNRule obtained the best reported mean accuracy on 10 datasets and was more robust than ordinary KNN under added uniformly distributed contrived features. Its sequential rule can nevertheless propagate an error from the first selected point, and the ensemble introduces substantial training and model-selection cost (Ali et al., 2022).

Information-weighted KNN

Information Modified KNN, or IMKNN, combines distance-based significance weights with mutual-information-based values and Shapley-inspired marginal contributions. Its final class score interpolates between distance-weighted voting and information-value-weighted voting. The paper reports average improvements over traditional KNN on 12 datasets of $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$13 in accuracy, $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$14 in precision, and $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$15 in recall. However, the manuscript does not fully specify the construction of joint distributions from continuous distance matrices or the exact enumeration and approximation of coalitions in the Shapley-inspired calculation (Vahedifar et al., 2023).

General limitations

KNN’s main advantages are nonparametric flexibility, simple implementation, nonlinear decision boundaries, and case-based interpretability. Its principal limitations are equally structural:

1. Metric dependence: inappropriate distances, feature scales, correlations, or categorical representations produce misleading neighborhoods.
2. Curse of dimensionality: as dimensionality increases, distances can become less discriminative and irrelevant features can distort local structure.
3. Choice of $$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$16: small values have high variance, while large values blur class boundaries and local regression structure.
4. Data density: sparse regions, distribution shift, and out-of-support queries are poorly handled by a forced fixed-$$\mathcal X=\{(X_1,L_1),(X_2,L_2),\ldots,(X_n,L_n)\},$$17 decision.
5. Class imbalance: majority voting can favor classes with more training observations, although class-weighted, local-prior, class-wise, and radius-based modifications address this in different ways.
6. Computational cost: exact prediction and graph construction can be expensive; weighting rules do not by themselves accelerate neighbor retrieval.
7. Outliers and noisy labels: isolated observations can dominate small neighborhoods, and graph-based methods can propagate incorrect constraints.
8. Limited uncertainty calibration: vote proportions are not automatically calibrated probabilities, and abstention mechanisms require an explicit policy for trading coverage against error.
9. Hyperparameter proliferation: extensions introduce bandwidths, decay factors, graph depths, feature weights, radius thresholds, ensemble sizes, local derivative orders, or information-aggregation parameters.
10. Empirical non-universality: reported improvements are dataset- and representation-dependent. Across the cited studies, modified KNN methods often outperform selected baselines on particular datasets, but no single variant is established as uniformly optimal.

KNN is therefore best understood not as one fixed algorithm but as a family of local decision procedures defined by a metric, neighborhood rule, aggregation mechanism, and computational search strategy. Ordinary majority-vote KNN is the simplest member of this family. Gaussian and adaptive weighting alter the aggregation rule; manifold, spatial, and temporal methods alter the geometry; approximate forests and neighbor descent alter the search procedure; open-set and radius-based methods add abstention; neural and quantum variants change the representation and computational substrate. The central methodological requirement remains unchanged: the selected similarity relation must make local proximity informative for the prediction task.