Deep k-Nearest Neighbors (DkNN)

• DkNN is a hybrid deep learning method that combines neural network representations with kNN searches to support predictions with layer-wise training evidence.
• It improves upon standard DNNs by using conformal prediction for calibrated confidence and credibility, addressing issues like adversarial fragility and interpretability.
• Recent variants, such as DAEkNN and kNet, extend the framework to enhance adversarial robustness and scalability through adaptive defenses and parametric approximations.

Searching arXiv for the cited DkNN-related papers to ground the article in current bibliographic records. Deep k-Nearest Neighbors (DkNN) is a hybrid inference procedure that combines a trained deep neural network with $k$-nearest-neighbor retrieval in the network’s learned representation spaces. Instead of relying solely on the final parametric classifier, DkNN compares a test input to neighboring training points at one or multiple layers and uses the labels of those neighbors to support prediction, confidence estimation, and explanation (Papernot et al., 2018). In its original formulation, the method was introduced to address three recurrent deficiencies of standard deep models—unreliable confidence away from the training manifold, limited interpretability, and fragility to adversarial perturbations—by grounding decisions in layer-wise neighborhood structure and calibrating support with conformal prediction (Papernot et al., 2018). Subsequent work examined its robustness under adaptive white-box attack (Sitawarin et al., 2019), extended the framework to adversarially enhanced neighbor aggregation (Wang et al., 2021), and explored parametric approximations of deep-space $k$NN behavior through a learned network, termed kNet, that operates on penultimate-layer features (Mizrahi et al., 2021).

1. Origins and defining formulation

DkNN was introduced in "Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning" (Papernot et al., 2018). The central construction is to take a trained deep network $f$ and, for each layer $\ell$, use the representation $f_\ell(x)$ of an input $x$ as the basis for a nearest-neighbor query against stored training representations from the same layer (Papernot et al., 2018).

In the original formulation, for a test input $x$ and layer $\ell$, one retrieves the $k$ nearest training representations under angular distance and denotes their labels by $\Omega_\ell(x)$ (Papernot et al., 2018). DkNN then defines a label-wise nonconformity score by counting disagreements between a candidate label $k$0 and the labels of retrieved neighbors across layers:

$k$1

and

$k$2

A labeled calibration set, disjoint from the training and evaluation sets, is used to form empirical $k$3-values

$k$4

with prediction, confidence, and credibility defined by

$k$5

$k$6

and

$k$7

(Papernot et al., 2018).

This formulation distinguishes DkNN from a conventional single-layer $k$8NN classifier. The method is not merely majority voting on a final embedding; rather, its original version aggregates layer-wise evidence through conformal nonconformity and uses a calibration set to obtain empirical support scores (Papernot et al., 2018). A later paper, "Deep Adversarially-Enhanced k-Nearest Neighbors," operationalized DkNN differently, using per-layer class-support fractions

$k$9

and summing these across selected layers without conformal calibration (Wang et al., 2021). This establishes an important terminological distinction: the same label “DkNN” has been used both for the original conformal formulation and for simpler multi-layer neighbor-voting variants.

2. Representation-space retrieval and inference mechanics

The defining mechanism of DkNN is representation-space retrieval. The original paper stores, for each training sample $f$0 with label $f$1, the representation $f$2 at each analyzed layer $f$3 together with $f$4 (Papernot et al., 2018). At inference time, the test input is propagated through the trained network, and each selected layer issues a nearest-neighbor query in that layer’s representation space.

The original DkNN work uses cosine similarity, or equivalently angular distance, between representation vectors, with data-dependent locality-sensitive hashing for angular distance as the approximate nearest-neighbor backend (Papernot et al., 2018). The 2019 robustness analysis notes that, in its equations, distance is written as Euclidean,

$f$5

but that the implementation follows Papernot et al. and uses cosine distance by normalizing all representations to unit norm; with normalization, cosine distance is equivalent to Euclidean distance (Sitawarin et al., 2019). By contrast, the DAEkNN paper uses Euclidean distance in the layer-wise feature space by default (Wang et al., 2021). These descriptions are method-specific rather than contradictory: they reflect different instantiations of deep-space neighbor search in later work.

The original end-to-end DkNN pipeline consists of four stages: train a DNN normally; precompute and store layer-wise training representations and build one approximate nearest-neighbor index per layer; hold out a labeled calibration set and compute calibration nonconformities; then, for each test input, retrieve neighbors at each layer, compute nonconformity scores and empirical $f$6-values, and output prediction, confidence, credibility, and optionally neighbor exemplars (Papernot et al., 2018).

A concise comparison of the principal formulations described in the cited papers is given below.

Variant	Layer usage	Decision mechanism
Original DkNN	All hidden layers were used	Conformal nonconformity and empirical $f$7-values (Papernot et al., 2018)
DkNN in DAEkNN paper	Selected layers $f$8	Sum of per-layer class supports $f$9 (Wang et al., 2021)
kNet instantiation	Penultimate layer	Neural approximation of deep-space $\ell$0NN distribution (Mizrahi et al., 2021)

The practical cost of this design is explicit in the source material. DkNN requires storage of layer-wise training representations and per-layer index structures, and query time adds multiple nearest-neighbor searches to the underlying forward pass (Papernot et al., 2018, Sitawarin et al., 2019). This suggests that DkNN trades low-latency parametric inference for exemplar-based evidence and calibration.

3. Confidence, credibility, and exemplar-based interpretability

A defining contribution of the original DkNN paper is the use of conformal prediction to convert layer-wise neighbor agreement into calibrated notions of support (Papernot et al., 2018). In that framework, the distinction between confidence and credibility is explicit. Confidence is

$\ell$1

whereas credibility is the largest label-wise $\ell$2-value,

$\ell$3

Confidence reflects separation from alternative labels, while credibility reflects how well any label is supported by the training manifold as represented by the calibration nonconformities (Papernot et al., 2018).

The paper reports that reliability diagrams show softmax confidence concentrated at $\ell$4 for most test inputs, while DkNN credibility spreads across the full range (Papernot et al., 2018). On out-of-distribution inputs, DkNN credibility drops substantially relative to softmax confidence. The reported mean softmax versus DkNN credibility values are: MNIST $\ell$5 NotMNIST, $\ell$6 versus $\ell$7; MNIST rotated $\ell$8, $\ell$9 versus $f_\ell(x)$0; SVHN $f_\ell(x)$1 CIFAR-10, $f_\ell(x)$2 versus $f_\ell(x)$3; and SVHN rotated $f_\ell(x)$4, $f_\ell(x)$5 versus $f_\ell(x)$6 (Papernot et al., 2018). These results support the original claim that DkNN estimates the lack of support for inputs outside the model’s training manifold (Papernot et al., 2018).

Interpretability in DkNN is exemplar-based rather than feature-attribution-based. The method returns, per layer, the actual training examples whose representations are closest to the test input (Papernot et al., 2018). The original paper presents an ImageNet/ResNet case in which an image of Barack Obama throwing a football is classified as basketball; the nearest neighbors at the last hidden layer are mostly basketball players, and cropping the ball changes the prediction to racket with tennis-player neighbors (Papernot et al., 2018). The same paper also reports that high-credibility DkNN misclassifications on MNIST and SVHN surfaced samples with incorrect or ambiguous ground-truth labels, suggesting a route to dataset cleaning (Papernot et al., 2018).

A common misconception is to treat DkNN credibility as interchangeable with conventional softmax confidence. The original formulation rejects that equivalence: credibility is explicitly tied to empirical support from the training manifold through conformal calibration, whereas softmax confidence is not (Papernot et al., 2018). Another misconception is to regard DkNN explanations as post hoc saliency. In the cited work, the explanation mechanism is instead the retrieved neighbor set itself (Papernot et al., 2018).

4. Robustness claims and adversarial evaluation

DkNN was proposed partly as a response to adversarial fragility, with the argument that adversarial examples often lose consistent layer-wise support from the training set (Papernot et al., 2018). The original evaluation reports accuracy improvements over the base DNN on adversarial examples generated by FGSM, BIM, and CW. For MNIST, DNN versus DkNN accuracy is reported as $f_\ell(x)$7 versus $f_\ell(x)$8 on FGSM, $f_\ell(x)$9 versus $x$0 on BIM, and $x$1 versus $x$2 on CW; for SVHN, $x$3 versus $x$4, $x$5 versus $x$6, and $x$7 versus $x$8; for GTSRB, $x$9 versus $x$0, $x$1 versus $x$2, and $x$3 versus $x$4 (Papernot et al., 2018). The same paper also states that DkNN credibility is typically below $x$5 for most adversarial inputs unless the prediction is correct (Papernot et al., 2018).

However, the later paper "On the Robustness of Deep K-Nearest Neighbors" argues that evaluating DkNN requires adaptive attacks tailored to the non-differentiability of $x$6NN voting (Sitawarin et al., 2019). It introduces a gradient-based surrogate in which the hard inclusion of a point among the $x$7 nearest neighbors is approximated by a sigmoid threshold on distance. For DkNN, the multi-layer surrogate objective is

$x$8

subject to norm and box constraints (Sitawarin et al., 2019).

Under this adaptive white-box threat model on MNIST, the reported clean DkNN accuracy is $x$9, but the proposed gradient attack reduces accuracy to $\ell$0 under $\ell$1 with $\ell$2, to $\ell$3 under $\ell$4, and to $\ell$5 in a fixed-$\ell$6 setting with $\ell$7 (Sitawarin et al., 2019). The paper further reports that, as $\ell$8 increases in the $\ell$9 setting, DkNN accuracy under the gradient attack reaches $k$0 at $k$1 (Sitawarin et al., 2019). It also notes that thresholding credibility at $k$2 reduces clean accuracy to $k$3 while still allowing $k$4 of adversarial examples at $k$5 and $k$6 at $k$7 to pass (Sitawarin et al., 2019).

These results establish a substantive controversy in the DkNN literature. The original work shows that DkNN can lower credibility on unsupported inputs and improve adversarial handling relative to the underlying classifier (Papernot et al., 2018). The later attack paper shows that these properties are insufficient against fully adaptive, white-box adversaries (Sitawarin et al., 2019). A plausible implication is that DkNN’s robustness claims are best interpreted as relative and threat-model-dependent rather than absolute.

5. Adversarially enhanced variants

"Deep Adversarially-Enhanced k-Nearest Neighbors" proposes DAEkNN as a variant intended to mitigate the robustness-accuracy trade-off observed when DkNN is applied across deeper layers (Wang et al., 2021). The paper states that, on a standard-trained model, adversarial accuracy under PGD drops dramatically as the layer goes deeper while standard accuracy monotonically increases with depth, yielding a pronounced trade-off (Wang et al., 2021).

DAEkNN modifies DkNN in two ways. First, it uses an adversarially trained backbone network. Second, it predicts by combining, per layer, a benign $k$8NN built from $k$9 and an adversarial $\Omega_\ell(x)$0NN built from an adversarially hardened training set $\Omega_\ell(x)$1 generated by PGD:

$\Omega_\ell(x)$2

Per-layer class supports are $\Omega_\ell(x)$3 and $\Omega_\ell(x)$4, and the two are aggregated with distance-based softmax weights $\Omega_\ell(x)$5 and $\Omega_\ell(x)$6 satisfying $\Omega_\ell(x)$7 (Wang et al., 2021). The final decision rule is

$\Omega_\ell(x)$8

The reported evaluations use MNIST with a three-convolutional-layer CNN backbone and CIFAR-10 with VGG16, Euclidean distance in feature space, MNIST layers $\Omega_\ell(x)$9 and $k$00, and CIFAR-10 layers $k$01 and $k$02 (Wang et al., 2021). Under $k$03-step PGD with $k$04 norm, evaluation attack powers are $k$05 for MNIST and $k$06 for CIFAR-10, both on pixel scale $k$07 (Wang et al., 2021). Table 1 in the paper reports the following:

Dataset / method	SA	AA
MNIST DkNN	98.78%	0.06%
MNIST DAEkNN	97.90%	68.26%
CIFAR-10 DkNN	91.78%	0.06%
CIFAR-10 DAEkNN	78.44%	50.48%

The same table gives harmonic mean values of $k$08 for DkNN on both MNIST and CIFAR-10, versus $k$09 and $k$10 for DAEkNN respectively (Wang et al., 2021). The paper’s leave-one-out ablation on CIFAR-10 reports that DAEkNN-WAT achieves SA $k$11, AA $k$12, and HM $k$13; DAEkNN-WAD achieves SA $k$14, AA $k$15, and HM $k$16; and full DAEkNN achieves SA $k$17, AA $k$18, and HM $k$19 (Wang et al., 2021).

This line of work reframes DkNN not as a complete defense, but as a component whose behavior depends strongly on the robustness of the underlying feature extractor and on how the reference distribution is constructed. The paper explicitly notes that broader threat models and mixtures of hardening strengths and attack types are future directions (Wang et al., 2021).

6. Efficiency, scalability, and parametric approximation

A recurrent practical limitation of DkNN is storage and retrieval overhead. The original formulation requires storing representations of all training points at each selected layer and building one approximate nearest-neighbor index per layer (Papernot et al., 2018). The 2019 robustness analysis restates that storage scales with $k$20 for precomputed representations, and query time requires $k$21 approximate nearest-neighbor searches plus aggregation (Sitawarin et al., 2019). The DAEkNN extension increases this burden further because it stores deep features for both $k$22 and $k$23, effectively doubling storage relative to DkNN (Wang et al., 2021).

The paper "kNet: A Deep kNN Network To Handle Label Noise" addresses precisely these deployment costs, although it does so in the context of label-noise robustness rather than the original conformal DkNN setting (Mizrahi et al., 2021). Its abstract states that exact $k$24NN requires a huge memory footprint to store all training samples and an advanced data structure for retrieval, and proposes a neural network, termed kNet, that learns to perform $k$25NN so that, once trained, one no longer needs to store the training data and processing a query sample is a simple matter of inference (Mizrahi et al., 2021).

In the detailed material provided for that paper, kNet is described as a network $k$26 trained on penultimate-layer embeddings $k$27 from a preliminary network $k$28, with targets given by a soft $k$29NN label distribution in deep feature space (Mizrahi et al., 2021). The target distribution is

$k$30

and the predictor is

$k$31

trained by cross-entropy to the $k$32NN soft targets (Mizrahi et al., 2021). The source states that kNet gives a smooth approximation of $k$33NN and cannot handle the sharp label changes between samples that $k$34NN can exhibit, indicating that currently kNet is best suited to approximate $k$35NN with a fairly large $k$36 (Mizrahi et al., 2021). The abstract reports that, in practice, kNet consistently improve the results of all preliminary networks, in all label noise regimes, by up to $k$37 (Mizrahi et al., 2021).

This relationship to DkNN is conceptually important. The original DkNN obtains interpretability and calibration by explicit retrieval from the training set at inference time (Papernot et al., 2018). kNet instead distills deep-space neighbor behavior into parametric weights, removing training-set storage and neighbor search at deployment (Mizrahi et al., 2021). A plausible implication is that such an approximation gains efficiency while trading away the exemplar lists and conformal support structure that make DkNN interpretable in the original sense.

7. Position within the broader deep-neighbor literature

Within the cited works, DkNN occupies a distinct position at the intersection of deep representation learning, conformal prediction, neighbor-based reasoning, and adversarial robustness. The original paper explicitly frames it as a hybrid classifier that combines the $k$38-nearest-neighbors algorithm with representations learned by each layer of a DNN (Papernot et al., 2018). The method therefore differs from ordinary deep classifiers by replacing sole reliance on the final linear head with training-set support checks in learned representation space.

The later literature shows two main directions of reinterpretation. One direction emphasizes robustness engineering: DAEkNN replaces the original conformal machinery with a weighted combination of benign and adversarial neighbor supports built on an adversarially trained backbone (Wang et al., 2021). The other emphasizes deployability: kNet learns a parametric approximation to deep-space neighbor voting so that inference requires only a forward pass (Mizrahi et al., 2021).

At the same time, the 2019 attack analysis demonstrates that DkNN is not a provable defense and that adaptive attacks can manipulate layer-wise neighbor composition, particularly in later layers (Sitawarin et al., 2019). This limits any strong claim that neighbor-based deep inference is intrinsically robust. What the cited literature does support is narrower: DkNN provides a principled mechanism for tying predictions to retrieved training evidence and, in its original formulation, for expressing support through conformal confidence and credibility (Papernot et al., 2018); its adversarial behavior, however, depends on attack adaptivity, representation quality, and the exact inference variant under consideration (Sitawarin et al., 2019, Wang et al., 2021).

Taken together, these papers define DkNN less as a single fixed algorithm than as a family of deep-space nearest-neighbor inference schemes. The original version is characterized by layer-wise retrieval, nonconformity, and conformal calibration (Papernot et al., 2018). Subsequent variants retain the central idea of neighbor support in learned representations while altering aggregation rules, robustness mechanisms, or computational realization (Wang et al., 2021, Mizrahi et al., 2021).