Weighted Distance NN Condensing

• Weighted Distance Nearest-Neighbor Condensing is a framework that uses learned positive weights to reduce dataset size in nearest-neighbor classification.
• It achieves dramatic data compression while maintaining or improving accuracy, offering theoretical guarantees similar to classical condensing methods.
• A greedy heuristic is commonly used to select representatives, though challenges remain in computational efficiency and approximation guarantees.

Weighted Distance Nearest-Neighbor Condensing (WNN) is a framework for data reduction in nearest-neighbor classification, generalizing traditional condensing by introducing learned, positive weights on representatives of the instance space. By leveraging weighted distances in the nearest-neighbor query, WNN enables more compact summarization of labeled datasets while preserving (or even improving) classification accuracy, and it achieves generalization performance guarantees essentially identical to those of standard nearest-neighbor condensing (Gottlieb et al., 2023).

1. Formal Foundations and Problem Statement

Let $(\mathcal X,d)$ be a separable metric space and $$S = \{(x_i, y_i)\}_{i=1}^n \subset \mathcal X \times \{-1,1\}$$ a labeled sample. WNN condensing selects a subset $\tilde S \subseteq S$ (the condensed set) and assigns to each $x\in\tilde S$ a positive weight $w(x) \in (0, \infty)$, with the convention $w(x)=1$ for $x\not\in \tilde S$. Classification of a query $q$ employs the weighted distance: $\tilde d(q,s) = \frac{d(q,s)}{w(s)}$ for $s \in \tilde S$. The WNN classifier is

$$S = \{(x_i, y_i)\}_{i=1}^n \subset \mathcal X \times \{-1,1\}$$0

A pair $$S = \{(x_i, y_i)\}_{i=1}^n \subset \mathcal X \times \{-1,1\}$$1 forms a consistent WNN condensing if

$$S = \{(x_i, y_i)\}_{i=1}^n \subset \mathcal X \times \{-1,1\}$$2

The weighted condensing problem seeks such a pair minimizing $$S = \{(x_i, y_i)\}_{i=1}^n \subset \mathcal X \times \{-1,1\}$$3.

This construction generalizes unweighted nearest-neighbor condensing: setting $$S = \{(x_i, y_i)\}_{i=1}^n \subset \mathcal X \times \{-1,1\}$$4 recovers the classical framework, while optimized weights may enable dramatically smaller condensed sets.

2. Theoretical Analysis and Generalization Properties

Separation of Representation Power

WNN provides potentially exponential gains in reduction compared to the unweighted regime. For any $$S = \{(x_i, y_i)\}_{i=1}^n \subset \mathcal X \times \{-1,1\}$$5, there exist datasets $$S = \{(x_i, y_i)\}_{i=1}^n \subset \mathcal X \times \{-1,1\}$$6 such that:

• The minimal consistent unweighted subset must be of size $$S = \{(x_i, y_i)\}_{i=1}^n \subset \mathcal X \times \{-1,1\}$$7.
• A consistent WNN condensed set of size $$S = \{(x_i, y_i)\}_{i=1}^n \subset \mathcal X \times \{-1,1\}$$8 suffices, given appropriate weights.

Typical instances include interleaved structures (“banana-shaped” clusters), where unweighted condensing cannot avoid including all points but weighted condensing, by boosting endpoints, can effect a succinct partition (Gottlieb et al., 2023).

Generalization Bounds

For empirically consistent $$S = \{(x_i, y_i)\}_{i=1}^n \subset \mathcal X \times \{-1,1\}$$9 with $\tilde S \subseteq S$0 and sample size $\tilde S \subseteq S$1, the probability of error on unseen data satisfies: $\tilde S \subseteq S$2 with probability at least $\tilde S \subseteq S$3. A permutation-invariant reconstruction sharpens this to: $\tilde S \subseteq S$4 These bounds closely track those for unweighted condensing, confirming that the generalization guarantees of WNN are essentially identical to the classical case (Gottlieb et al., 2023).

3. Greedy Condensing Heuristic and Algorithmic Aspects

The main practical approach is a greedy algorithm analogous to set cover:

• Let $\tilde S \subseteq S$5 denote the minimal distance from $\tilde S \subseteq S$6 to any opposite-label point,
• At each iteration, select $\tilde S \subseteq S$7 maximizing the number of same-label points within $\tilde S \subseteq S$8 in $\tilde S \subseteq S$9, the set of points not yet covered,
• Add $x\in\tilde S$0 to $x\in\tilde S$1 with $x\in\tilde S$2, remove all points of the same label within this ball from $x\in\tilde S$3,
• Iterate until all points are covered.

Formally, each iteration solves: $x\in\tilde S$4 subject to $x\in\tilde S$5.

In complexity, a naive implementation needs $x\in\tilde S$6 work for $x\in\tilde S$7 points, but distance maintenance and spatial indexing can reduce this to $x\in\tilde S$8 or better in practice (Gottlieb et al., 2023).

4. Bayes Consistency

Let $x\in\tilde S$9 denote the minimal distance to an opposite-label point (outside the condensed set, set $w(x) \in (0, \infty)$0). Consider the (computationally hard) minimal consistent condensed set: $w(x) \in (0, \infty)$1 Under mild conditions—separable $w(x) \in (0, \infty)$2, atomless $w(x) \in (0, \infty)$3 distribution, piecewise-continuous Bayes function $w(x) \in (0, \infty)$4—the classifier $w(x) \in (0, \infty)$5 achieves risk converging to Bayes risk zero almost surely as $w(x) \in (0, \infty)$6.

Furthermore, under mild tail assumptions on the metric distribution (e.g., bounded or Gaussian), the greedy WNN condensing heuristic is also Bayes-consistent. The size of its output is at most an $w(x) \in (0, \infty)$7 factor larger than the optimal, so the generalization excess vanishes (Gottlieb et al., 2023).

5. Empirical Results: Compression and Accuracy

Experiments compare WNN to MSS (modified selected subset), RSS (recent selective subset), and optimal unweighted integer-programming condensing (IP). Condensed set sizes are summarized as follows:

Dataset	Points	MSS	RSS	IP (opt. NN)	WNN
Circle	200	52	45	7	12
Banana	200	74	66	32	35
Iris	100	11	9	2	4

WNN substantially out-compresses classical heuristics (MSS, RSS) and approaches the performance of optimal unweighted condensing.

On the large-scale notMNIST dataset ($w(x) \in (0, \infty)$8 points, 10 classes, UMAP-reduced features), WNN approaches the test-set error of uncompressed 1-NN, with a compression ratio of $w(x) \in (0, \infty)$9 (retaining only $w(x)=1$0 of samples), while MSS and RSS either compress slightly better or worse with greater accuracy degradation (Gottlieb et al., 2023).

6. Limitations, Open Problems, and Future Directions

The greedy WNN heuristic lacks a constant-factor approximation guarantee and may be improved with more sophisticated approximation strategies or by leveraging advanced geometric data structures. Computational costs remain considerable for very large datasets without specialized acceleration.

Open problems include the computational complexity and hardness of weighted condensing, improved algorithms (e.g., guaranteed $w(x)=1$1-approximation), and extension to multi-class problems or broader classes of weight-dependent metric functions.

Future work involves faster nearest-neighbor search structures for weighted distances, alternative or enhanced condensing heuristics with provable properties, and integration with metric learning to adapt or optimize the base distance $w(x)=1$2 in tandem with the condensing process (Gottlieb et al., 2023).

Weighted Distance Nearest-Neighbor Condensing thus strictly generalizes classical NN condensing, combines strong theoretical guarantees with dramatic reductions in storage, and presents numerous directions for continued research into scalable, high-fidelity, label-preserving data summarization.