Distance-to-Closest-Record Filtering

• Distance-to-Closest-Record Filtering is a design principle where records are evaluated and selected based on their computed distance to the nearest admissible or representative record using various distance metrics.
• It is applied across domains such as vector search, record linkage, image denoising, phylogenetics, and synthetic data privacy auditing to balance accuracy and computational efficiency.
• Methodologies often integrate domain-specific penalties, dynamic programming, or aggregation strategies to address challenges like filter compliance, noise reduction, and redundancy control.

Searching arXiv for the cited papers to ground the article in the literature. arXiv search query: (Sutradhar et al., 6 Nov 2025) Distance-to-Closest-Record Filtering denotes a family of selection, ranking, and pruning strategies in which the decisive quantity is a distance to a nearest admissible, neighboring, or representative record. In the literature, this idea appears in several technically distinct forms: as a filtered nearest-neighbor ranking criterion in vector search, as an inclusion rule after blocking in probabilistic record linkage, as a minimal-neighbor statistic in reduced-ordering vector filters, as an average-distance-to-nearest-reference objective on phylogenetic trees, as a vulnerability score based on nearest neighbors in synthetic-data privacy auditing, and as a separation constraint in nearest-neighbor condensation (Sutradhar et al., 6 Nov 2025, Murray, 2016, Celebi, 2010, Matsen et al., 2012, Meeus et al., 2023, Flores-Velazco, 2020).

1. Formal characterizations

A common abstraction is to define, for each object, a nearest feasible or nearest retained counterpart and to use that quantity either as a score or as a filter. In record linkage after blocking, a generic DCR-type inclusion rule keeps a pair $(i,j)$ if it is sufficiently close to the closest observed distance for $i$, for example

$$I_{ij} = 1\{ d(x_i,x_j) \le f(d_i^{\min}) \}, \qquad d_i^{\min} = \min_{j} d(x_i,x_j),$$

or, in a top-$k$ form, $I_{ij} = 1\{ j \in \mathsf{NN}_k(i)\}$ (Murray, 2016).

In filtered ANN search with discrete labels, the closest-record notion is made explicit through a learned filter-aware distance

$$D_f(q, x; w_m) = d(q, x) + w_m \cdot (1 - m(S, L(x))),$$

where $m(S, L(x)) = |S \cap L(x)| / |S|$ measures filter compliance. Feasible records satisfy $S \subseteq L(x)$ and therefore incur no penalty, whereas infeasible records incur an additive penalty proportional to their normalized mismatch fraction (Sutradhar et al., 6 Nov 2025).

In reduced-ordering vector filtering for color images, the decisive statistic can be the minimal neighbor distance

$d_{\min}(x_k) = \min_{i \neq k} d(x_k, x_i),$

with output rule

$x_* = \arg\min_k d_{\min}(x_k).$

This realizes DCR as a local neighborhood operator: the selected vector is the sample whose nearest neighbor is as close as possible (Celebi, 2010).

In phylogenetic subset selection, the closest-record quantity is averaged over a demand distribution. For a selected leaf set $i$0,

$i$1

and the optimization problem is to choose $i$2 leaves minimizing the average distance to the closest selected leaf (Matsen et al., 2012).

In privacy auditing for synthetic data, the same pattern appears as a nearest-neighbor isolation score,

$i$3

where $i$4 are ordered by increasing distance from $i$5. Larger $i$6 indicates that a record is more isolated from its neighbors in the real dataset (Meeus et al., 2023).

These formulations show that “closest record” need not mean a literal top-1 Euclidean neighbor. Depending on the domain, the closest object may be feasible under filters, closest under a set distance, closest inside a local image window, closest selected leaf in a tree metric, or closest among encoded mixed-type tabular records. This suggests that Distance-to-Closest-Record Filtering is best understood as a design principle rather than a single algorithmic template.

2. Filter-aware nearest-neighbor search

In filtered ANN search, the task is to retrieve the $i$7 nearest vectors to a query $i$8 subject to an AND-style filter requirement $i$9. The formulation in "Learning Filter-Aware Distance Metrics for Nearest Neighbor Search with Multiple Filters" replaces fixed penalties with a learned scalar $$I_{ij} = 1\{ d(x_i,x_j) \le f(d_i^{\min}) \}, \qquad d_i^{\min} = \min_{j} d(x_i,x_j),$$0 in the additive distance

$$I_{ij} = 1\{ d(x_i,x_j) \le f(d_i^{\min}) \}, \qquad d_i^{\min} = \min_{j} d(x_i,x_j),$$1

so that each missing label contributes $$I_{ij} = 1\{ d(x_i,x_j) \le f(d_i^{\min}) \}, \qquad d_i^{\min} = \min_{j} d(x_i,x_j),$$2 to the effective distance. During index construction, an analogous asymmetric pairwise form

$$I_{ij} = 1\{ d(x_i,x_j) \le f(d_i^{\min}) \}, \qquad d_i^{\min} = \min_{j} d(x_i,x_j),$$3

encourages edges where $$I_{ij} = 1\{ d(x_i,x_j) \le f(d_i^{\min}) \}, \qquad d_i^{\min} = \min_{j} d(x_i,x_j),$$4’s labels cover most of $$I_{ij} = 1\{ d(x_i,x_j) \le f(d_i^{\min}) \}, \qquad d_i^{\min} = \min_{j} d(x_i,x_j),$$5’s labels while preserving geometric proximity (Sutradhar et al., 6 Nov 2025).

The weight is learned by a constrained linear program with slack. For each training query, positives are exact top-$$I_{ij} = 1\{ d(x_i,x_j) \le f(d_i^{\min}) \}, \qquad d_i^{\min} = \min_{j} d(x_i,x_j),$$6 unfiltered neighbors that satisfy the filter, and negatives are geometrically closer points that violate the filter. The optimization minimizes

$$I_{ij} = 1\{ d(x_i,x_j) \le f(d_i^{\min}) \}, \qquad d_i^{\min} = \min_{j} d(x_i,x_j),$$7

subject to ranking constraints with a margin $$I_{ij} = 1\{ d(x_i,x_j) \le f(d_i^{\min}) \}, \qquad d_i^{\min} = \min_{j} d(x_i,x_j),$$8; in the reported experiments, $$I_{ij} = 1\{ d(x_i,x_j) \le f(d_i^{\min}) \}, \qquad d_i^{\min} = \min_{j} d(x_i,x_j),$$9. Because positives have zero filter penalty, the program learns the smallest $k$0 that separates feasible neighbors from closer violating neighbors except where slack absorbs unavoidable violations. A useful derived bound is

$k$1

for a feasible $k$2 and infeasible $k$3 to ensure that the feasible item outranks the violating one under $k$4.

The learned distance is integrated into graph construction and search. The index build uses greedy expansion and robust prune steps keyed by $k$5, and query-time graph search maintains a min-heap prioritized by $k$6. Query planning routes highly selective queries, estimated satisfying set $k$7, to a brute-force pass over the feasible subset using inverted indices; otherwise graph search with $k$8 is used. Complexity is dominated by the number of graph expansions; per expansion the method computes one base distance and a constant-time label mismatch via bitset operations or precomputed inverted indices.

The reported empirical results are on YFCC1M and Wikipedia-35M. YFCC1M uses 1M CLIP image embeddings (192-D) with metadata labels and 1,727 evaluation queries; the learned weight is $k$9. Wikipedia-35M uses 35M sentence embeddings (768-D) with 464 evaluation queries; the learned weight is $I_{ij} = 1\{ j \in \mathsf{NN}_k(i)\}$0. Across both datasets, using $I_{ij} = 1\{ j \in \mathsf{NN}_k(i)\}$1 in both build and search increases Recall@10 by 5–10% over fixed-penalty and post-filter baselines at similar or lower distance-comparison counts, while Figure 1 shows comparable unfiltered Recall@10 and distance comparisons when filters are omitted. The implementation details reported are cosine similarity for both datasets, bitset intersections for $I_{ij} = 1\{ j \in \mathsf{NN}_k(i)\}$2, PuLP for the LP, and $I_{ij} = 1\{ j \in \mathsf{NN}_k(i)\}$3 tuned by grid search on a validation split (Sutradhar et al., 6 Nov 2025).

Within this framework, “Distance-to-Closest-Record Filtering” is realized by an effective distance to the closest feasible record. For feasible $I_{ij} = 1\{ j \in \mathsf{NN}_k(i)\}$4, $I_{ij} = 1\{ j \in \mathsf{NN}_k(i)\}$5; for infeasible $I_{ij} = 1\{ j \in \mathsf{NN}_k(i)\}$6,

$I_{ij} = 1\{ j \in \mathsf{NN}_k(i)\}$7

With sufficiently large $I_{ij} = 1\{ j \in \mathsf{NN}_k(i)\}$8, the nearest neighbor under $I_{ij} = 1\{ j \in \mathsf{NN}_k(i)\}$9 must lie in the feasible set $$D_f(q, x; w_m) = d(q, x) + w_m \cdot (1 - m(S, L(x))),$$0.

3. Record linkage and conditional filtering

Record linkage uses DCR ideas at both the candidate-generation and final-matching stages. In "Hausdorff Distance-Based Record Linkage for Improved Matching of Households and Individuals in Different Databases", the household-level distance between two households $$D_f(q, x; w_m) = d(q, x) + w_m \cdot (1 - m(S, L(x))),$$1 and $$D_f(q, x; w_m) = d(q, x) + w_m \cdot (1 - m(S, L(x))),$$2 is the symmetric Hausdorff distance

$$D_f(q, x; w_m) = d(q, x) + w_m \cdot (1 - m(S, L(x))),$$3

where the individual-level distance is a non-negative weighted sum of attribute distances,

$$D_f(q, x; w_m) = d(q, x) + w_m \cdot (1 - m(S, L(x))),$$4

For categorical attributes the distance is $$D_f(q, x; w_m) = d(q, x) + w_m \cdot (1 - m(S, L(x))),$$5 for agreement and $$D_f(q, x; w_m) = d(q, x) + w_m \cdot (1 - m(S, L(x))),$$6 otherwise; for ANASC (year of birth) the distance is $$D_f(q, x; w_m) = d(q, x) + w_m \cdot (1 - m(S, L(x))),$$7. The household match probability is modeled as

$$D_f(q, x; w_m) = d(q, x) + w_m \cdot (1 - m(S, L(x))),$$8

with $$D_f(q, x; w_m) = d(q, x) + w_m \cdot (1 - m(S, L(x))),$$9 enforcing monotonicity. A household $m(S, L(x)) = |S \cap L(x)| / |S|$0 is matched to the household $m(S, L(x)) = |S \cap L(x)| / |S|$1 with the largest estimated probability provided it exceeds a threshold $m(S, L(x)) = |S \cap L(x)| / |S|$2, which is calibrated during training to align the estimated proportion of matched households with the known true proportion (Menezes et al., 2024).

Within matched households, the paper fits a penalized logistic regression for individual pairs,

$m(S, L(x)) = |S \cap L(x)| / |S|$3

with ridge regularization estimated via glmnet and $m(S, L(x)) = |S \cap L(x)| / |S|$4 tuned by cross-validation. Final one-to-one matches are obtained from a linear program maximizing $m(S, L(x)) = |S \cap L(x)| / |S|$5 subject to one-to-one constraints and household-average probability thresholds. On the Italian SHIW application, the datasets contain 19,366 individuals and 8,156 households in 2014, 16,462 individuals and 7,420 households in 2016, and 15,198 individuals and 6,239 households in 2020. In external validation, hhlink outperforms fastLink at the individual level: for 2014–2016, hhlink reports $m(S, L(x)) = |S \cap L(x)| / |S|$6, $m(S, L(x)) = |S \cap L(x)| / |S|$7, $m(S, L(x)) = |S \cap L(x)| / |S|$8, $m(S, L(x)) = |S \cap L(x)| / |S|$9, and Recall $S \subseteq L(x)$0, whereas fastLink reports $S \subseteq L(x)$1, $S \subseteq L(x)$2, $S \subseteq L(x)$3, $S \subseteq L(x)$4, and Recall $S \subseteq L(x)$5; analogous gains appear for 2016–2020 (Menezes et al., 2024).

The probabilistic-record-linkage literature also treats DCR explicitly as a filtering event that must be modeled conditionally. "Probabilistic Record Linkage and Deduplication after Indexing, Blocking, and Filtering" distinguishes indexing, blocking, and filtering, and places DCR squarely in the filtering stage. With a blocking indicator $S \subseteq L(x)$6 and DCR inclusion indicator $S \subseteq L(x)$7, inference is performed on the selected event $S \subseteq L(x)$8. After such filtering, the Fellegi–Sunter mixture must be conditioned on selection: $S \subseteq L(x)$9 A central result is that filtering can change the support of the comparison patterns, producing structural zeros, and can also change likelihood-ratio weight rankings when the filter is not a deterministic function of the comparison vector. The paper therefore recommends conditional quasi-independence with structural zeros and EM estimation on the retained pairs (Murray, 2016).

A recurrent misconception is that nearest-neighbor filtering in record linkage is merely a computational preprocessing step. The conditional analysis shows that, after DCR-type pruning, the statistical model itself changes: estimated match proportions, support, and error-rate interpretations are all conditional on having passed the filter.

4. Reduced-ordering vector filters in image processing

In color image denoising, DCR appears as a local ordering statistic. "Distance Measures for Reduced Ordering Based Vector Filters" studies nonlinear, order-statistics-based vector filters on RGB pixels. Given a window $d_{\min}(x_k) = \min_{i \neq k} d(x_k, x_i),$0, classical reduced ordering uses the aggregate statistic

$d_{\min}(x_k) = \min_{i \neq k} d(x_k, x_i),$1

and selects the vector minimizing $d_{\min}(x_k) = \min_{i \neq k} d(x_k, x_i),$2. The DCR alternative replaces the aggregate with the minimal-neighbor statistic

$d_{\min}(x_k) = \min_{i \neq k} d(x_k, x_i),$3

with an optional $d_{\min}(x_k) = \min_{i \neq k} d(x_k, x_i),$4-nearest-neighbor variant

$d_{\min}(x_k) = \min_{i \neq k} d(x_k, x_i),$5

The interpretation given is that true neighborhood samples tend to have at least one nearby similar sample, whereas impulses lack close neighbors and thus have larger $d_{\min}(x_k) = \min_{i \neq k} d(x_k, x_i),$6 (Celebi, 2010).

The paper evaluates 18 ordering criteria, including $d_{\min}(x_k) = \min_{i \neq k} d(x_k, x_i),$7, $d_{\min}(x_k) = \min_{i \neq k} d(x_k, x_i),$8, $d_{\min}(x_k) = \min_{i \neq k} d(x_k, x_i),$9, squared Euclidean, cosine angle, chord distance, divergence coefficient, Bray–Curtis, Canberra, Soergel, Ware–Hedges, Goude distance, and several fuzzy similarities convertible to distances via $x_* = \arg\min_k d_{\min}(x_k).$0 or $x_* = \arg\min_k d_{\min}(x_k).$1. The combined fuzzy similarity with spatial proximity is

$x_* = \arg\min_k d_{\min}(x_k).$2

Filtering is performed in RGB, while CIELAB is used for the NCD evaluation measure.

The experimental setup uses 100 high-quality RGB images, correlated impulsive noise at 10%, 20%, and 30%, and a $x_* = \arg\min_k d_{\min}(x_k).$3 window. Reported metrics are MAE, MSE, NCD, and CPU time. The principal findings are that $x_* = \arg\min_k d_{\min}(x_k).$4, $x_* = \arg\min_k d_{\min}(x_k).$5, and $x_* = \arg\min_k d_{\min}(x_k).$6 are the top performers overall; among traditional distances, $x_* = \arg\min_k d_{\min}(x_k).$7 and the divergence coefficient outperform $x_* = \arg\min_k d_{\min}(x_k).$8; direction-only measures such as cosine angle perform poorly; and $x_* = \arg\min_k d_{\min}(x_k).$9 is among the worst in effectiveness when used in the closest-to-mean reduced-ordering approximation. The paper reports overall mean effectiveness ranks of approximately $i$00 for $i$01, $i$02 for $i$03, $i$04 for $i$05, $i$06 for $i$07, $i$08 for divergence, and $i$09 for $i$10. For a 512×512 image with $i$11, classical reduced ordering requires more than 9.4 million distance evaluations; DCR has the same asymptotic per-window cost $i$12 because it still requires the pairwise distance matrix (Celebi, 2010).

The image-processing literature therefore treats closest-record filtering as a robustness device against impulsive outliers. At the same time, the paper notes that in textured or multimodal neighborhoods, $i$13 can be unstable, and $i$14 or a secondary aggregate statistic can mitigate that instability.

5. Representative subsets, tree metrics, and separation constraints

On phylogenetic trees, DCR becomes a global subset-selection objective. "Minimizing the average distance to a closest leaf in a phylogenetic tree" formalizes the problem of choosing $i$15 leaves that minimize the average distance from all demand mass to the nearest selected leaf. The objective is ADCL: $i$16 in the continuous case, or a weighted discrete analogue over mass points. The paper shows that a natural greedy deletion rule is not effective, that a variant of Partitioning Around Medoids (PAM) can get stuck in local minima, and that tree additivity enables an exact dynamic program based on bubble partitions, root boundary conditions, and lower envelopes of affine subwork lines. The exact program returns solutions for all numbers of leaves less than or equal to the target number, whereas PAM returns only a solution for the pre-specified number of leaves. On real data, ADCL minimization chooses chimeric sequences less often than random subsets, while phylogenetic diversity maximization chooses them more often than random (Matsen et al., 2012).

A different use of closest-record filtering appears in nearest-neighbor condensation. "Social Distancing is Good for Points too!" studies the problem of finding a small consistent subset $i$17 for nearest-neighbor classification. The FCNN heuristic can behave poorly when points are too close to each other. The paper’s modification, SFCNN, changes batch addition to one-add-per-iteration and thereby enforces a scale-adaptive separation condition. For

$i$18

any two points $i$19 satisfy

$i$20

This “social distancing” prevents the pathological selection of many arbitrarily close representatives. In doubling metrics with constant doubling dimension,

$i$21

where $i$22 is the number of distinct nearest-enemy points and $i$23 is the smallest nearest-enemy distance. Empirically, SFCNN’s runtime and selected subset size are equivalent to FCNN across the reported datasets, while enabling provable upper bounds (Flores-Velazco, 2020).

These two lines of work use closest-record quantities in opposite directions. ADCL minimizes the average distance from all demand points to a retained subset, whereas SFCNN filters out excessively close additions to control redundancy. A plausible implication is that DCR methods can serve both coverage and sparsification, depending on whether the objective is representativeness or exclusion.

6. Vulnerable-record identification in synthetic data

In synthetic-data privacy auditing, DCR is used to identify records most likely to be exposed by membership inference attacks. "Achilles' Heels: Vulnerable Record Identification in Synthetic Data Publishing" defines a mixed-type distance space by one-hot encoding categorical attributes and min–max normalizing continuous attributes. The main distance is a generalized cosine distance over the categorical and continuous parts, weighted by the fraction of attributes of each type, and the vulnerability score is the average $i$24-nearest-neighbor distance

$i$25

Records are ranked by decreasing $i$26, and the paper selects the top-$i$27 records with $i$28; ties at the boundary are randomly broken. The default auditing setting uses $i$29 (Meeus et al., 2023).

The paper evaluates this procedure on UK Census (569,741 records, 17 categorical columns) and Adult (48,842 records, 15 columns: 9 categorical and 6 continuous), using SynthPop, BayNet, and PrivBayes. The query-based MIA uses shadow modeling with a random forest on $i$30-way marginal counting queries, $i$31 random queries, 100 trees, and maximum depth 10. A second evaluation uses a target-attention attack. For Adult, the protocol uses $i$32, $i$33, $i$34, $i$35, and synthetic release size $i$36; for UK Census, the corresponding auxiliary and test sizes are 50,000 and 25,000 (Meeus et al., 2023).

Distance-based selection consistently outperforms Random, Rare value, and Log-likelihood baselines. For the query-based attack, UK Census with Synthpop reports mean AUCs $i$37 for Random, $i$38 for Rare value, $i$39 for Log-likelihood, and $i$40 for Distance; UK Census with BayNet reports $i$41, $i$42, $i$43, and $i$44, respectively. The paper states that distance-based selection increases AUC by 7.2 percentage points on average versus baselines for the query-based attack and by 5.2 percentage points on average for the target-attention attack. The method is reported to be robust across $i$45 and across cosine and Minkowski distances with $i$46, with slightly weaker performance for higher $i$47. Under PrivBayes, attack AUCs drop as $i$48 decreases, and the paper reports that MIAs fail when $i$49 (Meeus et al., 2023).

Here, closest-record filtering is not used to recover matches or denoise images, but to identify isolated real records. The paper’s interpretation is that more isolated records are more vulnerable because generators may need to memorize rare or atypical examples to reproduce local structure or maintain utility.

7. Recurrent assumptions, trade-offs, and misconceptions

Across domains, DCR methods rely on assumptions about what nearest-neighbor structure captures. In filtered ANN, the learned scalar $i$50 is intended to adapt to label selectivity, with scarce feasible neighbors yielding larger $i$51 and abundant feasibility yielding smaller $i$52 (Sutradhar et al., 6 Nov 2025). In household linkage, the symmetric Hausdorff distance is sensitive to the worst nearest-neighbor relation, which the paper notes can be desirable for strict matching but can also be misleading when households split, merge, or change substantially (Menezes et al., 2024). In probabilistic record linkage after filtering, the retained comparison patterns may occupy only a strict subset of the original support, so post-filter inference is inherently conditional and can exhibit structural zeros (Murray, 2016).

The literature also repeatedly emphasizes precision–recall and robustness–cost trade-offs. Stricter household thresholds $i$53 improve precision and reduce computational cost at the expense of recall (Menezes et al., 2024). DCR filtering in probabilistic linkage improves computational efficiency and often increases the match proportion among retained pairs, but it can induce selection bias and recall loss by discarding true matches whose nearest-neighbor distance is large or poorly measured (Murray, 2016). In image filtering, $i$54 can be too local in multimodal windows, whereas $i$55 increases robustness (Celebi, 2010). In condensation, larger separation improves compression guarantees but can break consistency if pushed beyond the local margin $i$56 (Flores-Velazco, 2020).

A frequent misconception is that closest-record filtering is always a hard top-1 rule. The surveyed methods contradict that view. The filtered ANN formulation uses a soft penalty on normalized mismatch fraction (Sutradhar et al., 6 Nov 2025). Privacy auditing uses an average over the $i$57 nearest neighbors rather than a single closest record (Meeus et al., 2023). Phylogenetic ADCL minimizes an average nearest-reference distance over a mass distribution rather than a pointwise nearest neighbor (Matsen et al., 2012). Record linkage uses a sup-of-inf set distance at the household level and a thresholded probability map rather than a single raw nearest-pair decision (Menezes et al., 2024).

Another misconception is that DCR is intrinsically geometry-only. Several formulations incorporate non-geometric semantics directly into the distance: label coverage in filtered ANN, attribute-level weights and monotonicity constraints in record linkage, spatial proximity in fuzzy image similarities, and mixed-type preprocessing in privacy auditing. This suggests that the decisive issue is not whether a method uses a nearest record, but which notion of admissible closeness it encodes.