---
title: Distance-to-Closest-Record Filtering
url: https://www.emergentmind.com/topics/distance-to-closest-record-filtering
type: topic
---

# Distance-to-Closest-Record Filtering

Searching arXiv for the cited papers to ground the article in the literature.
arXiv search query: 2511.04073
Distance-to-Closest-Record Filtering denotes a family of selection, ranking, and pruning strategies in which the decisive quantity is a distance to a nearest admissible, neighboring, or representative record. In the literature, this idea appears in several technically distinct forms: as a filtered nearest-neighbor ranking criterion in vector search, as an inclusion rule after blocking in probabilistic record linkage, as a minimal-neighbor statistic in reduced-ordering vector filters, as an average-distance-to-nearest-reference objective on phylogenetic trees, as a vulnerability score based on nearest neighbors in synthetic-data privacy auditing, and as a separation constraint in nearest-neighbor condensation [2511.04073] [1603.07816] [1009.0957] [1205.6867] [2306.10308] [2006.15650].

## 1. Formal characterizations

A common abstraction is to define, for each object, a nearest feasible or nearest retained counterpart and to use that quantity either as a score or as a filter. In record linkage after blocking, a generic DCR-type inclusion rule keeps a pair $(i,j)$ if it is sufficiently close to the closest observed distance for $i$, for example
\[
I_{ij} = 1\{ d(x_i,x_j) \le f(d_i^{\min}) \}, \qquad d_i^{\min} = \min_{j} d(x_i,x_j),
\]
or, in a top-$k$ form, $I_{ij} = 1\{ j \in \mathsf{NN}_k(i)\}$ [1603.07816].

In filtered ANN search with discrete labels, the closest-record notion is made explicit through a learned filter-aware distance
\[
D_f(q, x; w_m) = d(q, x) + w_m \cdot (1 - m(S, L(x))),
\]
where $m(S, L(x)) = |S \cap L(x)| / |S|$ measures filter compliance. Feasible records satisfy $S \subseteq L(x)$ and therefore incur no penalty, whereas infeasible records incur an additive penalty proportional to their normalized mismatch fraction [2511.04073].

In reduced-ordering vector filtering for color images, the decisive statistic can be the minimal neighbor distance
\[
d_{\min}(x_k) = \min_{i \neq k} d(x_k, x_i),
\]
with output rule
\[
x_* = \arg\min_k d_{\min}(x_k).
\]
This realizes DCR as a local neighborhood operator: the selected vector is the sample whose nearest neighbor is as close as possible [1009.0957].

In phylogenetic subset selection, the closest-record quantity is averaged over a demand distribution. For a selected leaf set $S$,
\[
ADCL(S) = \frac{1}{M}\int_{x\in T} m(x)\, d_T(x,S)\, dx,
\qquad
d_T(x,S)=\min_{s\in S} d_T(x,s),
\]
and the optimization problem is to choose $k$ leaves minimizing the average distance to the closest selected leaf [1205.6867].

In privacy auditing for synthetic data, the same pattern appears as a nearest-neighbor isolation score,
\[
V_k(x_i) := \frac{1}{k}\sum_{j=1}^k d(x_i, x_{i_j}),
\]
where $x_{i_1},\dots,x_{i_{|D|-1}}$ are ordered by increasing distance from $x_i$. Larger $V_k(x_i)$ indicates that a record is more isolated from its neighbors in the real dataset [2306.10308].

These formulations show that “closest record” need not mean a literal top-1 Euclidean neighbor. Depending on the domain, the closest object may be feasible under filters, closest under a set distance, closest inside a local image window, closest selected leaf in a tree metric, or closest among encoded mixed-type tabular records. This suggests that Distance-to-Closest-Record Filtering is best understood as a design principle rather than a single algorithmic template.

## 2. Filter-aware nearest-neighbor search

In filtered ANN search, the task is to retrieve the $k$ nearest vectors to a query $q$ subject to an AND-style filter requirement $S \subseteq L(x)$. The formulation in "Learning Filter-Aware Distance Metrics for Nearest Neighbor Search with Multiple Filters" replaces fixed penalties with a learned scalar $w_m$ in the additive distance
\[
D_f(q, x; w_m) = d(q, x) + w_m \cdot (1 - m(S, L(x))),
\]
so that each missing label contributes $w_m / |S|$ to the effective distance. During index construction, an analogous asymmetric pairwise form
\[
D_f(x_i, x_j; w_m) = d(x_i, x_j) + w_m \cdot (1 - m(L(x_i), L(x_j)))
\]
encourages edges where $x_j$’s labels cover most of $x_i$’s labels while preserving geometric proximity [2511.04073].

The weight is learned by a constrained linear program with slack. For each training query, positives are exact top-$k$ unfiltered neighbors that satisfy the filter, and negatives are geometrically closer points that violate the filter. The optimization minimizes
\[
w_m + \lambda \sum s_{q,i,j}
\]
subject to ranking constraints with a margin $\epsilon > 0$; in the reported experiments, $\epsilon = 0.01$. Because positives have zero filter penalty, the program learns the smallest $w_m$ that separates feasible neighbors from closer violating neighbors except where slack absorbs unavoidable violations. A useful derived bound is
\[
w_m \ge \frac{d(q, x_i) - d(q, x_j) + \epsilon}{1 - m(S, L(x_j))}
\]
for a feasible $x_i$ and infeasible $x_j$ to ensure that the feasible item outranks the violating one under $D_f$.

The learned distance is integrated into graph construction and search. The index build uses greedy expansion and robust prune steps keyed by $D_f$, and query-time graph search maintains a min-heap prioritized by $D_f(q,\cdot;w_m)$. Query planning routes highly selective queries, estimated satisfying set $< 100{,}000$, to a brute-force pass over the feasible subset using inverted indices; otherwise graph search with $D_f$ is used. Complexity is dominated by the number of graph expansions; per expansion the method computes one base distance and a constant-time label mismatch via bitset operations or precomputed inverted indices.

The reported empirical results are on YFCC1M and Wikipedia-35M. YFCC1M uses 1M CLIP image embeddings (192-D) with metadata labels and 1,727 evaluation queries; the learned weight is $w_m = 0.017787$. Wikipedia-35M uses 35M sentence embeddings (768-D) with 464 evaluation queries; the learned weight is $w_m = 0.204148$. Across both datasets, using $D_f$ in both build and search increases Recall@10 by 5–10% over fixed-penalty and post-filter baselines at similar or lower distance-comparison counts, while Figure 3 shows comparable unfiltered Recall@10 and distance comparisons when filters are omitted. The implementation details reported are cosine similarity for both datasets, bitset intersections for $m$, PuLP for the LP, and $\lambda$ tuned by grid search on a validation split [2511.04073].

Within this framework, “Distance-to-Closest-Record Filtering” is realized by an effective distance to the closest feasible record. For feasible $x$, $D_f(q,x;w_m)=d(q,x)$; for infeasible $x$,
\[
D_f(q,x;w_m)= d(q,x) + w_m \cdot |S \setminus L(x)|/|S| \ge d(q,x)+w_m/|S|.
\]
With sufficiently large $w_m$, the nearest neighbor under $D_f$ must lie in the feasible set $X_S$.

## 3. Record linkage and conditional filtering

Record linkage uses DCR ideas at both the candidate-generation and final-matching stages. In "Hausdorff Distance-Based Record Linkage for Improved Matching of Households and Individuals in Different Databases", the household-level distance between two households $H_s$ and $H_t$ is the symmetric Hausdorff distance
\[
\Delta_{st} =
\max\Big\{
\max_{i\in H_s}\min_{j\in H_t} d_{ij},
\;
\max_{j\in H_t}\min_{i\in H_s} d_{ij}
\Big\},
\]
where the individual-level distance is a non-negative weighted sum of attribute distances,
\[
d_{ij} = \sum_{k=1}^K \beta_k d_{ijk}, \qquad \beta_k \ge 0.
\]
For categorical attributes the distance is $0$ for agreement and $1$ otherwise; for ANASC (year of birth) the distance is $|ANASC_i-ANASC_j|/50$. The household match probability is modeled as
\[
p_{st} = \frac{e^{\beta_0-\Delta_{st}}}{1+e^{\beta_0-\Delta_{st}}},
\]
with $\beta_k \ge 0$ enforcing monotonicity. A household $H_s$ is matched to the household $H_t$ with the largest estimated probability provided it exceeds a threshold $\tau$, which is calibrated during training to align the estimated proportion of matched households with the known true proportion [2404.05566].

Within matched households, the paper fits a penalized logistic regression for individual pairs,
\[
q_{ij} =
\frac{e^{\alpha_0 - \sum_{k=1}^K \alpha_k d_{ijk}}}
     {1 + e^{\alpha_0 - \sum_{k=1}^K \alpha_k d_{ijk}}},
\qquad \alpha_k \ge 0,
\]
with ridge regularization estimated via glmnet and $\lambda$ tuned by cross-validation. Final one-to-one matches are obtained from a linear program maximizing $\sum \hat q_{ij} z_{ij}$ subject to one-to-one constraints and household-average probability thresholds. On the Italian SHIW application, the datasets contain 19,366 individuals and 8,156 households in 2014, 16,462 individuals and 7,420 households in 2016, and 15,198 individuals and 6,239 households in 2020. In external validation, hhlink outperforms fastLink at the individual level: for 2014–2016, hhlink reports $F1=87.29$, $FNR=13.25$, $FPR=5.05$, $PPV=87.84$, and Recall $=86.75$, whereas fastLink reports $F1=30.37$, $FNR=57.47$, $FPR=0.12$, $PPV=23.62$, and Recall $=42.53$; analogous gains appear for 2016–2020 [2404.05566].

The probabilistic-record-linkage literature also treats DCR explicitly as a filtering event that must be modeled conditionally. "Probabilistic Record Linkage and Deduplication after Indexing, Blocking, and Filtering" distinguishes indexing, blocking, and filtering, and places DCR squarely in the filtering stage. With a blocking indicator $\beta_{ij}$ and DCR inclusion indicator $\iota_{ij}$, inference is performed on the selected event $S_{ij}=1\{\beta_{ij}=1,\iota_{ij}=1\}$. After such filtering, the Fellegi–Sunter mixture must be conditioned on selection:
\[
\Pr[\gamma_{ij}=g \mid S_{ij}=1]
=
p_{M\mid S}\,\pi_{g\mid M,S}
+
(1-p_{M\mid S})\,\pi_{g\mid U,S}.
\]
A central result is that filtering can change the support of the comparison patterns, producing structural zeros, and can also change likelihood-ratio weight rankings when the filter is not a deterministic function of the comparison vector. The paper therefore recommends conditional quasi-independence with structural zeros and EM estimation on the retained pairs [1603.07816].

A recurrent misconception is that nearest-neighbor filtering in record linkage is merely a computational preprocessing step. The conditional analysis shows that, after DCR-type pruning, the statistical model itself changes: estimated match proportions, support, and error-rate interpretations are all conditional on having passed the filter.

## 4. Reduced-ordering vector filters in image processing

In color image denoising, DCR appears as a local ordering statistic. "Distance Measures for Reduced Ordering Based Vector Filters" studies nonlinear, order-statistics-based vector filters on RGB pixels. Given a window $W=\{x_1,\dots,x_n\}$, classical reduced ordering uses the aggregate statistic
\[
D_i = \sum_{j=1}^{n} d(x_i, x_j)
\]
and selects the vector minimizing $D_i$. The DCR alternative replaces the aggregate with the minimal-neighbor statistic
\[
d_{\min}(x_k)=\min_{i\neq k} d(x_k,x_i),
\qquad
x_*=\arg\min_k d_{\min}(x_k),
\]
with an optional $K$-nearest-neighbor variant
\[
d_K(x_k)=\frac{1}{K}\sum_{i\in\mathcal N_K(x_k)} d(x_k,x_i).
\]
The interpretation given is that true neighborhood samples tend to have at least one nearby similar sample, whereas impulses lack close neighbors and thus have larger $d_{\min}$ [1009.0957].

The paper evaluates 18 ordering criteria, including $d_1$, $d_2$, $d_\infty$, squared Euclidean, cosine angle, chord distance, divergence coefficient, Bray–Curtis, Canberra, Soergel, Ware–Hedges, Goude distance, and several fuzzy similarities convertible to distances via $d_{\text{fuzzy}}(x,y)=1-s(x,y)$ or $1/s(x,y)-1$. The combined fuzzy similarity with spatial proximity is
\[
s_{\text{cfs}}^{C,t}(x_i,x_j)
=
\frac{C}{C+d_2(x_i,x_j)}
\cdot
\frac{t}{t+\max(|r_i-r_j|,|c_i-c_j|)}.
\]
Filtering is performed in RGB, while CIELAB is used for the NCD evaluation measure.

The experimental setup uses 100 high-quality RGB images, correlated impulsive noise at 10%, 20%, and 30%, and a $3\times 3$ window. Reported metrics are MAE, MSE, NCD, and CPU time. The principal findings are that $s_{\text{cfs}}$, $s_{\text{fmds}}$, and $s_{\text{fms}}$ are the top performers overall; among traditional distances, $d_1$ and the divergence coefficient outperform $d_2$; direction-only measures such as cosine angle perform poorly; and $d_2^2$ is among the worst in effectiveness when used in the closest-to-mean reduced-ordering approximation. The paper reports overall mean effectiveness ranks of approximately $0.33$ for $s_{\text{cfs}}$, $2.06$ for $s_{\text{fmds}}$, $2.54$ for $s_{\text{fms}}$, $3.67$ for $d_1$, $4.26$ for divergence, and $5.88$ for $d_2$. For a 512×512 image with $n=9$, classical reduced ordering requires more than 9.4 million distance evaluations; DCR has the same asymptotic per-window cost $O(n^2)$ because it still requires the pairwise distance matrix [1009.0957].

The image-processing literature therefore treats closest-record filtering as a robustness device against impulsive outliers. At the same time, the paper notes that in textured or multimodal neighborhoods, $K=1$ can be unstable, and $K>1$ or a secondary aggregate statistic can mitigate that instability.

## 5. Representative subsets, tree metrics, and separation constraints

On phylogenetic trees, DCR becomes a global subset-selection objective. "Minimizing the average distance to a closest leaf in a phylogenetic tree" formalizes the problem of choosing $k$ leaves that minimize the average distance from all demand mass to the nearest selected leaf. The objective is ADCL:
\[
ADCL(S)=\frac{1}{M}\int_{x\in T} m(x)\, d_T(x,S)\, dx
\]
in the continuous case, or a weighted discrete analogue over mass points. The paper shows that a natural greedy deletion rule is not effective, that a variant of Partitioning Around Medoids (PAM) can get stuck in local minima, and that tree additivity enables an exact dynamic program based on bubble partitions, root boundary conditions, and lower envelopes of affine subwork lines. The exact program returns solutions for all numbers of leaves less than or equal to the target number, whereas PAM returns only a solution for the pre-specified number of leaves. On real data, ADCL minimization chooses chimeric sequences less often than random subsets, while phylogenetic diversity maximization chooses them more often than random [1205.6867].

A different use of closest-record filtering appears in nearest-neighbor condensation. "Social Distancing is Good for Points too!" studies the problem of finding a small consistent subset $R \subseteq P$ for nearest-neighbor classification. The FCNN heuristic can behave poorly when points are too close to each other. The paper’s modification, SFCNN, changes batch addition to one-add-per-iteration and thereby enforces a scale-adaptive separation condition. For
\[
R_{p,\sigma} = \{\, r\in R \mid ne(r)=p \;\wedge\; d(r,p)\in[\sigma,2\sigma) \,\},
\]
any two points $a,b \in R_{p,\sigma}$ satisfy
\[
d(a,b) \ge \sigma.
\]
This “social distancing” prevents the pathological selection of many arbitrarily close representatives. In doubling metrics with constant doubling dimension,
\[
|R| = O\!\Big(\kappa \,\big\lceil \log\frac{1}{\gamma}\big\rceil \, 4^{\text{ddim}+1}\Big),
\]
where $\kappa$ is the number of distinct nearest-enemy points and $\gamma$ is the smallest nearest-enemy distance. Empirically, SFCNN’s runtime and selected subset size are equivalent to FCNN across the reported datasets, while enabling provable upper bounds [2006.15650].

These two lines of work use closest-record quantities in opposite directions. ADCL minimizes the average distance from all demand points to a retained subset, whereas SFCNN filters out excessively close additions to control redundancy. A plausible implication is that DCR methods can serve both coverage and sparsification, depending on whether the objective is representativeness or exclusion.

## 6. Vulnerable-record identification in synthetic data

In synthetic-data privacy auditing, DCR is used to identify records most likely to be exposed by membership inference attacks. "Achilles' Heels: Vulnerable Record Identification in Synthetic Data Publishing" defines a mixed-type distance space by one-hot encoding categorical attributes and min–max normalizing continuous attributes. The main distance is a generalized cosine distance over the categorical and continuous parts, weighted by the fraction of attributes of each type, and the vulnerability score is the average $k$-nearest-neighbor distance
\[
V_k(x_i)=\frac{1}{k}\sum_{j=1}^k d(x_i,x_{i_j}).
\]
Records are ranked by decreasing $V_k$, and the paper selects the top-$R$ records with $R=10$; ties at the boundary are randomly broken. The default auditing setting uses $k=5$ [2306.10308].

The paper evaluates this procedure on UK Census (569,741 records, 17 categorical columns) and Adult (48,842 records, 15 columns: 9 categorical and 6 continuous), using SynthPop, BayNet, and PrivBayes. The query-based MIA uses shadow modeling with a random forest on $k$-way marginal counting queries, $N=100{,}000$ random queries, 100 trees, and maximum depth 10. A second evaluation uses a target-attention attack. For Adult, the protocol uses $|D_{\text{aux}}|=10{,}000$, $|D_{\text{test}}|=5{,}000$, $n_{\text{shadow}}=4{,}000$, $n_{\text{test}}=200$, and synthetic release size $|D|=|D^s|=1{,}000$; for UK Census, the corresponding auxiliary and test sizes are 50,000 and 25,000 [2306.10308].

Distance-based selection consistently outperforms Random, Rare value, and Log-likelihood baselines. For the query-based attack, UK Census with Synthpop reports mean AUCs $0.732\pm0.046$ for Random, $0.802\pm0.044$ for Rare value, $0.790\pm0.041$ for Log-likelihood, and $0.879\pm0.021$ for Distance; UK Census with BayNet reports $0.535\pm0.055$, $0.644\pm0.086$, $0.731\pm0.090$, and $0.858\pm0.040$, respectively. The paper states that distance-based selection increases AUC by 7.2 percentage points on average versus baselines for the query-based attack and by 5.2 percentage points on average for the target-attention attack. The method is reported to be robust across $k \in [1,50]$ and across cosine and Minkowski distances with $p\in\{1,2,3,4\}$, with slightly weaker performance for higher $p$. Under PrivBayes, attack AUCs drop as $\epsilon$ decreases, and the paper reports that MIAs fail when $\epsilon=1$ [2306.10308].

Here, closest-record filtering is not used to recover matches or denoise images, but to identify isolated real records. The paper’s interpretation is that more isolated records are more vulnerable because generators may need to memorize rare or atypical examples to reproduce local structure or maintain utility.

## 7. Recurrent assumptions, trade-offs, and misconceptions

Across domains, DCR methods rely on assumptions about what nearest-neighbor structure captures. In filtered ANN, the learned scalar $w_m$ is intended to adapt to label selectivity, with scarce feasible neighbors yielding larger $w_m$ and abundant feasibility yielding smaller $w_m$ [2511.04073]. In household linkage, the symmetric Hausdorff distance is sensitive to the worst nearest-neighbor relation, which the paper notes can be desirable for strict matching but can also be misleading when households split, merge, or change substantially [2404.05566]. In probabilistic record linkage after filtering, the retained comparison patterns may occupy only a strict subset of the original support, so post-filter inference is inherently conditional and can exhibit structural zeros [1603.07816].

The literature also repeatedly emphasizes precision–recall and robustness–cost trade-offs. Stricter household thresholds $\tau$ improve precision and reduce computational cost at the expense of recall [2404.05566]. DCR filtering in probabilistic linkage improves computational efficiency and often increases the match proportion among retained pairs, but it can induce selection bias and recall loss by discarding true matches whose nearest-neighbor distance is large or poorly measured [1603.07816]. In image filtering, $K=1$ can be too local in multimodal windows, whereas $K>1$ increases robustness [1009.0957]. In condensation, larger separation improves compression guarantees but can break consistency if pushed beyond the local margin $\gamma$ [2006.15650].

A frequent misconception is that closest-record filtering is always a hard top-1 rule. The surveyed methods contradict that view. The filtered ANN formulation uses a soft penalty on normalized mismatch fraction [2511.04073]. Privacy auditing uses an average over the $k$ nearest neighbors rather than a single closest record [2306.10308]. Phylogenetic ADCL minimizes an average nearest-reference distance over a mass distribution rather than a pointwise nearest neighbor [1205.6867]. Record linkage uses a sup-of-inf set distance at the household level and a thresholded probability map rather than a single raw nearest-pair decision [2404.05566].

Another misconception is that DCR is intrinsically geometry-only. Several formulations incorporate non-geometric semantics directly into the distance: label coverage in filtered ANN, attribute-level weights and monotonicity constraints in record linkage, spatial proximity in fuzzy image similarities, and mixed-type preprocessing in privacy auditing. This suggests that the decisive issue is not whether a method uses a nearest record, but which notion of admissible closeness it encodes.

Source: https://www.emergentmind.com/topics/distance-to-closest-record-filtering