Record Linking & Entity Resolution

• Record linking is the process of identifying records that refer to the same entity across different data sources even when unique identifiers are unreliable.
• It involves a classical workflow of pre-processing, blocking, similarity computation, and decision-making using techniques such as the Fellegi–Sunter model.
• Advanced methods incorporate Bayesian formulations and privacy-preserving protocols to handle noisy, inconsistent, and incomplete data in various domains.

Record linkage (RL), also called data matching, entity resolution, duplicate detection, or exact file matching, is the task of identifying records in one or more datasets that refer to the same real-world entity when no perfectly reliable, unique identifier is available. In practice, RL operates on quasi-identifiers such as names, addresses, dates of birth, gender, or other partially identifying attributes, all of which may be missing, erroneous, inconsistent, or differently formatted across files. The topic spans deterministic and probabilistic linkage, Bayesian and graphical models, privacy-preserving protocols, and domain-specific systems for health, government, business, and social-network data (Larsen, 2012, Vatsalan et al., 2022).

1. Problem formulation and data representation

A standard bipartite RL setting considers two files, $A$ and $B$, and asks, for each pair $(a,b)$ with $a \in A$ and $b \in B$, whether the two records represent the same underlying entity. In its simplest formalization, the input is a comparison vector over $K$ fields,

$y(a,b) = \{Y_1(a,b),\dots,Y_K(a,b)\},$

where $Y_k(a,b)$ may be binary agreement indicators or, in richer variants, graded or ordinal similarities. The latent target is a match indicator

$$I(a,b) = \begin{cases} 1, & \text{if } a \text{ and } b \text{ are matches},\ 0, & \text{otherwise.} \end{cases}$$

This formulation already makes clear why RL is difficult: the task is a latent classification problem defined on noisy cross-file comparisons rather than on direct identifiers (Larsen, 2012).

The same conceptual structure generalizes to more than two files and to de-duplication within files. In graphical Bayesian formulations, records are not linked directly to one another; instead, they are linked to latent true individuals, so that two observed records match if and only if they share the same latent individual. This representation naturally enforces transitivity and accommodates both cross-file linkage and within-file duplicates (Steorts et al., 2013).

A further refinement appears when some variables exist in only one file. In that setting, the linkage structure is often written as an $n_A \times n_B$ binary matrix $B$0 with one-to-one constraints, where $B$1 indicates that record $B$2 in file $B$3 and record $B$4 in file $B$5 refer to the same person. The methodological significance is that RL need not be restricted to variables shared by all files; under Bayesian formulations, variables exclusive to one file can also inform the linkage and the downstream analysis (Kamat et al., 2023).

2. Classical workflow and probabilistic foundations

A standard RL pipeline consists of pre-processing and cleaning, blocking or indexing, similarity computation, classification, clustering or consolidation, and evaluation. Pre-processing standardizes formats, parses composite fields, and may apply edit distance, HMMs, or spelling correction. Blocking reduces the infeasible all-pairs comparison problem, which in the two-file case is $B$6, by restricting comparisons to candidate pairs within shared blocks. Similarities are then computed field-wise and combined into a record-level score, after which thresholding, rule-based methods, probabilistic linkage, or supervised classifiers are used to decide match, non-match, or possible match (Vatsalan et al., 2022).

The classical probabilistic reference point is Fellegi–Sunter-style linkage. For each field $B$7, one defines an $B$8-probability for agreement among true matches and a $B$9-probability for agreement among non-matches. Agreement and disagreement contribute weights

$(a,b)$0

and the total pair score is

$(a,b)$1

This produces a likelihood-ratio ranking of candidate pairs and, with thresholds, a match/non-match decision rule (Biswas, 2024).

Latent class formulations recast the same logic as a two-component mixture over matches $(a,b)$2 and nonmatches $(a,b)$3: $(a,b)$4 Under conditional independence within classes, [ \Pr(y\mid C)=\prod_{k=1}^K \Pr(Y_k\mid C)^{y_k}\left(1-\Pr(Y_k\mid C)\right)^{1-y_k},\quad C\in{M,U}. \