Streaming Bayesian Record Linkage

• Streaming Bayesian record linkage comprises Bayesian methods that incrementally update probabilistic inferences for real-time entity resolution across sequential data sources.
• Key methodologies include Negative Binomial (Poisson–Gamma) classifiers and Bayesian Fellegi–Sunter models, which leverage Dirichlet and Beta priors for robust matching.
• Efficient streaming update algorithms, such as PPRB-w-G and SMCMC, deliver scalable performance with high F1 scores and reduced computational overhead in practical applications.

Streaming Bayesian record linkage refers to the set of Bayesian methodologies for linking records from multiple databases or files that arrive sequentially over time, while efficiently updating probabilistic inferences and model parameters as new data streams in. This paradigm addresses the problem of combining information about overlapping entities across sources lacking unique identifiers, and is particularly suited to modern contexts such as longitudinal surveys, electronic health records, and real-time event monitoring (Taylor et al., 2023, K et al., 2019).

1. Probabilistic Models for Streaming Record Linkage

Two principal Bayesian graphical models for streaming record linkage are prominent in the literature:

• Negative Binomial (Poisson-Gamma) Classifier: For each record-pair and field, observed error counts $X_f$ (e.g., edit-distance) are modeled as Poisson random variables conditional on latent error-rates $\theta_f$:

$$X_f \mid \theta_f \sim \text{Poisson}(\theta_f), \qquad \theta_f \sim \text{Gamma}(\alpha_f, \beta_f)$$

Integrating out $\theta_f$ leads to a Negative Binomial marginal:

$$X_f \sim \text{NB}(r=\alpha_f,\; p=\beta_f/(\beta_f+1))$$

Match and non-match record-pairs each possess class-specific hyperparameters $(\alpha_f^{(M)}, \beta_f^{(M)})$ and $(\alpha_f^{(U)}, \beta_f^{(U)})$ (K et al., 2019).

• Multi-file Bayesian Fellegi–Sunter Model: For $k$ sequentially arriving files $X_1,\ldots,X_k$, and $F$ common fields, pairwise binary “agreement” vectors $\theta_f$0 are constructed. The linkage structure is parameterized by matching vectors $\theta_f$1, which encode transitive clusters under the constraint of no within-file duplicates. Matching indicators follow a mixture likelihood, with Dirichlet priors on per-field match/non-match multinomial parameters $\theta_f$2 (Taylor et al., 2023).

2. Streaming Update Algorithms

Efficient streaming record linkage requires algorithms that integrate new data with minimal recomputation over the full record set. Two main strategies have been developed:

• Supervised Negative Binomial (Poisson-Gamma) Streaming: For each incoming labeled record-pair and field $\theta_f$3 with observed count $\theta_f$4, class-dependent Gamma hyperparameters are updated via:

$\theta_f$5

where $\theta_f$6 indicates the observed label ("M" for match, "U" for non-match). Only $\theta_f$7 hyperparameters and two class priors are maintained; per record-pair cost is $\theta_f$8 (K et al., 2019).

• Streaming Bayesian Fellegi–Sunter Updates:
  • Prior-Proposal-Recursive-Bayes within Gibbs (PPRB-w-G): Posterior draws from the previous stage are recycled as proposals for Metropolis-Hastings steps at stage $\theta_f$9. New file integration involves partitioning parameters into previously estimated, currently updated, and newly arrived segments; only $$X_f \mid \theta_f \sim \text{Poisson}(\theta_f), \qquad \theta_f \sim \text{Gamma}(\alpha_f, \beta_f)$$0 computational effort is required per file for key steps, enabling handling of large streaming datasets.
  • Sequential MCMC (SMCMC): Maintaining ensembles of parameter and matching draws, SMCMC applies a combination of jumping kernels for new data and transition kernels for global update. Chains are run in parallel, with performance controlled by ensemble size and number of iterations. Complete data retention is required (Taylor et al., 2023).

3. Priors and Hyperparameter Specification

Prior specification in streaming Bayesian record linkage directly affects robustness and adaptivity:

• Dirichlet Priors: For multinomial parameters ($$X_f \mid \theta_f \sim \text{Poisson}(\theta_f), \qquad \theta_f \sim \text{Gamma}(\alpha_f, \beta_f)$$1, $$X_f \mid \theta_f \sim \text{Poisson}(\theta_f), \qquad \theta_f \sim \text{Gamma}(\alpha_f, \beta_f)$$2), weakly-informative defaults set $$X_f \mid \theta_f \sim \text{Poisson}(\theta_f), \qquad \theta_f \sim \text{Gamma}(\alpha_f, \beta_f)$$3; anticipated per-field error rates can be encoded via $$X_f \mid \theta_f \sim \text{Poisson}(\theta_f), \qquad \theta_f \sim \text{Gamma}(\alpha_f, \beta_f)$$4 with total strength $$X_f \mid \theta_f \sim \text{Poisson}(\theta_f), \qquad \theta_f \sim \text{Gamma}(\alpha_f, \beta_f)$$5.
• Beta Priors: For match-rate $$X_f \mid \theta_f \sim \text{Poisson}(\theta_f), \qquad \theta_f \sim \text{Gamma}(\alpha_f, \beta_f)$$6, a $$X_f \mid \theta_f \sim \text{Poisson}(\theta_f), \qquad \theta_f \sim \text{Gamma}(\alpha_f, \beta_f)$$7 prior governs expected linkage sparsity/density; default is (1, 1).
• Effect of Priors: In high-error, low-overlap settings, strong informative priors on $$X_f \mid \theta_f \sim \text{Poisson}(\theta_f), \qquad \theta_f \sim \text{Gamma}(\alpha_f, \beta_f)$$8 yield significant $$X_f \mid \theta_f \sim \text{Poisson}(\theta_f), \qquad \theta_f \sim \text{Gamma}(\alpha_f, \beta_f)$$9 gains (up to +0.4 compared to flat priors) (Taylor et al., 2023).
• Adaptivity: In Poisson–Gamma models, incremental re-estimation of $\theta_f$0 with each labeled pair enables adaptation to nonstationary field-error distributions, with rates tunable by decaying old counts or binning updates (K et al., 2019).

4. Joint Likelihood, Independence, and Scoring

Both Bayesian approaches maintain tractability via a conditional independence assumption across fields:

• Joint Predictive Distribution: For a record-pair $\theta_f$1, the posterior match probability is computed as:

$\theta_f$2

or, for Dirichlet-multinomial models, using the appropriate mixture likelihoods across matching/non-matching clusters.

• Practical Computation: For stability, log-likelihoods are used:

$\theta_f$3

• Memory and Computation: Only order-$\theta_f$4 state is needed for the Poisson–Gamma model. For multi-file Fellegi–Sunter models, PPRB-w-G minimizes storage by not retaining old data; SMCMC trades memory for parallelizability (Taylor et al., 2023, K et al., 2019).

5. Empirical Performance and Evaluation

Empirical validation has established the efficacy and efficiency of streaming Bayesian record linkage:

• Metrics: Standard linkage-accuracy metrics are employed: precision, recall, $\theta_f$5, and AUC, calculated over all labeled pairs observed up to time $\theta_f$6 (K et al., 2019, Taylor et al., 2023).
• Simulation Studies: In scenarios with four files of 200 records (overlaps 10–90%, errors per duplicate = 2–6), streaming models with flat/weak/strong priors achieve $\theta_f$7–$\theta_f$8, meeting or exceeding non-Bayesian baselines (multiLink/blink/SVM). The posterior on total entities is tightly concentrated around the ground truth; SVM linkage may violate transitivity (Taylor et al., 2023).
• Real-World Case Study: Application to Polish SDS survey files (2007–2013; 1,980 records, 910 true entities, six binary fields) yields $\theta_f$9 for all Bayesian methods. Sampling times: full-Gibbs = 121 hr; PPRB-w-G = 10.9 hr; SMCMC variants = 3.5–6.9 hr—demonstrating substantial speedups (Taylor et al., 2023).
• Adaptation to Data Streams: The Poisson–Gamma model demonstrates robustness to the sparsity typical of streaming or actively labeled data (K et al., 2019).

6. Guidelines and Practical Considerations

Effective application of streaming Bayesian record linkage benefits from the following:

• Algorithm Selection: When parallel compute is available, SMCMC is preferred (ensemble size $$X_f \sim \text{NB}(r=\alpha_f,\; p=\beta_f/(\beta_f+1))$$0–$$X_f \sim \text{NB}(r=\alpha_f,\; p=\beta_f/(\beta_f+1))$$1, 50–200 transition steps); for limited memory or thread count, PPRB-w-G with $$X_f \sim \text{NB}(r=\alpha_f,\; p=\beta_f/(\beta_f+1))$$2 and locally-balanced proposals for $$X_f \sim \text{NB}(r=\alpha_f,\; p=\beta_f/(\beta_f+1))$$3 is advantageous. For PPRB-w-G, periodic full-Gibbs refresh may be necessary to avoid degeneracy in posterior samples (Taylor et al., 2023).
• Priors: Tuning Dirichlet prior parameters to reflect empirically observed or expected error rates is recommended; otherwise, use flat priors. The overall match-rate prior should be chosen to reflect likely linkage density.
• Scalability for Large Files: Files may be block-partitioned by time or geography; blocks are processed as mini-streams to reduce comparison cost.
• Uncertainty Assessment: Examination of the posterior distribution over cluster count and $$X_f \sim \text{NB}(r=\alpha_f,\; p=\beta_f/(\beta_f+1))$$4-score is essential for uncertainty quantification.
• Adaptive Updating: The Poisson–Gamma streaming approach allows incremental adaptation to distributional changes; minibatch or count decay strategies can further enhance responsiveness to nonstationarity (K et al., 2019).

7. Summary Table: Core Methods and Properties

Method	Key Features	Reference
Poisson–Gamma NB Classifier	$$X_f \sim \text{NB}(r=\alpha_f,\; p=\beta_f/(\beta_f+1))$$5 memory; adaptive; field-wise NB scores; active/sparse data	(K et al., 2019)
Multi-file Bayesian Fellegi–Sunter	Dirichlet-multinomial; full posterior on clusters; match vectors $$X_f \sim \text{NB}(r=\alpha_f,\; p=\beta_f/(\beta_f+1))$$6	(Taylor et al., 2023)
PPRB-w-G (streaming Gibbs)	Fast update, no old data storage; risk of degeneracy	(Taylor et al., 2023)
SMCMC	Fully parallelizable; ensemble never degenerates; higher memory	(Taylor et al., 2023)

Streaming Bayesian record linkage encompasses a family of rigorously formulated, efficiently updated probabilistic approaches for real-time and sequential entity resolution. Models based on Poisson–Gamma and multi-file Dirichlet-multinomial likelihoods—along with their associated streaming update algorithms—provide robust, interpretable, and computationally scalable solutions for modern record linkage tasks (K et al., 2019, Taylor et al., 2023).