ApproxJoin: Approximate Join Frameworks

• ApproxJoin is a set of frameworks for approximate join processing that relax exact execution to lower data movement or verification costs while preserving key accuracy guarantees.
• In Apache Spark, it interleaves Bloom-filter filtering with stratified sampling during the join to reduce shuffling and provide unbiased aggregate estimates with provable error bounds.
• Other variants apply to uniform sampling over join unions and approximate verification in fuzzy similarity joins, achieving significant speedups and efficiency improvements.

ApproxJoin denotes a set of approximate join frameworks that reduce the cost of join processing while preserving a task-specific guarantee. In the supplied literature, the name is used for three distinct settings: approximate distributed equi-joins in Apache Spark, random sampling from the set union of multiple joins without computing the full join or union, and approximate verification in fuzzy set similarity join. Across these settings, the common design pattern is not a single algorithm but a controlled relaxation of exact join execution in favor of lower latency, lower communication, or lower verification cost, while retaining unbiased estimation, uniform and independent sampling, or high recall, respectively (Quoc et al., 2018, Liu et al., 2023, Mandulak et al., 25 Jul 2025).

1. Terminological scope and problem classes

The term ApproxJoin is not tied to one canonical formulation. It refers, in the provided sources, to systems that approximate different objects: the output of a distributed equi-join, the union of several join results, or the verification phase of a fuzzy set similarity join. This distinction matters because the approximation target determines both the algorithmic structure and the admissible error notion (Quoc et al., 2018, Liu et al., 2023, Mandulak et al., 25 Jul 2025).

Variant	Problem setting	Core mechanism
ApproxJoin in Apache Spark	Approximate equi-joins over massive datasets	Bloom-filter sketching and stratified sampling during the join
ApproxJoin for union of joins	Uniform, independent sampling from $J_1 \cup \cdots \cup J_n$	Histogram or random-walk size estimation plus online union sampling
ApproxJoin for fuzzy set similarity join	Verification of candidate set pairs under fuzzy similarity	Approximate maximum-weight matching in the filter-verify pipeline

A common source of confusion is to treat ApproxJoin as a synonym for approximate query processing over joins. The Spark system is explicitly framed in the approximate computing paradigm, but the union-of-joins variant is a sampling framework with uniformity and independence guarantees, and the fuzzy-set variant is a verification acceleration method inside similarity join. This suggests that ApproxJoin is best understood as a label for several approximation strategies organized around different join bottlenecks rather than a single unified method.

2. Approximate distributed equi-joins in Apache Spark

In Apache Spark, ApproxJoin is implemented as a new PairRDD operation approxJoin() that replaces Spark’s native join and interleaves two stages: a Bloom-filter–based filtering stage and a stratified sampling–during–join stage. The filtering stage eliminates non-joinable tuples before shuffling, and the sampling stage produces an unbiased sample of the join output with provable error bounds. The design objective is explicit: minimize data movement first, then sample during the cross-product rather than before or after the join, because naive sampling over inputs does not preserve the statistical properties of the join output (Quoc et al., 2018).

The Bloom-filter stage is defined partitionwise. Each partition $p_{i,j}$ builds a local Bloom filter $p\text{-}BF_{i,j}$ of $m$ bits with $k$ independent hash functions; partition filters are merged by bitwise OR into $BF_i$ for dataset $R_i$; and the global join filter $BF$ is the bitwise AND of all $BF_i$. The filter is broadcast to all workers, which discard tuples whose join key is reported absent. Its false-positive probability is

$P_{fp}=\bigl(1-e^{-\,k n / m}\bigr)^k,$

and the filter size is chosen as

$p_{i,j}$0

to meet a target $p_{i,j}$1.

After filtering, ApproxJoin defines each distinct join key $p_{i,j}$2 as a stratum containing $p_{i,j}$3 filtered tuples. In the two-way case, if $p_{i,j}$4 and $p_{i,j}$5 are the tuples from $p_{i,j}$6 and $p_{i,j}$7 with key $p_{i,j}$8, then $p_{i,j}$9 sampled join-pairs are obtained by repeatedly drawing $p\text{-}BF_{i,j}$0 and $p\text{-}BF_{i,j}$1 at random and yielding $p\text{-}BF_{i,j}$2. For a join-sum $p\text{-}BF_{i,j}$3, the estimator is

$p\text{-}BF_{i,j}$4

with variance estimate

$p\text{-}BF_{i,j}$5

and confidence interval

$p\text{-}BF_{i,j}$6

where $p\text{-}BF_{i,j}$7. A Horvitz–Thompson alternative is also given for sampling without replacement or deduplicated sampling.

The system further introduces a cost function that chooses the sampling fraction $p\text{-}BF_{i,j}$8 from either a desired latency or a desired error budget. Under a latency bound,

$p\text{-}BF_{i,j}$9

while under an error bound,

$m$0

This makes the operator budget-aware rather than purely heuristic.

Its complexity claims are centered on communication avoidance and cross-product reduction. Shuffled-data volume for the ApproxJoin filtering and sampling stage is

$m$1

and, with minimal false positives, the Bloom-filter overhead is small and the second term dominates, yielding 5–82× less shuffle than full repartition. Computationally, exact joins materialize

$m$2

per key $m$3, whereas ApproxJoin samples a fraction $m$4, giving

$m$5

with empirical speedups of 6–9× at moderate sampling rates such as $m$6.

The reported evaluation emphasizes both systems and statistical outcomes. Filtering alone, with join overlap $m$7, is 2–3× faster than repartition or native joins and uses 12–45× less shuffle. For 3–4-way joins, speedup reaches 8× and shuffle reduction reaches 82×. SamplingDuringJoin is 10–30× lower latency than post-join stratified sampling for the same sample sizes. In TPC-H Q3, Q4, and Q10, ApproxJoin is 1.2–1.8× faster than SnappyData on 10 GB scale-10 data; at 60% sample, it attains 1.77× lower latency than SnappyData with 0.021% versus 0.016% error. The CAIDA network-traffic case study reports 300× less shuffle under filtering-only execution and, at 10% sampling, 42× lower error than repartition plus pre-sample. The Netflix Prize case study reports 6–9× speedup at 10% sample with $m$8 accuracy loss.

3. Random sampling over the union of joins

A different ApproxJoin formulation addresses the problem of drawing i.i.d. tuples from the set union of multiple join results without computing either the full joins or their union. Let $m$9 be schema-aligned multi-way equi-joins, with

$k$0

and

$k$1

The goal is to produce $k$2 tuples $k$3 such that, for every $k$4, $k$5, and the draws are mutually independent. The supplied details attribute this framework to Liu et al., VLDB ’20, and describe it as ApproxJoin: Random Sampling over the Union of Joins (Liu et al., 2023).

The key difficulty is that sampling from the disjoint union $k$6 is easy, but sampling from the set union $k$7 is not, because overlapping tuples must be returned exactly once while still remaining uniform. The framework therefore requires estimates of each $k$8, the union size $k$9, and all overlap sizes

$BF_i$0

for $BF_i$1.

Two estimation methods are given. The histogram-based direct method uses simple column statistics such as histograms and max-degrees. For equal-length aligned chain joins, it defines overlap bounds $BF_i$2 by

$BF_i$3

and recursively,

$BF_i$4

Then $BF_i$5. The random-walk method instead uses online unbiased estimators over layered join graphs, following WanderJoin-style sampling. For a sampled walk $BF_i$6,

$BF_i$7

and an unbiased estimator of $BF_i$8 is

$BF_i$9

Overlap estimation is then obtained by membership tests across the remaining joins.

The online sampling algorithm couples a warm-up phase with a refinement phase. Warm-up estimates $R_i$0, overlap sizes, and cover-weights $R_i$1, where $R_i$2 is a cover partition of overlaps. Online execution repeatedly selects a join index $R_i$3 with probability $R_i$4, samples from a pre-sample pool $R_i$5 or via WanderJoin, and applies an earliest-claim rule with revision and rejection so that duplicate appearances across joins do not violate set semantics. Periodic refinement updates the weights and uses backtracking to restore i.i.d. behavior after estimates change.

The theoretical statement is explicit: if, at every iteration, the algorithm selects $R_i$6 with probability $R_i$7 and then samples $R_i$8 uniformly, the earliest-claim rule guarantees that every $R_i$9 is returned with probability $BF$0, and that draws are independent. The histogram method provides only worst-case overlap bounds and can be loose under skew, whereas the random-walk estimator is unbiased and has confidence intervals shrinking at approximately $BF$1.

The reported TPC-H evaluation exposes a clear throughput–accuracy trade-off. For workload UQ1 with five joins, histogram-based ApproxJoin reaches 45 K tuples/s with 12% union-size error and 2.1% downstream query error, while walk-based ApproxJoin reaches 18 K tuples/s with 2.3% union-size error and 0.4% query error. For UQ2, the corresponding numbers are 70 K tuples/s, 7.5%, and 1.4% versus 27 K tuples/s, 1.1%, and 0.2%. For UQ3, they are 50 K tuples/s, 10%, and 1.9% versus 20 K tuples/s, 1.8%, and 0.3%. With sample reuse enabled, end-to-end throughput improves by $BF$2. Increasing the warm-up budget from 5 K to 50 K walk samples reduces union-size error from about 2.5% to about 0.8% while increasing warm-up time from about 2 s to about 18 s.

4. ApproxJoin in fuzzy set similarity join

In fuzzy set similarity join, ApproxJoin addresses a different bottleneck: the verification stage after candidate generation. Let $BF$3 and $BF$4 be collections of sets. For $BF$5 and $BF$6, with element similarity $BF$7, the framework defines a complete bipartite graph $BF$8, edge weights $BF$9, a maximum-weight bipartite matching $BF_i$0, fuzzy overlap

$BF_i$1

and fuzzy set similarity

$BF_i$2

Given threshold $BF_i$3, the threshold-based fuzzy set similarity join is

$BF_i$4

The verification problem is an assignment integer program, and the dominant exact baseline is Hungarian matching, which induces high cost (Mandulak et al., 25 Jul 2025).

ApproxJoin replaces exact verification by approximate maximum-weight matching. Three algorithms are evaluated. Greedy (GD) has approximation ratio $BF_i$5 and time complexity $BF_i$6 to sort edges and $BF_i$7 to scan them, with practical complexity $BF_i$8 when $BF_i$9. Locally-Dominant (LD) also has approximation ratio $P_{fp}=\bigl(1-e^{-\,k n / m}\bigr)^k,$0, scans all edges per pass, and in practice behaves as $P_{fp}=\bigl(1-e^{-\,k n / m}\bigr)^k,$1 because of data structures and pointer updates. Paz–Schwartzman (PS) has approximation ratio $P_{fp}=\bigl(1-e^{-\,k n / m}\bigr)^k,$2, one-pass $P_{fp}=\bigl(1-e^{-\,k n / m}\bigr)^k,$3 streaming time plus $P_{fp}=\bigl(1-e^{-\,k n / m}\bigr)^k,$4 post-processing, and $P_{fp}=\bigl(1-e^{-\,k n / m}\bigr)^k,$5 space.

These approximate matchers are inserted into a largely unchanged TokenJoin-style filter–verify pipeline. The filter phase includes size filtering, prefix or positional filtering over 3-gram tokens, and a joint utility filter. Verification then proceeds via deduplication of exact matches, graph building, an upper-bound test, and matching. ApproxJoin changes only the matching step: one of GD, LD, or PS is used instead of Hungarian (HG) or Efficient Verification (EV).

The evaluation spans self-joins on ten datasets, including LIVEJ, AOL, KOSARAK, ENRON, DBLP, FLICKR, GDELT, BMS-POS, YELP, and MIND, with default threshold $P_{fp}=\bigl(1-e^{-\,k n / m}\bigr)^k,$6. Verification speedups versus Hungarian are substantial: geometric-mean 4.5× for AJ-PS, 4.7× for AJ-GD, and 4.3× for AJ-LD, with a maximum of 19.1× on KOSARAK for AJ-PS. End-to-end runtime speedups versus Hungarian have geometric mean about 2.5× and peak at 6.7× on KOSARAK. Against Efficient Verification, the speedups are 2.3×, 2.2×, and 2.0× for verification and about 1.5× end-to-end. In raw approximate mode, average recall is 0.997 for AJ-PS and 0.982 for AJ-GD and AJ-LD. Replacing the final score with the dual-based upper bound, or doubling GD/LD weights, recovers 100% recall at the cost of less than 2% extra candidates, with precision approximately 0.998. AJ-PS also uses on average 23% less peak memory than Hungarian or Efficient Verification, and up to 72% less on KOSARAK.

The conceptual significance of this version of ApproxJoin is that the approximation lies not in join sampling or join-size estimation but in the matching primitive used to decide whether a candidate pair satisfies the fuzzy overlap threshold. The result is a verification-oriented ApproxJoin whose primary accuracy metric is recall rather than unbiasedness.

5. Guarantees, objectives, and recurrent misconceptions

The supplied ApproxJoin systems differ less in name than in the invariant each tries to preserve. In the Spark setting, the target is an unbiased estimator for an aggregate over the join output, together with variance estimates and confidence intervals. In the union-of-joins setting, the target is exact uniformity and independence of output samples from the set union, given correct cover weights. In the fuzzy-similarity setting, the target is to preserve candidate quality under approximate verification, typically reported through recall, precision, and approximation ratios rather than confidence intervals (Quoc et al., 2018, Liu et al., 2023, Mandulak et al., 25 Jul 2025).

This difference has methodological consequences. Bloom-filter filtering and stratified sampling are appropriate when the central cost is network shuffle and Cartesian-product materialization. Histogram and random-walk estimation are appropriate when the challenge is sampling from a union of overlapping join results without materializing overlaps. Approximate matchers are appropriate when the dominant cost is cubic or near-cubic matching in candidate verification. A practical implication is that “ApproxJoin” does not imply any fixed error model. Depending on the formulation, the relevant quantity may be $P_{fp}=\bigl(1-e^{-\,k n / m}\bigr)^k,$7, a confidence interval around $P_{fp}=\bigl(1-e^{-\,k n / m}\bigr)^k,$8, a uniformity proof, an overlap bound, a q-error-like downstream effect, an approximation ratio $P_{fp}=\bigl(1-e^{-\,k n / m}\bigr)^k,$9 or $p_{i,j}$00, or empirical recall.

Another recurring misconception is that approximation always occurs before or after a join. The Spark system explicitly argues against pre-join or post-join sampling because such methods do not preserve the join output’s statistical properties; it instead samples during the join. The fuzzy-set system, by contrast, leaves candidate generation largely intact and approximates only the verification step. The union-of-joins system approximates neither a traditional join operator nor a matching primitive, but the combinatorics of overlap-aware sampling over $p_{i,j}$01.

6. Relation to adjacent approximate join research

ApproxJoin occupies only part of the broader design space for approximate join research. Closely related work pursues different abstractions and optimization objectives. “Model Joins: Enabling Analytics Over Joins of Absent Big Tables” formulates a model-join problem in which absent raw tables are replaced by per-table compact models, then joined through a PGM-style inference and sampling pipeline to produce a uniform and independent sample that is a high-quality approximation of a uniform and independent sample of the actual raw-data join. Under exact marginals and conditionals, its SkeletonSample procedure yields $p_{i,j}$02 i.i.d. draws from the uniform distribution on $p_{i,j}$03; when models are approximate, all approximation error stems from model error rather than the join machinery (Shanghooshabad et al., 2022).

“Sketched Sum-Product Networks for Joins” addresses cost-based query optimization rather than runtime join execution or verification. It uses SPNs whose leaves store sketches, allowing the sketch of a filtered relation $p_{i,j}$04 to be approximated on the fly and then used for join cardinality estimation. Reported numbers include median q-error around 1.15 on JOB-light for Sketched SPN + Fast-AGMS Max, inference times of approximately 11 ms, and end-to-end query runtimes within 3% of a baseline with oracle cardinalities (Tsan et al., 16 Jun 2025).

“QJoin: Transformation-aware Joinable Data Discovery Using Reinforcement Learning” addresses join discovery in heterogeneous repositories, where the challenge is learning transformation chains that make columns joinable. It casts the search for a transformation sequence as an MDP with state $p_{i,j}$05, a uniqueness-aware reward, agent transfer, and transformation reuse. On the AutoJoin Web benchmark, it reports average $p_{i,j}$06, and on 19,990 join tasks over NYC and Chicago open data it reports runtime reduction by up to 7.4% through reuse (Wang et al., 2 Dec 2025).

These neighboring systems clarify the scope of ApproxJoin. ApproxJoin, in its named forms here, is concerned with approximating join execution, union sampling, or fuzzy verification. Adjacent work extends the agenda to absent-table analytics, optimizer-oriented synopses, and transformation-aware join discovery. A plausible implication is that approximate join research is better organized by approximation locus—execution, sampling, verification, optimization, or discovery—than by surface terminology alone.