Index-Based Join Sampling (IBJS)

• Index-Based Join Sampling (IBJS) is a framework that efficiently generates exact or subset-unbiased samples of join outputs using precomputed index structures.
• It employs uniform, weighted, and streaming reservoir sampling techniques to avoid full join materialization while ensuring optimal runtime and correctness.
• IBJS is central to advances in join size estimation, analytic query processing, and dynamic data analytics, supporting both static and streaming environments.

Index-Based Join Sampling (IBJS) is a foundational framework for sampling join results in large-scale relational data processing, enabling exact or subset-unbiased samples of join outputs without materializing the full join. IBJS algorithms exploit indexed structures and join-specific combinatorial properties, supporting uniform, weighted, and streaming sampling, while maintaining strong optimality, complexity, and correctness guarantees. IBJS unifies a series of influential algorithmic approaches and is central to multiple recent advances in join size estimation, analytic query processing, and streaming reservoir sampling.

1. Problem Formulation and Context

The canonical IBJS setting involves a join query $Q$ over $k$ base relations $R_1, ..., R_k$ defined on attribute sets $\{A_1, ..., A_d\}$, where each tuple is over domain $\{1,...,n\}^{|A_i|}$ (Esmailpour et al., 18 Dec 2025, Abo-Khamis et al., 2020). The goal is to sample $q$ join results from $J = R_1 \bowtie ... \bowtie R_k$, with one of the following guarantees:

• Uniform Sampling: Each output tuple is drawn exactly uniformly at random from $J$, with or without replacement (Abo-Khamis et al., 2020, Kim et al., 2023, Dai et al., 2024).
• Subset/Weighted Sampling: Each join tuple $u$ is included in the output subset independently with probability $p(u)=\mathcal{F}(p_1(t_1),...,p_k(t_k))$, where $k$0, $k$1, and $k$2 is a decomposable aggregation (e.g., product, sum, min, max) (Esmailpour et al., 18 Dec 2025).
• Streaming Reservoir Sampling: Maintain a sliding reservoir of $k$3 uniformly random join results as tuples are streamed into base relations (Dai et al., 2024).

Traditional approaches (e.g., full materialization with random selection) are computationally infeasible for large $k$4 ($k$5 exponential in $k$6). IBJS algorithms achieve practical and theoretically optimal runtime, avoiding join result materialization, and extending to both acyclic and cyclic schemas, as well as static and streaming settings (Kim et al., 2023, Esmailpour et al., 18 Dec 2025, Dai et al., 2024).

2. Index Structures and Preprocessing

IBJS techniques depend critically on precomputed index structures:

Approach	Index Type	Complexity
Dyadic-gap box (Welltris)	Trie of maximal gap boxes	$k$7 per relation
Trie/B-tree (AGM-IBJS)	Trie per attribute ordering	$k$8 per relation
Reservoir (streaming)	Dynamic join/partition index	$k$9 (acyclic), $R_1, ..., R_k$0 (cyclic)
Static subset (Poisson)	Join-tree counters (W, M)	$R_1, ..., R_k$1 space

• Dyadic box approach: For each $R_1, ..., R_k$2, enumerate all maximal dyadic gap boxes covering gaps in $R_1, ..., R_k$3 not present in $R_1, ..., R_k$4. Represent as a trie to support $R_1, ..., R_k$5 coverage tests (Abo-Khamis et al., 2020).
• Trie/B-tree indexing: For every relation, build a trie keyed by all attribute orders. Support constant-time degree, projection, access, existence, and uniform sampling over slices (Kim et al., 2023).
• Join-tree counters: For subset/weighted sampling, preprocess W and M counters bottom-up via dynamic programming and FFT-based convolution (Esmailpour et al., 18 Dec 2025).
• Dynamic join index: For streaming, maintain bucketed lists, prefix trees, and constant-dense mini-batches for efficient delta result handling on insertion (Dai et al., 2024).

Preprocessing typically requires $R_1, ..., R_k$6 time and $R_1, ..., R_k$7 space, where $R_1, ..., R_k$8 (Esmailpour et al., 18 Dec 2025). The complexity is scalable with schema arity and admits dynamic updates for reservoir sampling.

3. Core Algorithmic Principles

IBJS algorithms operate by an indexed, recursive, or rejection-based strategy, tailored to the sampling model:

• Uniform (Dyadic Gap): Maintain set $R_1, ..., R_k$9 of discovered gap boxes; iteratively sample candidate tuples from $\{A_1, ..., A_d\}$0 uniformly using a Klee-measure recursion; emit valid join tuples; upon failure to find new join tuples, update $\{A_1, ..., A_d\}$1 with all covering boxes for a missed candidate (Abo-Khamis et al., 2020).
• Degree-based Rejection (AGM-IBJS): Select random attribute, sample relevant edge, compute degrees in all incident relations, and accept with probability proportional to relative degree ratios and AGM bound. Recurse to fix the next attribute. The process ensures exact uniformity over all join results (Kim et al., 2023).
• Subset Sampling: Partition join results by bucketed score, and for each bucket, enable direct access to any join tuple by rank via a recursive join-tree traversal using W/M statistics. For each $\{A_1, ..., A_d\}$2-bucket, use geometric-jump skipping and rejection to simulate Poisson sampling (Esmailpour et al., 18 Dec 2025).
• Streaming Reservoir: Use a predicate-aware skip-based reservoir sampler (modulo dummy/real items) combined with dynamic join/partition indices. Each arriving tuple is indexed and processed to identify and enumerate new join contributions without explicit materialization (Dai et al., 2024).

These strategies can be combined with parallel, batched, and cache-efficient implementations, and admit extensions to generalized hypertree decompositions (GHDs) for cyclic joins (Kim et al., 2023, Esmailpour et al., 18 Dec 2025). All approaches ensure exact uniformity or subset-unbiasedness by construction.

4. Complexity Analysis and Optimality

IBJS achieves (near-)instance optimal complexity in key models:

Model	Single Sample	Batch $\{A_1, ..., A_d\}$3 Samples / Reservoir $\{A_1, ..., A_d\}$4	Preprocessing/Update
Dyadic (certificate)	$\{A_1, ..., A_d\}$5	$\{A_1, ..., A_d\}$6	$\{A_1, ..., A_d\}$7
AGM-IBJS	$\{A_1, ..., A_d\}$8	$\{A_1, ..., A_d\}$9 (per sample, up to $\{1,...,n\}^{|A_i|}$0 factors)	$\{1,...,n\}^{|A_i|}$1 per relation
Streaming reservoir	$\{1,...,n\}^{|A_i|}$2 (access)	$\{1,...,n\}^{|A_i|}$3 total maintenance	$\{1,...,n\}^{|A_i|}$4 per insertion
Subset (static index)	$\{1,...,n\}^{|A_i|}$5 per sample	$\{1,...,n\}^{|A_i|}$6 (one-shot)	$\{1,...,n\}^{|A_i|}$7

• Instance (certificate) optimality: Time matches the size of the minimal certificate covering the non-join region, up to polylogarithmic factors (Abo-Khamis et al., 2020).
• AGM optimality: Sampling time per tuple is $\{1,...,n\}^{|A_i|}$8, where AGM is the optimal fractional edge-cover bound; further improved to $\{1,...,n\}^{|A_i|}$9 under small join size and available GHD (Kim et al., 2023).
• Streaming optimality: Reservoir maintenance and access are $q$0 per join sample, with total update cost $q$1 and linear space (Dai et al., 2024).
• Subset/Poisson sampling: Each subset sample is $q$2 in expectation, where $q$3 is the expected sample size; dynamic maintenance is $q$4 amortized per insertion (Esmailpour et al., 18 Dec 2025).

These bounds are either instance-optimal or nearly tight within the respective computational models.

5. Correctness, Uniformity, and Extensions

IBJS guarantees:

• Exact uniformity: Every join tuple (or subset sample in the subset setting) is included with exactly the desired probability—uniform or weighted as specified. There is no approximation error or failure probability in the uniform case, and subset inclusion is independent per join result (Abo-Khamis et al., 2020, Kim et al., 2023, Esmailpour et al., 18 Dec 2025).
• Streaming unbiasedness: Reservoir contents at any time represent a uniform $q$5-subset over all join results seen so far, complying with the classical Vitter distribution extended to interleaved real/dummy candidates (Dai et al., 2024).
• Poisson subset sampling: Each join tuple's inclusion is independent, supporting downstream applications requiring unbiased estimators, bootstraps, or randomized structure learning (Esmailpour et al., 18 Dec 2025).

Extensions and variants encompass:

• General scoring functions: Subset sampling supports aggregation policies beyond product, including min, max, and sum, by modifying score bucketing and direct-access indexing (Esmailpour et al., 18 Dec 2025).
• Cyclic schemas: Replace join-trees with decompositions of bounded fractional hypertree width, with all complexity parameters scaling as $q$6 (Esmailpour et al., 18 Dec 2025).
• One-shot and batched modes: IBJS supports both statically indexed repeated queries and batched/lazy query plans for amortized cost savings (Esmailpour et al., 18 Dec 2025).

6. Empirical Observations and Applications

Empirical evaluations of IBJS have been conducted for static, streaming, and reservoir settings:

• Reservoir sampling over joins: Implementations demonstrate maintenance and sampling throughput orders of magnitude faster than prior art, with update time $q$7–$q$8s per tuple and memory usage $q$9–$J = R_1 \bowtie ... \bowtie R_k$0 of competing methods; KL-divergence from true join distribution is negligible (Dai et al., 2024).
• Static/one-shot subset sampling: Theoretical performance guarantees are confirmed, with preprocessing, memory, and query time scaling sublinearly in the size of the join output (Esmailpour et al., 18 Dec 2025).
• Applications: IBJS is utilized in analytic query answering, uniform data subsampling for machine learning, approximate query processing, join size estimation, online learning, and dynamic data analytics (Abo-Khamis et al., 2020, Kim et al., 2023, Esmailpour et al., 18 Dec 2025, Dai et al., 2024).

Significant speedups are reported over naive materialize-then-sample and Markov chain Monte Carlo methods in both well-certified static scenarios and highly dynamic streaming workloads.

7. Open Problems and Research Directions

Recent work highlights several ongoing challenges:

• Non-monotonic weights: Extending IBJS to support non-monotonic aggregate functions (e.g., median) for subset sampling remains unresolved (Esmailpour et al., 18 Dec 2025).
• Dynamic lower bounds: Characterizing lower bounds for subset sampling and uniformity under dynamic insertions and deletions is an open question (Esmailpour et al., 18 Dec 2025).
• System integration: Practical implementation and integration of IBJS in distributed systems (Spark, DukeDB), as well as scaling experiments beyond current workloads, constitute ongoing future work (Esmailpour et al., 18 Dec 2025).

A plausible implication is that as relational analytics scale in size and velocity, the adoption of IBJS variants for real-time, scalable, and provably unbiased join sampling will be increasingly central, provided further system-level optimizations and theoretical guarantees can be established.