OmniSketch Join Estimator

• The OmniSketch Join Estimator is a sketch-based approach that approximates join cardinality without repeated scans by using reusable synopses and dynamic sketch combinations.
• It leverages two architectures: one using sketched Sum-Product Networks for on-the-fly, predicate-conditioned estimation, and the other combining Count-Min Sketches with k-minwise hashing for multi-join queries.
• Empirical results demonstrate significant reductions in computation time with manageable error rates, making it practical for adaptive query planning in cost-based optimizers.

OmniSketch Join Estimator denotes a class of sketch-based methods for join cardinality estimation in relational query optimization. Its central objective is to estimate the output size of binary and multi-way joins, often under selection predicates, without relying on repeated scans of base tables or on strong uniformity and independence assumptions. Recent work uses the term in two closely related ways: as a conceptual label for sketched Sum-Product Networks (SPNs) that approximate selection-conditioned join sketches on-the-fly, and as an extension of the OmniSketch synopsis combining Count-Min Sketches and K-minwise hashing for alpha-acyclic multi-join estimation (Tsan et al., 16 Jun 2025, Justen et al., 25 Aug 2025).

1. Problem setting and estimator objective

Join cardinality estimation concerns the size of a relational join result, which is a direct proxy for the computational cost faced by a cost-based optimizer during plan enumeration. For a two-way equi-join on a single domain $I$, the exact size is

$$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$

where $f_k(i)$ is the frequency of value $i$ in relation $T_k$. For an $n$-way equi-join with multiple join attributes and join graph edge set $E$, the exact size can be written as

$\left|T_1 \Join \cdots \Join T_n\right| \;=\; \sum_{i \in I_1 \times \cdots \times I_n} f_1(i)\cdots f_n(i) \prod_{\{u,v\}\in E}\mathds{1}_{i_u = i_v}.$

This formulation makes clear that join size estimation is an inner-product or multilinear aggregation problem over relation-specific frequency vectors or tensors. The practical difficulty is that optimizers require many such estimates for subplans, often under changing filters. Classical sketch pipelines can estimate joins accurately, but they are typically built for predefined selections and therefore require rescanning or reconstructing sketches when previously unseen predicates appear. The OmniSketch line of work addresses precisely this gap: reusable synopses are constructed offline, and query-time estimation is reduced to sketch combination, witness intersection, or on-the-fly sketch approximation rather than base-table rescans (Tsan et al., 16 Jun 2025).

2. Two architectural interpretations of OmniSketch

One interpretation is architectural rather than nominative. In "Sketched Sum-Product Networks for Joins" (Tsan et al., 16 Jun 2025), sketched SPNs act as an OmniSketch Join Estimator by learning one SPN per relation, storing sketches at leaf nodes, and producing approximate sketches for arbitrary selections at query time. The paper explicitly positions this as a unified, general-purpose sketch-based estimator that supports binary and multi-way joins, adapts to new queries without predefined selections, and integrates with Fast-AGMS or Bound Sketch downstream.

The second interpretation is a named synopsis. "Join Cardinality Estimation with OmniSketches" (Justen et al., 25 Aug 2025) extends OmniSketch itself to multi-join estimation by combining Count-Min Sketch counters with bottom-$k$ samples of hashed primary-key identifiers. In this formulation, counters provide multiplicity scale and bottom-$k$ samples provide overlap evidence, or “witnesses,” across tables. Cross-table interoperability is achieved by using a single 64-bit hash $$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$0 both for value hashing and for primary-key identifier hashing, so that witnesses extracted from one table can be probed directly into another table’s sketch.

The two lines differ in mechanism but share the same systems goal. Sketched SPNs approximate a predicate-conditioned sketch and then invoke a join sketch estimator. OmniSketches based on Count-Min and K-minwise hashing estimate predicates and joins directly from witness intersections and counter-based scaling. A plausible implication is that “OmniSketch Join Estimator” is best understood as a systems pattern: a reusable synopsis layer that supports dynamic, optimizer-facing join estimation across large query workloads.

3. Sketched SPNs as an on-the-fly join-sketch estimator

In the SPN-based variant, each relation is modeled as a multivariate distribution represented by a rooted DAG whose leaves store univariate PDFs or sketches, and whose internal nodes are sums or products. Sum nodes encode mixtures with non-negative weights summing to $$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$1; product nodes encode factorization over locally independent scopes. Structure learning is recursive: a single-attribute scope, or a scope containing only join attributes, becomes a leaf; decomposable scopes become product nodes; otherwise tuples are clustered and represented by a sum node.

The training recurrence is

$$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$2

At query time, the SPN receives a predicate $$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$3 and a requested join-attribute set $$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$4. Leaves either return the sketch of $$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$5 or the selectivity $$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$6; product nodes multiply child outputs element-wise; sum nodes either sum child sketches directly or sum them weighted by cluster proportion. The result is an approximate root-level sketch for $$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$7, written as $$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$8.

This mechanism is designed to support arbitrary selections without precomputing per-selection synopses. Leaves store Fast-AGMS sketches or Count-Min/degree components for Bound Sketch, and because the sketches are linear and hash-aligned, the SPN can compose them element-wise to approximate the sketch required by the join estimator. For multi-way joins, cross-correlation enables vector-only implementations and avoids tensor storage; estimation across $$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$9 relations then costs $f_k(i)$0 due to FFT-based computation (Tsan et al., 16 Jun 2025).

The empirical results reported for this design emphasize the time advantage of approximation over exact sketch construction. For Fast-AGMS with width $f_k(i)$1, exact total sketching time versus approximation was $f_k(i)$2 versus $f_k(i)$3 on JOB-light and $f_k(i)$4 versus $f_k(i)$5 on Stats-CEB. For Bound Sketch, the corresponding figures were $f_k(i)$6 versus $f_k(i)$7 on JOB-light and $f_k(i)$8 versus $f_k(i)$9 on Stats-CEB. Approximate q-error remained higher than exact sketches but approached them as clustering thresholds tightened to $i$0 and $i$1; the independence threshold was reported as optimally set to $i$2. When Fast-AGMS used maximum-of-$i$3 aggregation and the upward-biased min-product node, total query execution times were within $i$4 of the fastest ground truth on JOB-light and Stats-CEB (Tsan et al., 16 Jun 2025).

4. OmniSketches based on Count-Min Sketches and K-minwise hashing

The OmniSketch extension in (Justen et al., 25 Aug 2025) maintains, for each attribute $i$5 of table $i$6, a Count-Min-like array of depth $i$7 and width $i$8. Each cell stores a counter $i$9 and a bottom-$T_k$0 sample $T_k$1 of hashed primary-key identifiers. Updates are driven by a shared 64-bit hash $T_k$2. For an inserted record with attribute value $T_k$3 and primary key $T_k$4, the value hash $T_k$5 is split into two 32-bit parts and used to derive row hash functions $T_k$6; the primary-key hash $T_k$7 is inserted into bottom-$T_k$8 samples in the addressed cells. This unification of value hashing and identifier hashing is the basis of sketch interoperability across tables.

For a single-table predicate, OmniSketch probes the relevant cells, intersects their bottom-$T_k$9 samples to obtain a witness set $n$0, and takes the maximum touched counter as $n$1. The estimator is

$n$2

Witnesses are hashed primary keys that survive all relevant intersections. They serve as direct overlap evidence. In a two-table join $n$3 with predicate $n$4 on $n$5, OmniSketch first probes $n$6 to obtain witnesses and a predicate estimate, computes the empirical sampling probability

$n$7

then probes each witness into $n$8’s foreign-key sketch and rescales the sum of per-witness contributions by $n$9:

$E$0

For star schemas, the paper introduces secondary sketches. Dimension-table sketches are built so that their bottom-$E$1 samples directly contain fact-table primary-key witnesses, which reduces both latency and the dependence on the representativeness of a random witness sample. In that case, estimation reduces to a single intersection in a common primary-key witness domain,

$E$2

The multi-way algorithm applies to alpha-acyclic join graphs through a directed traversal inspired by ear removal. Predicates on a node are applied before witnesses are probed into neighboring foreign-key sketches. When a primary-key side fans out to multiple foreign keys, the estimator performs a primary-key expansion: witnesses are probed into one foreign-key sketch, filtered by matches, and then propagated further. The method can also use secondary sketches in these expansion steps (Justen et al., 25 Aug 2025).

Empirically, this variant shows a marked workload dependence. On SSB-skew, with parameters $E$3, $E$4, $E$5 for the fact table and $E$6 for dimension tables, primary sketches occupied about $E$7, and adding secondary sketches raised this to about $E$8. The longest end-to-end estimation time was $E$9 with primary sketches only and $\left|T_1 \Join \cdots \Join T_n\right| \;=\; \sum_{i \in I_1 \times \cdots \times I_n} f_1(i)\cdots f_n(i) \prod_{\{u,v\}\in E}\mathds{1}_{i_u = i_v}.$0 with secondary sketches. Total intermediates decreased from $\left|T_1 \Join \cdots \Join T_n\right| \;=\; \sum_{i \in I_1 \times \cdots \times I_n} f_1(i)\cdots f_n(i) \prod_{\{u,v\}\in E}\mathds{1}_{i_u = i_v}.$1 to $\left|T_1 \Join \cdots \Join T_n\right| \;=\; \sum_{i \in I_1 \times \cdots \times I_n} f_1(i)\cdots f_n(i) \prod_{\{u,v\}\in E}\mathds{1}_{i_u = i_v}.$2, with reductions up to $\left|T_1 \Join \cdots \Join T_n\right| \;=\; \sum_{i \in I_1 \times \cdots \times I_n} f_1(i)\cdots f_n(i) \prod_{\{u,v\}\in E}\mathds{1}_{i_u = i_v}.$3, and execution time decreased from $\left|T_1 \Join \cdots \Join T_n\right| \;=\; \sum_{i \in I_1 \times \cdots \times I_n} f_1(i)\cdots f_n(i) \prod_{\{u,v\}\in E}\mathds{1}_{i_u = i_v}.$4 to $\left|T_1 \Join \cdots \Join T_n\right| \;=\; \sum_{i \in I_1 \times \cdots \times I_n} f_1(i)\cdots f_n(i) \prod_{\{u,v\}\in E}\mathds{1}_{i_u = i_v}.$5, with individual improvements up to $\left|T_1 \Join \cdots \Join T_n\right| \;=\; \sum_{i \in I_1 \times \cdots \times I_n} f_1(i)\cdots f_n(i) \prod_{\{u,v\}\in E}\mathds{1}_{i_u = i_v}.$6. On JOB-light, using $\left|T_1 \Join \cdots \Join T_n\right| \;=\; \sum_{i \in I_1 \times \cdots \times I_n} f_1(i)\cdots f_n(i) \prod_{\{u,v\}\in E}\mathds{1}_{i_u = i_v}.$7, $\left|T_1 \Join \cdots \Join T_n\right| \;=\; \sum_{i \in I_1 \times \cdots \times I_n} f_1(i)\cdots f_n(i) \prod_{\{u,v\}\in E}\mathds{1}_{i_u = i_v}.$8, and $\left|T_1 \Join \cdots \Join T_n\right| \;=\; \sum_{i \in I_1 \times \cdots \times I_n} f_1(i)\cdots f_n(i) \prod_{\{u,v\}\in E}\mathds{1}_{i_u = i_v}.$9 across all tables for a total of about $k$0, the longest end-to-end estimation time was $k$1, but the method largely suffered from loss of witnesses and unfavorable join graph shapes, producing systematic over-estimation for joins with at least three relations (Justen et al., 25 Aug 2025).

5. Estimator backbones, guarantees, and computational trade-offs

The SPN-based and CMS-plus-bottom-$k$2 variants rely on different statistical backbones. In the SPN line, Fast-AGMS provides unbiased estimators when sketches share aligned hash functions and the random sign function $k$3 is pairwise independent. For two-way joins, $k$4; for transitive three-way joins, $k$5; and for general multi-way joins, circular cross-correlation is used to avoid tensor storage. Bound Sketch instead yields pessimistic upper bounds, such as

$k$6

with analogous three-way upper-bound expressions. The SPN paper also states a conjectured $k$7 error bound for linear sketches, namely that the approximation error of the selection-conditioned sketch is bounded by the error incurred by fully assuming independence and scaling by exact univariate selectivities (Tsan et al., 16 Jun 2025).

A second backbone is Count Sketch with circular convolution and circular cross-correlation. "Convolution and Cross-Correlation of Count Sketches Enables Fast Cardinality Estimation of Multi-Join Queries" proves that a Count Sketch-based multi-join estimator is unbiased and has the same error guarantees as the AMS-based method, while reducing update time to be independent of sketch width $k$8 and evaluating multi-joins in $k$9 via FFT. For acyclic join graphs, the estimator satisfies

$k$0

matching the AMS-based baseline. At $k$1 memory, the reported throughput was $k$2 tuples/s for AMS, $k$3 tuples/s for COMPASS (partition), $k$4 tuples/s for COMPASS (merge), and $k$5 tuples/s for the Count Sketch method (Heddes et al., 2024).

The CMS-plus-bottom-$k$6 OmniSketch variant has a different guarantee profile. It inherits standard Count-Min bounds for point frequencies and standard bottom-$k$7 variance approximations for Jaccard estimation, but the paper does not provide formal error bounds for the multi-join PK-sample or expansion-based estimators. Its dominant failure mode is witness depletion: repeated intersections can drive witness sets to zero, forcing fallback heuristics such as minimal non-zero estimates or selectivity multiplication (Justen et al., 25 Aug 2025).

These differences yield distinct trade-offs. Fast-AGMS and Count Sketch formulations have explicit unbiasedness and variance statements. Bound Sketch provides upper bounds but may become loose under filters or heavy skew. CMS-plus-bottom-$k$8 OmniSketch targets plan quality through direct overlap evidence rather than through a closed-form multi-join variance theorem, and therefore performs best when witness preservation is structurally favorable, especially in star schemas with secondary sketches.

6. Optimizer integration, limitations, and relation to adjacent estimators

In optimizer integration, the SPN-based design exposes a cardinality API at the subplan level: for each relation $k$9, selection predicate $$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$00, and join attributes $$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$01, the system returns an approximate selection-conditioned sketch $$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$02; relation-level sketches are then combined through Fast-AGMS cross-correlation or Bound Sketch formulas to obtain a scalar join estimate. Equality and range predicates are supported, with ranges decomposed into dyadic intervals. Incremental updates are possible because SPNs can be updated locally and leaf sketches are mergeable (Tsan et al., 16 Jun 2025).

The CMS-plus-bottom-$$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$03 OmniSketch variant plugs into a conventional cost-based optimizer differently. It replaces the cardinality estimator itself: single-table estimates use $$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$04, while join estimates are computed by the alpha-acyclic traversal, PK-sample joins, primary-key expansions, or secondary-sketch intersections. The design preserves the standard dynamic-programming plan enumeration pipeline and avoids model retraining, but its accuracy depends heavily on join graph shape and witness survival (Justen et al., 25 Aug 2025).

The broader research landscape clarifies where OmniSketch-style estimators fit. "Instance Optimal Join Size Estimation" gives the Welltris estimator, which is not sketch-based: it preprocesses dyadic gap boxes, works with certificate size rather than input size or join size, and returns a one-sided estimate $$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$05 with probability at least $$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$06, while also supporting uniform sampling from the join result. This places Welltris in an indexed, certificate-based regime rather than a synopsis-based one (Abo-Khamis et al., 2020). "Sketches-based join size estimation under local differential privacy" adapts Fast-AGMS-style sketches to the client-side privacy setting using a Hadamard mechanism, and "Similarity Join and Similarity Self-Join Size Estimation in a Streaming Environment" develops one-pass Fast-AGMS-based estimators for similarity joins through projection streams and combinatorial correction (Zhang et al., 2024, Rafiei et al., 2018).

Several limitations recur across the OmniSketch literature. SPN factorizations still depend on local independence assumptions, so strong residual dependence can increase bias. Bound Sketch can overestimate heavily, especially under skew. CMS-plus-bottom-$$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$07 OmniSketch can lose witnesses under many intersections, high-degree primary-key nodes, or large foreign-key domains, and the paper explicitly reports that JOB-light largely suffers from a loss of witnesses due to unfavorable join graph shapes and large numbers of unique values in foreign key columns. A plausible synthesis is that OmniSketch Join Estimator is most effective when synopsis reuse, query-time adaptability, and low-latency optimizer integration are more important than exactness, and when the workload structure aligns with the estimator’s chosen evidence mechanism—linear sketch composition in the SPN case, or witness preservation in the bottom-$$\left|T_1 \Join T_2\right| \;=\; \sum_{i\in I} f_1(i)\,f_2(i),$$08 case (Justen et al., 25 Aug 2025).