OmniSketch: Multi-Dimensional Stream Synopsis

• OmniSketch is a streaming synopsis data structure that efficiently summarizes multi-dimensional, high-velocity data streams for OLAP-style queries with arbitrary conjunctive predicates.
• It combines Count-Min sketches with bottom-k minwise sampling to provide scalable, memory-efficient, and probabilistically accurate estimations while dynamically supporting filters on various attributes.
• Extensions of OmniSketch enable join cardinality estimation, offering significant throughput improvements and controlled error profiles over prior approaches such as Hydra in high-dimensional settings.

OmniSketch is a streaming synopsis data structure capable of efficiently summarizing multi-dimensional, high-velocity data streams for OLAP-style analytic queries with arbitrary conjunctive predicates. It provides a scalable, memory-efficient, and probabilistically accurate count-aggregate estimation over streams with multiple attributes, dynamically supporting filters on arbitrary subsets of attributes selected at query time, and extends to enable join cardinality estimation across tables in relational systems (Punter et al., 2023, Justen et al., 25 Aug 2025).

1. Motivation and Limitations of Prior Synopses

OmniSketch addresses key limitations of traditional streaming synopses—particularly Count-Min (CM) sketch and related structures—which primarily support aggregate queries over a single attribute and are intractable for general multi-attribute predicate queries. Handling $p$ predicates over $|A|$ attributes in standard approaches would require $2^p-1$ sketches or exponential per-update cost, which is prohibitive for even moderate values of $p$ (e.g., $p \geq 10$). These approaches are also incapable of efficiently supporting queries with “don’t-care” attributes without incurring exponential space or computational blowup (Punter et al., 2023).

2. Data Structure Overview, Initialization, and Update

OmniSketch consists of $|A|$ independent attribute sketches, each adopting a two-level structure:

• An outer Count-Min sketch per attribute $a_i$, comprising a $d \times w$ array of cells ($d$: depth, $w$: width).
• Each cell $|A|$0 maintains:
  • A counter $|A|$1,
  • A minwise sample $|A|$2 (“bottom-$|A|$3” hash values of record IDs), and
  • The largest hash in the sample $|A|$4.

During initialization, for user-supplied error ($|A|$5) and confidence ($|A|$6), parameters are set as:

• $|A|$7,
• $|A|$8,
• $|A|$9,
• $2^p-1$0. Each cell’s sample has size $2^p-1$1.

For each new record arrival:

• For each attribute, and each row $2^p-1$2:
  • Hash the attribute value to column $2^p-1$3, increment $2^p-1$4.
  • Hash record ID to $2^p-1$5, maintain the $2^p-1$6 smallest such hashes in $2^p-1$7 (using red-black tree structures for $2^p-1$8 updates).

This construction ensures per-record update cost is $2^p-1$9, and total memory $p$0 (Punter et al., 2023).

3. Query Processing and Estimation Guarantees

To estimate the count of records matching a conjunction of $p$1 arbitrary predicates, for example:

$p$2,

OmniSketch:

1. Identifies, for each predicate, the corresponding cells in the attribute CM sketches.
2. Computes, over all involved cells, the maximum counter $p$3.
3. Intersects all corresponding bottom-$p$4 samples to yield $p$5 of size $p$6.
4. Returns estimator $p$7.

This process leverages two main sources of accuracy:

• Outer CM hashing bounds hash-collision false positives,
• Inner K-minwise samples yield an unbiased estimate of intersection size.

Theoretical guarantees (with properly chosen $p$8, $p$9, $p \geq 10$0) state that with probability at least $p \geq 10$1:

$p \geq 10$2

where $p \geq 10$3 is the stream length. When the intersection sample $p \geq 10$4 is too small, the estimator falls back to an upper bound with a controlled one-sided error (Punter et al., 2023).

4. Space, Time Complexity, and Parameterization

• Space complexity: For $p \geq 10$5 attributes, space is $p \geq 10$6 bits; more succinctly $p \geq 10$7.
• Update cost: $p \geq 10$8 per stream record.
• Query cost: $p \geq 10$9 worst-case per query, but typically lower due to early pruning in intersection.
• Parameter selection: $|A|$0, $|A|$1, $|A|$2 maximized given memory $|A|$3. Increasing $|A|$4 lowers sampling error, increasing $|A|$5 mitigates collision error.

A quantitative trade-off is provided between memory, accuracy, and throughput. Empirical measurements confirm that, for up to $|A|$6 attributes, ingest throughput remains over $|A|$7 rec/s and accuracy is within the configured $|A|$8 across several real and synthetic datasets (Punter et al., 2023).

5. Comparison with Existing Approaches

Hydra, the prior state-of-the-art for multi-attribute stream synopses, maintains a single CM sketch with “universal” summaries in each cell, necessitating $|A|$9 update complexity per record. Experimentally, Hydra incurs both exponential update time ($a_i$0 ms/update for $a_i$1) and exponential hash-collision errors for larger $a_i$2, rendering it impractical for high-velocity or high-dimensional streams.

OmniSketch achieves $a_i$3 per-update complexity, comparable or superior accuracy, and update throughput two to three orders of magnitude higher for $a_i$4. Error profiles remain stable under skew (e.g., Zipfian), and in range query settings, extension via a logarithmic number of dyadic CM sub-sketches yields range-count error at most $a_i$5 with negligible query-time overhead (Punter et al., 2023).

6. Extensions: Join Cardinality Estimation

The OmniSketch framework has been extended to support join cardinality estimation for cost-based query optimization in relational databases (Justen et al., 25 Aug 2025). It merges count-min sketches and K-minwise hashing to estimate multi-way join sizes, without uniformity or independence assumptions.

Join Cardinality Algorithms

• PK-Sample-Join: Samples primary-key side using OmniSketch, computes the inclusion probability $a_i$6, and for each sample, probes the foreign-key side. The overall join estimate is $a_i$7. This approach is sensitive to key-skew and composes poorly over multiple joins due to “witness loss.”
• Secondary Sketches: Constructs secondary OmniSketches on dimension tables, reprojecting fact-side samples onto the secondary sketch’s sample space, which enables a join over multiple tables to reduce to a single intersection of sample sets. This allows efficient join-cardinality estimation, especially in star schemas.
• Interoperability: All sketches use unified hash functions and root seeds, making probe and sample composition across tables possible.
• Acyclic Join Graphs: An algorithm, based on GYO-ear-removal, sequences probe and intersection steps to guarantee witness flow and correctness in α-acyclic join structures.

Empirical Evaluation

• On SSB-skew (star schema), secondary-sketch OmniSketch yields reductions of up to $a_i$8 in intermediate result sizes and speedups up to $a_i$9 in total execution times compared to DuckDB’s cost-based optimizer.
• On JOB-light (general schema), performance is mixed due to high foreign-key cardinalities and complex join graphs causing frequent fallback events.
• Core strength is in avoiding independence/uniformity assumptions, but performance degrades when compounded sampling probabilities become too small (Justen et al., 25 Aug 2025).

7. Practical Considerations and Future Directions

When deploying OmniSketch, practical parameter selection requires balancing memory constraints, target accuracy $d \times w$0, and query throughput demands. Open challenges and extensions include:

• Deletions/turnstile streams: Possible with two-level/bottom-$d \times w$1 sketches; incurs modest update overhead.
• Very high dimensionality $d \times w$2: Potentially addressed via adaptive attribute selection or hierarchical sketching.
• Skew adaptation: Detecting and reallocating buckets/samples for skewed attributes at runtime.
• Range queries: Supported via dyadic decomposition, with logarithmic space/time overhead per attribute.

A plausible implication is that OmniSketch generalizes efficiently to high-dimensional, heterogeneous streaming and relational-analytic workloads with minimal modification, provided that attribute-range domains and sampling scales remain within practical sketch-memory budgets (Punter et al., 2023, Justen et al., 25 Aug 2025).