Pessimistic Cardinality Estimation (2412.00642v1)

Abstract: Cardinality Estimation is to estimate the size of the output of a query without computing it, by using only statistics on the input relations. Existing estimators try to return an unbiased estimate of the cardinality: this is notoriously difficult. A new class of estimators have been proposed recently, called "pessimistic estimators", which compute a guaranteed upper bound on the query output. Two recent advances have made pessimistic estimators practical. The first is the recent observation that degree sequences of the input relations can be used to compute query upper bounds. The second is a long line of theoretical results that have developed the use of information theoretic inequalities for query upper bounds. This paper is a short overview of pessimistic cardinality estimators, contrasting them with traditional estimators.

• The paper introduces pessimistic cardinality estimation as a method that computes guaranteed upper bounds to enhance query optimization by avoiding underestimations.
• It leverages detailed degree sequences and ℓₚ-norm based statistics to achieve tighter bounds compared to traditional density-based approaches.
• The study demonstrates that using theoretical frameworks like the Chain and Polymatroid bounds can significantly improve estimation accuracy in complex queries.

An Overview of Pessimistic Cardinality Estimation

The research article provides a comprehensive analysis of pessimistic cardinality estimation (PCE) as an alternative method to traditional cardinality estimation techniques in database systems. While traditional methods focus on yielding unbiased estimates, PCE aims to compute guaranteed upper bounds on the output sizes of queries, which can be advantageous in query optimization scenarios where avoiding underestimation is critical.

The primary contribution of the paper is a detailed examination of PCE methods, contrasting them with density-based estimators that significantly rely on assumptions like data uniformity and independence. These traditional estimators often exhibit large errors, particularly for queries with numerous joins and predicates, and they lack theoretical guarantees. In contrast, PCE offers a one-sided theoretical guarantee by providing a bound that is always larger than or equal to the actual size.

The paper describes several approaches within PCE frameworks, particularly focusing on the utilization of degree sequences and advanced theoretical tools such as entropic inequalities. It notes two recent advances that have made PCE more practical: (1) leveraging degree sequences of input relations to compute query upper bounds, and (2) exploiting information-theoretic inequalities. These advances enable tighter bounds and, in some cases, surpass the performance of traditional cardinality estimation methods.

Degree Sequences and Norms

The concept of degree sequences offers a more refined set of statistics compared to simple cardinality counts. This involves considering the degrees of elements in relations, capturing the distribution of distinct values and their occurrences. Various norm-based statistics derived from these sequences form the basis for computing more nuanced upper bounds on query outputs. Particularly, the article highlights how $\ell_p$-norms (including $\ell_2$ and $\ell_\infty$) applied to degree sequences provide a means to traditional bounding techniques, such as the Chain Bound and Polymatroid Bound.

Theoretical Bounds

The paper discusses several theoretical frameworks for PCE, such as the AGM Bound, Chain Bound, and Polymatroid Bound, detailing how these leverage degree sequence statistics. The AGM Bound presents a fundamental yet basic method, while the Chain Bound and Polymatroid Bound extend these concepts by using more detailed degree information, thus achieving tighter cardinality bounds. Through linear programming frameworks, these bounds can be computed efficiently, albeit with some complexity considerations for larger queries.

Practical Implications and Future Directions

From a practical perspective, PCE can significantly improve query optimization by avoiding underestimated output sizes, which can lead to inefficient resource usage and potential system failures due to memory overruns. This is particularly beneficial in environments where ensuring system stability is critical and overestimated resources can be provisioned more safely.

However, there are computational considerations and implementation challenges. The article points to optimizations required for handling large queries, such as selective preprocessing of statistics, data-driven decisions on which statistics to maintain, and effective compression techniques for degree sequences.

Future research may explore more computationally efficient models and the application of PCE in distributed database settings. Incremental updates, a concern across all database statistics maintenance, also remain an open area for development, potentially leveraging sketching techniques to support incremental changes.

Overall, the paper situates PCE as a promising alternative to traditional cardinality estimation approaches, particularly for systems where ensuring pessimism in estimation aligns better with service-level goals than pursuing unbiased but occasionally inaccurate estimates.