Integrate a statistics-aware distributed optimizer into TQP

Develop and integrate a statistics-aware distributed query optimizer for the Tensor Query Processor (TQP) that can automatically select data exchange operations (e.g., shuffle versus broadcast) and join orderings based on estimated cardinalities for distributed execution, replacing the current manual optimization of compiled tensor programs.

Background

Distributed TQP currently compiles queries into tensor programs and then relies on manual plan improvements (such as changing data exchange operations or join ordering) to achieve high performance. The authors note that accurate estimation of intermediate cardinalities is a longstanding hard problem and that optimizers often rely on estimates that can be wrong at runtime, leading to suboptimal plans or requiring runtime correction.

To demonstrate the upper bound on performance potential, the paper defers the design and integration of a statistics-aware distributed query optimizer. Making this integration explicit would automate plan choices that are currently applied by hand and align distributed TQP with common practice in MPP analytical databases.