Dice Question Streamline Icon: https://streamlinehq.com

Deterministic parallel counters space lower bound

Prove that any deterministic single-pass streaming algorithm for the k-parallel counters problem—where a stream of length n consists of increments i1, i2, ..., in in [k], each incrementing the specified counter by 1, and the algorithm must output at the end the count of each counter within additive error n/k—requires at least Ω(k log(n/k)) bits of memory.

Information Square Streamline Icon: https://streamlinehq.com

Background

The paper gives an O(1/ε)-word deterministic quantile sketch in the bounded-universe setting and discusses optimality. While Ω((1/ε) log U) bits is unavoidable, establishing an Ω((1/ε) log n) deterministic lower bound would show the space bound is optimal more generally.

To obtain such a lower bound, the authors propose the deterministic parallel counters conjecture. Any ε-approximate quantile sketch (Problem 1) can simulate k-parallel counters with k = Θ(1/ε), so proving Ω(k log(n/k)) bits for deterministic parallel counting would imply an Ω((1/ε) log n) deterministic lower bound for quantile sketching. They note related randomized lower bounds are known, but the deterministic case remains unproven.

References

We define the k-parallel counters problem as following: There are k counters initiated to 0. Given a stream of increments i_1, i_2, \dots, i_n \in [k] where i_t means to increment the i_t-th counter by 1, the algorithm has to output the final count of each counter up to an additive error of n / k.

We conjecture that any deterministic algorithm for this problem requires at least \Omega(k \log (n / k)) bits of memory.

Optimal quantile estimation: beyond the comparison model (2404.03847 - Gupta et al., 5 Apr 2024) in Discussion and further directions (Subsection 1.1), Conjecture: Deterministic parallel counters