A New Information Complexity Measure for Multi-pass Streaming with Applications (2403.20283v1)

Abstract: We introduce a new notion of information complexity for multi-pass streaming problems and use it to resolve several important questions in data streams. In the coin problem, one sees a stream of $n$ i.i.d. uniform bits and one would like to compute the majority with constant advantage. We show that any constant pass algorithm must use $\Omega(\log n)$ bits of memory, significantly extending an earlier $\Omega(\log n)$ bit lower bound for single-pass algorithms of Braverman-Garg-Woodruff (FOCS, 2020). This also gives the first $\Omega(\log n)$ bit lower bound for the problem of approximating a counter up to a constant factor in worst-case turnstile streams for more than one pass. In the needle problem, one either sees a stream of $n$ i.i.d. uniform samples from a domain $[t]$, or there is a randomly chosen needle $\alpha \in[t]$ for which each item independently is chosen to equal $\alpha$ with probability $p$, and is otherwise uniformly random in $[t]$. The problem of distinguishing these two cases is central to understanding the space complexity of the frequency moment estimation problem in random order streams. We show tight multi-pass space bounds for this problem for every $p < 1/\sqrt{n \log^3 n}$, resolving an open question of Lovett and Zhang (FOCS, 2023); even for $1$-pass our bounds are new. To show optimality, we improve both lower and upper bounds from existing results. Our information complexity framework significantly extends the toolkit for proving multi-pass streaming lower bounds, and we give a wide number of additional streaming applications of our lower bound techniques, including multi-pass lower bounds for $\ell_p$-norm estimation, $\ell_p$-point query and heavy hitters, and compressed sensing problems.

• The paper introduces a novel info complexity framework for multi-pass streaming, establishing memory lower bounds for crucial coin and needle problems.
• It proves that k-pass algorithms for the coin problem require Ω(log n/k) bits, highlighting the trade-off between pass count and memory usage.
• The findings extend to ℓ_p-norm estimation, heavy hitters, and compressed sensing, offering tighter bounds for efficient algorithm design.

A New Information Complexity Measure for Multi-pass Streaming with Applications

The paper investigates the challenges associated with multi-pass streaming problems, introducing a novel information complexity framework specifically tailored for such computations. By applying this framework, the authors address several key questions in data stream processing, focusing particularly on the coin problem and the needle problem.

In the coin problem, the task is to compute the majority of a stream of independent and identically distributed (i.i.d.) uniform bits. The authors extend prior work to prove that any algorithm requiring multiple passes over the data stream must use $\Omega(\frac{\log n}{k})$ bits of memory, where $k$ denotes the number of passes. This result broadens existing lower bounds that applied only to single-pass algorithms. The research implies substantial increases in memory usage with even modest increases in allowed passes, highlighting the inefficiencies introduced when attempting to bypass memory constraints via additional passes.

For the needle problem, the goal is to distinguish between two distributions: one that is uniformly random and another that includes a "needle" element that appears with higher frequency, interspersed among uniform samples. This problem is critical in understanding the space complexity of frequency moment estimation. The authors succeed in closing a significant gap in multi-pass space bounds, achieving tight bounds that had previously been open.

The proposed multi-pass information complexity measure captures the essence of multi-pass streaming challenges. It leverages conditional mutual information, considering the data processing constraints unique to streaming, where the inputs are processed in sequence rather than all at once. By rigorously exploring these dependencies, the authors identify a path to create lower bounds for more complex streaming scenarios.

The paper extends its results beyond these primary problems, demonstrating the strength of its new framework across a variety of applications, including:

• Multi-pass bounds for $\ell_p$-norm estimation: Specifically, the framework applies to estimating norms across data streams where space complexity has traditionally been loosely bounded.
• $\ell_p$-point query and heavy hitters: The model provides a new lens through which to view classic data stream problems, offering tighter and more detailed bounds than prior frameworks.
• Compressed sensing problems: The novel framework incorporates techniques from compressed sensing, showcasing its broad applicability beyond standard streaming challenges.

From a theoretical standpoint, this work enriches the discourse on data streams by exploring the limits of what can be computed under strict space and pass constraints. Practically, it lays the groundwork for improved algorithm designs in real-time data processing systems, where resource constraints necessitate extremely efficient computations.

The results pivot on bridging streaming information complexity with foundational problems in computing, making substantial contributions to both theory and application. It challenges the prevailing assumptions about the power of multiple passes in stream processing and provides nuanced insights into polynomially greater memory requirements. As the algorithms that process large-scale data continue to evolve, frameworks like the one presented will be instrumental in guiding future technology toward more efficient and powerful data handling methods.