Unlocking Fractional Moments in Delphic Set Streams

Abstract: We consider estimation of non-integer frequency moments $F_k$ and related Bernstein-type statistics in the Delphic set stream model under a bounded-frequency assumption: every universe element appears at most $τ$ times. The main challenge of this model is to keep space low while also keeping update time low, which is not trivial because the sets can be exponential in size compared to their representations. Our core insight is that by sampling the stream at different rates and observing the resulting distinct-counts, we can 'probe' the frequency distribution and numerically integrate these probes to reconstruct a broad class of statistics. Building on that, we crucially observe that the distinct-count of a randomly sampled substream, viewed as a function of the sampling rate, is a single analytic object whose evaluations determine a broad class of statistics via a complementary Laplace-type integral. Algorithmically we exploit this by: 1. estimating those evaluations using only standard $F_0$ (distinct-count) algorithms on sampled substreams and 2. recovering target statistics by controlled numerical integration on a judiciously chosen grid. For $F_k$ with $k\in (0,1)$ we obtain the first one-pass streaming algorithms for Delphic set streams whose space and per-set update time are $$\mathrm{poly}(\log|Ω|,\log m,\varepsilon<sup>{-1},\log(1/δ))$$ in the practically relevant regime $τ=\mathrm{polylog}(|Ω|,m)$; in general the bounds are polynomial in $τ$ and $\varepsilon<sup>{-1}$ and logarithmic in $δ<sup>{-1}$. We also give a complexity-theoretic barrier explaining why lower bounds for removing the bounded-frequency assumption appear difficult: ruling out polylogarithmic algorithms for unrestricted Delphic $F_k$ would imply a linear-space threshold-counting separation.

• The paper introduces the Expected Support Polynomial and complementary Laplace-transform integration to estimate fractional moments F_k for 0 < k < 1 without enumerating exponentially large sets.
• The algorithm simulates element subsampling through Delphic membership, size, and sampling oracles, achieving polylogarithmic space and update time when the frequency bound τ is polylogarithmic, although its ε- and τ-dependence is not optimal.
• The framework also estimates saturated richness, smoothed log-frequency aggregates, and broader Bernstein-type statistics, while identifying complexity-theoretic barriers to removing the bounded-frequency assumption.

Setting and problem statement

This paper studies frequency-moment estimation in the Delphic set-stream model, where each stream update is a set $S_i \subseteq \Omega$ equipped with efficient size, membership, and uniform-sampling oracles. The statistic of interest is $F_k = \sum_{x \in \Omega} f_x^k$, where $f_x$ is the number of sets containing $x$. The model strictly generalizes both ordinary data streams (singleton sets) and the Klee's measure setting (axis-parallel boxes), since a box containing exponentially many grid points is representable with only $O(d \log m)$ bits. The central difficulty is that any algorithm that enumerates the elements of a set incurs update time proportional to $|S_i|$, which can be exponential in the representation size; hence per-set update time must remain polylogarithmic.

Prior work resolved $F_0$: the estimator of Nandi et al. achieves $(\varepsilon, \delta)$-approximation with space and update time $\tilde O(\varepsilon^{-2} \log^2 |\Omega|)$ by simulating hash-based distinct-count sketches through subsampling, exploiting the Delphic sampling primitive. For $k \in (0,1)$, essentially nothing was known in this model, even under a bounded-frequency assumption. The paper closes this gap for $F_k = \sum_{x \in \Omega} f_x^k$0-frequency-bounded streams, where every element appears in at most $F_k = \sum_{x \in \Omega} f_x^k$1 sets.

Why existing techniques fail

Data-stream algorithms for fractional moments do not transfer. HyperLogLog-style extensions of Cohen require touching every stream element, and Indyk's $F_k = \sum_{x \in \Omega} f_x^k$2-stable distribution sketches require per-element linear updates $F_k = \sum_{x \in \Omega} f_x^k$3 over implicit sets, which cannot be computed from the Delphic oracles without enumeration. The $F_k = \sum_{x \in \Omega} f_x^k$4 breakthrough worked because its estimator is purely sampling-based, and sampling-based algorithms can in principle be ported via the Delphic sampling primitive. Fractional-moment estimators, by contrast, are sketch-based rather than sampling-based, so a new mechanism is required.

The sampling-plus-integration paradigm

The paper's central object is the Expected Support Polynomial (ESP). Perform element-level subsampling: each element $F_k = \sum_{x \in \Omega} f_x^k$5 is retained independently with probability $F_k = \sum_{x \in \Omega} f_x^k$6, yielding a thinned substream $F_k = \sum_{x \in \Omega} f_x^k$7. The expected distinct count of the thinned stream is

$F_k = \sum_{x \in \Omega} f_x^k$8

Under the change of variables $F_k = \sum_{x \in \Omega} f_x^k$9, the paper defines $f_x$0, which is exactly the complementary Laplace transform of the frequency measure $f_x$1. Since $f_x$2 is a Bernstein function for $f_x$3 with Lévy density $f_x$4, the identity

$f_x$5

holds. The estimator therefore reduces $f_x$6 estimation to: (i) evaluating $f_x$7 at a grid of points via black-box $f_x$8 estimation on simulated thinned substreams, and (ii) controlled numerical integration (trapezoidal rule) with truncation of the singular origin and the infinite tail. The truncation parameters $f_x$9, $x$0, and grid size $x$1 are set so that each error source (small-part truncation, tail truncation, trapezoidal discretization, and per-point estimation noise) contributes at most $x$2 of the total.

Two technical components make this work. First, the thinned sets are never materialized; the oracles of the subsampled set (membership, size, uniform sampling) are simulated directly from the mother set. Membership is answered by a coin flip after a Delphic membership query; the size oracle exploits a binomial decomposition lemma, answering with $x$3 conditioned on the frozen membership answers; uniform sampling uses rejection sampling. The paper proves that any $x$4 algorithm making at most one membership query per element, with membership queries preceding size queries preceding sampling queries, is statistically oblivious to the substitution. Second, the Las Vegas runtime of rejection sampling is handled by a global union bound over all sampling calls, adding only logarithmic factors.

Main results

For $x$5-frequency-bounded Delphic set streams, the main theorem gives a one-pass streaming algorithm for $x$6, $x$7, with space and per-set update time

$x$8

When $x$9, this is the first polylogarithmic-space, polylogarithmic-update algorithm for fractional moments in this model. The dependence on $O(d \log m)$0 is notably worse than the optimal $O(d \log m)$1 of the data-stream setting — the paper concedes that the general bounds are polynomial in $O(d \log m)$2 and $O(d \log m)$3, and frames the contribution as a feasibility result rather than an optimality result. The $O(d \log m)$4-dependence enters through two channels: bounding the small-$O(d \log m)$5 truncation error via the inequality $O(d \log m)$6, and bounding second derivatives of the integrand for the trapezoidal error analysis.

The same machinery yields the first algorithms for additional Bernstein-type statistics:

• Saturated Richness $O(d \log m)$7, via the Lévy density $O(d \log m)$8, in $O(d \log m)$9 space/time.
• Smoothed Log-Frequency Aggregate $|S_i|$0, via the density $|S_i|$1, with an additional $|S_i|$2 factor.
• General Bernstein functions with explicit, $|S_i|$3 non-negative Lévy densities: the identity $|S_i|$4 holds unconditionally, and the estimator's complexity is expressed in terms of the sup-norms $|S_i|$5 of $|S_i|$6 and its derivatives on $|S_i|$7.

For $|S_i|$8, the paper gives two algorithms. A modification of the AMS sampling scheme (sampling an element proportionally to set size, counting subsequent occurrences) achieves $|S_i|$9 space and update time, matching the data-stream bound up to $F_0$0. Separately, the analytic route uses the derivative identity $F_0$1, approximated by forward finite differences, yielding $F_0$2 — worse than the AMS modification, but demonstrating the flexibility of the ESP framework.

The complexity-theoretic barrier

The paper explains why removing the bounded-frequency assumption appears hard. Stable sketches would lift to unrestricted Delphic streams given a value aggregate oracle $F_0$3 computing $F_0$4 for implicit sets; with this oracle, the Delphic sketch produces exactly the counter vector of the expanded singleton stream. The associated threshold language $F_0$5 is shown to lie in $F_0$6, the polynomial-time, linear-space analogue of PP defined via $F_0$7. Consequently, if $F_0$8, then unrestricted Delphic $F_0$9 estimation (for $(\varepsilon, \delta)$0) admits a polylogarithmic-space, polylogarithmic-update algorithm. Contrapositively, any lower bound ruling out such algorithms would imply $(\varepsilon, \delta)$1 — a threshold-counting time-space separation. This is an $(\varepsilon, \delta)$2 analogue of the $(\varepsilon, \delta)$3 oracle barrier, with nondeterminism replaced by threshold counting. The paper is explicit that this does not prove the bounded-frequency assumption is necessary; it shows only that unconditional lower bounds require progress on a major open problem in complexity theory.

Limitations and open questions

The paper acknowledges several limitations at the point where they bear on the results. The $(\varepsilon, \delta)$4-dependence is inherent to the analytic reconstruction step, which uses the frequency cap both to truncate the singular integral near $(\varepsilon, \delta)$5 and to bound derivatives for the quadrature error; the method genuinely requires the cap. When the Lévy density of a Bernstein function is not explicit, Laplace inversion is needed, and the paper states plainly that numerical inversion methods (Bromwich integral, Post's formula) lack theoretical guarantees in this context — no $(\varepsilon, \delta)$6-multiplicative estimates of Laplace inversions are known in general, and their space complexity in streaming settings is understudied. The simulation theorem imposes structural constraints on the $(\varepsilon, \delta)$7 subroutine (query ordering, one membership query per element), so the framework is tied to estimators with that specific oracle access pattern. Open problems include removing or weakening the bounded-frequency assumption, improving the $(\varepsilon, \delta)$8-dependence toward the data-stream-optimal $(\varepsilon, \delta)$9, exploring alternative integral transforms (Fourier, Mellin), developing streaming-friendly approximate Laplace inversion, and extending to distributed streaming models.

Conclusion

The paper establishes the feasibility of estimating fractional frequency moments and a broad class of Bernstein-type statistics in Delphic set streams under a bounded-frequency assumption, using a unified sampling-plus-integration framework built on the complementary Laplace transform of the empirical frequency measure. Its algorithms are the first in this model for $\tilde O(\varepsilon^{-2} \log^2 |\Omega|)$0 with polylogarithmic resources when $\tilde O(\varepsilon^{-2} \log^2 |\Omega|)$1 is polylogarithmic, and its barrier result clarifies the complexity-theoretic obstruction facing any attempt to prove lower bounds for the unrestricted model. The main quantitative gap — polynomial rather than polylogarithmic dependence on $\tilde O(\varepsilon^{-2} \log^2 |\Omega|)$2 and $\tilde O(\varepsilon^{-2} \log^2 |\Omega|)$3 in general — remains open.