On the (Generative) Linear Sketching Problem

Abstract: Sketch techniques have been extensively studied in recent years and are especially well-suited to data streaming scenarios, where the sketch summary is updated quickly and compactly. However, it is challenging to recover the current state from these summaries in a way that is accurate, fast, and real. In this paper, we seek a solution that reconciles this tension, aiming for near-perfect recovery with lightweight computational procedures. Focusing on linear sketching problems of the form $\boldsymbolΦf \rightarrow f$, our study proceeds in three stages. First, we dissect existing techniques and show the root cause of the sketching dilemma: an orthogonal information loss. Second, we examine how generative priors can be leveraged to bridge the information gap. Third, we propose FLORE, a novel generative sketching framework that embraces these analyses to achieve the best of all worlds. More importantly, FLORE can be trained without access to ground-truth data. Comprehensive evaluations demonstrate FLORE's ability to provide high-quality recovery, and support summary with low computing overhead, outperforming previous methods by up to 1000 times in error reduction and 100 times in processing speed compared to learning-based solutions.

• The paper identifies orthogonal irrecoverability in sparse hashing sketches and shows that recovering an s-sparse vector can require O(s²) measurements instead of O(s log(N/s)), motivating learned priors.
• The paper introduces FLORE, a conditional flow-based system that combines latent-space invertible modeling, EM-refined pseudo-labels, consistency, orthogonality, sparsity losses, and heavy-hitter filtering without adding data-plane overhead.
• The paper reports up to 1,000× lower frequency-estimation error, 90–100% heavy-hitter F1 at 64 KB, and substantially faster decoding than learned or optimization-based baselines, while noting limitations in point queries, key discovery, and low-skew streams.

Overview and problem formulation

The paper addresses the linear sketching problem for data streams: a stream of tuples $(\text{key}_t, v_t)$ is summarized into counters $b = \boldsymbol{\Phi}f$, where $f \in \mathbb{C}^N$ is the frequency vector over a known key set $\mathcal{I}$, and $\boldsymbol{\Phi}$ is the sparse $(0,1)$ indicator matrix induced by hashing. The goal is to recover statistics of $f$—per-element frequencies, heavy hitters, distribution, entropy—from $b$ accurately and quickly. The authors cast recovery as a compressive-sensing (CS) inverse problem, $\min_{\boldsymbol{\Phi}f=b}\|f\|_1$, and analyze why existing sketch families fail to solve it well. A key assumption is that the key set is known up front; when it is not, they add a Bloom Filter in the data plane and report zero estimates for untracked keys.

The paper's central diagnostic result is what it calls orthogonal irrecoverability: via range-null space decomposition ($f = f_{\Phi} + f_N$ from the SVD of $b = \boldsymbol{\Phi}f$0), only $b = \boldsymbol{\Phi}f$1 is captured by $b = \boldsymbol{\Phi}f$2, while the null-space component $b = \boldsymbol{\Phi}f$3 is orthogonal to $b = \boldsymbol{\Phi}f$4 and provably irrecoverable from the summary alone. This motivates supplying external prior information to "infill" the null space—the role assigned to generative models.

Why existing techniques fall short

The analysis covers three classes of prior methods:

• Randomized approximation (Count-Min, Count Sketch): per-element estimates from $b = \boldsymbol{\Phi}f$5 hash probes; error scales as $b = \boldsymbol{\Phi}f$6 or $b = \boldsymbol{\Phi}f$7 and is bounded by the counter budget.
• Disentangled augmentation (Augmented Sketch, learning-augmented sketches): splitting $b = \boldsymbol{\Phi}f$8's columns between heavy and light parts reduces underdetermination but the light part retains large bounds under fixed total memory.
• CS-based decoding (PR-sketch, NZE-sketch): simultaneous estimation of all elements with an $b = \boldsymbol{\Phi}f$9-error bound $f \in \mathbb{C}^N$0, tighter than the above, but with high query cost and no practical gain because the sketching matrices violate the restricted isometry property (RIP).

Empirically, the paper measures RIP distance and finds that sketch-induced matrices exhibit RIP distances exceeding 120 while classical CS matrices (Bernoulli, Fourier, Gaussian, Incoherence) stay below 0.3. Theorem 2 quantifies the consequence: recovering an $f \in \mathbb{C}^N$1-sparse vector requires $f \in \mathbb{C}^N$2 measurements for random dense matrices but $f \in \mathbb{C}^N$3 for $f \in \mathbb{C}^N$4-sparse matrices like Count-Min's—a quadratic penalty that makes CS-style perfect recovery infeasible at streaming memory budgets. This is the structural root cause behind the orthogonal loss: sparse hashing matrices cannot compress without discarding recoverable information unless priors are injected.

Generative priors and model selection

The proposed remedy maximizes the likelihood $f \in \mathbb{C}^N$5 by training a generator $f \in \mathbb{C}^N$6 with latent noise $f \in \mathbb{C}^N$7. Theorem 3 establishes feasibility: for a rank-$f \in \mathbb{C}^N$8 matrix $f \in \mathbb{C}^N$9 there exists a Gaussian vector $\mathcal{I}$0 and a learnable mapping such that $\mathcal{I}$1 uniquely determines $\mathcal{I}$2—the null-space component can be generated rather than recovered.

A systematic comparison of VAEs, GANs, diffusion models, and flow-based generative models (FGMs) on synthetic streams (1K–100K keys) shows that all fit low-cardinality distributions, but GANs are unstable at scale, VAEs degrade beyond ~10K keys due to their loose variational objective, and diffusion models require seconds per sample (~10³× slower), ruling them out for streaming latencies despite strong distribution fidelity. FGMs are selected as the backbone on the strength of their trade-off among fidelity, inference speed, and trainability.

FLORE design

FLORE (FLow-based Orthogonal REcovery) has four components. Invertible architecture: a conditional invertible neural network (cINN) built from NICE-style affine coupling blocks with permutation layers, plus two autoencoders that project $\mathcal{I}$3 and $\mathcal{I}$4 into a shared lower-dimensional latent space so dimensionality does not grow linearly with stream size; a segmented conditioning scheme shares parameters across key-space partitions so newly added dimensions require fine-tuning of only the latest segment, reducing parameter complexity from $\mathcal{I}$5 to segment length (over 200× reduction at 500K keys). Training objective: consistency ($\mathcal{I}$6), reconstruction, invertibility, an orthogonality loss pushing $\mathcal{I}$7 toward $\mathcal{I}$8 via KLD or MMD, and an $\mathcal{I}$9 sparsity term. Notably the standard normalizing-flow negative log-likelihood is omitted; Theorem 4 shows that if these losses vanish, the learned bijection recovers the true posterior $\boldsymbol{\Phi}$0. GT-free learning: since ground truth is unavailable, Count-Min estimates are refined by an EM algorithm (Theorem 5 gives the multiplicative update derived from KL-minimizing maximum likelihood, in the spirit of Richardson–Lucy deconvolution); refinement runs only 3–5 steps once as preprocessing because full convergence is prohibitively expensive at 150K+ dimensions. Stream filtering: an Ostracism-based separate-and-guard-hot mechanism routes frequent items to an augmented filter with $\boldsymbol{\Phi}$1 updates, so generative recovery applies only to the light Count-Min part before merging.

Evaluation

Experiments use five real-world datasets (CAIDA-2018, MAWI, Webdocs, Kosarak, Retail; 16K–160K distinct keys) plus five synthetic heavy-tailed streams, against nine baselines spanning CM, CS, CU, AG, PR, NZE, and three learning-augmented sketches (LCM, LCS, LS). Reported results include:

Task	Headline result
Frequency estimation	Up to ~$\boldsymbol{\Phi}$2 error reduction vs. prior methods; >400× better than CM at 256KB
Heavy-hitter detection	90–100% F1 at 64KB; up to 80+ point gains
Distribution/entropy	WMRE parity with baselines using 32× less memory; entropy AE near/below 10
Speed	No data-plane overhead; $\boldsymbol{\Phi}$3 faster than learned baselines; $\boldsymbol{\Phi}$4$\boldsymbol{\Phi}$5 faster decoding than LP solvers

Ablations attribute the gains jointly to the generative paradigm (a naive one-way mapping collapses), the EM-based pseudo-ground-truth channel, orthogonality, consistency, and sparsity losses; removing any one produces extreme outliers (e.g., absolute errors of 20,000 on Kosarak at 16KB). EM alone improves CM's AAE/ARE by roughly 76–86%, confirming its value as both a standalone refinement and a training signal. Robustness tests under temporal fluctuation, natural drift, and spatial frequency shifts show ARE degradation generally within a few percent. An important empirical finding is that per-element accuracy does not imply global distributional fidelity: CS-based methods can win point estimates while failing to match the true distribution shape.

Limitations and open questions

Several constraints are acknowledged explicitly. The known-key-set assumption is central; the Bloom Filter fallback introduces false positives whose avoidance requires nearly linear memory in $\boldsymbol{\Phi}$6. Recovery reconstructs all elements simultaneously, making single-key point queries $\boldsymbol{\Phi}$7 absent accelerated index structures—an open problem the authors flag. EM refinement is expensive at scale (minutes on CPU or tens of GB of GPU memory even parallelized), though needed only once during training. Performance also degrades under low-skew distributions where heavy hitters become hard to distinguish, narrowing FLORE's advantage. Finally, whether Zipfian-style skew assumptions transfer across all real workloads remains unverified beyond the datasets tested.

Conclusion

The paper reframes linear sketching as posterior generation conditioned on the summary, grounding the need for generative priors in a range-null-space analysis showing that sparse sketching matrices cannot satisfy RIP within streaming budgets. FLORE operationalizes this with a GT-free-trained conditional flow over autoencoded latents plus lightweight stream filtering, delivering large accuracy gains over nine state-of-the-art baselines while preserving—or improving—data-plane throughput. The main unresolved questions concern sub-linear point-query access, scalable key tracking without Bloom Filter overhead, and generalization to workloads whose skew structure departs from those evaluated.