Cardinality Estimation for High Dimensional Similarity Queries with Adaptive Bucket Probing

Abstract: In this work, we address the problem of cardinality estimation for similarity search in high-dimensional spaces. Our goal is to design a framework that is lightweight, easy to construct, and capable of providing accurate estimates with satisfying online efficiency. We leverage locality-sensitive hashing (LSH) to partition the vector space while preserving distance proximity. Building on this, we adopt the principles of classical multi-probe LSH to adaptively explore neighboring buckets, accounting for distance thresholds of varying magnitudes. To improve online efficiency, we employ progressive sampling to reduce the number of distance computations and utilize asymmetric distance computation in product quantization to accelerate distance calculations in high-dimensional spaces. In addition to handling static datasets, our framework includes updating algorithm designed to efficiently support large-scale dynamic scenarios of data updates.Experiments demonstrate that our methods can accurately estimate the cardinality of similarity queries, yielding satisfying efficiency.

• The paper presents a novel adaptive probing strategy that leverages LSH to estimate cardinality with high accuracy and efficiency.
• It employs progressive sampling combined with Chernoff bounds to dynamically balance error constraints and computational costs.
• Empirical results demonstrate superior Q-error and latency performance over DNN-based methods, ensuring scalability in real-world high-dimensional datasets.

Cardinality Estimation for High-Dimensional Similarity Queries with Adaptive Bucket Probing

Problem Statement and Motivation

Accurate and efficient cardinality estimation for high-dimensional similarity queries underpins query optimization in vector databases and broader data management systems. Traditional solutions either depend on uniform sampling, which suffers from coarse estimates and inefficiency, or leverage DNN-based estimators, incurring significant costs in training, maintenance, and explainability. The proposed framework addresses the cardinality estimation for similarity queries (CE4HD) by integrating locality-sensitive hashing (LSH) with an adaptive bucket probing strategy. The method achieves robustness across static and highly dynamic settings, with strong theoretical efficiency and error guarantees.

The core insight is derived from the observation that, following LSH partitioning, data points proximal in Euclidean space are likely to reside in neighboring hash buckets. This motivates a probing approach that adapts the neighborhood exploration dynamically based on the query’s threshold and data locality statistics, as visualized in the empirical selectivity decay over bucket Hamming distance.

Figure 1: Selectivity monotonically decreases with Hamming distance from the central bucket, motivating focused, adaptive probing for efficiency and accuracy.

Framework Architecture

LSH Partitioning and Bucket Probing

The framework constructs a hash-based index via E2LSH, which encodes each data vector into a hash bucket according to multiple LSH functions. For an incoming query-point and threshold, the system first identifies the central bucket to which the query hashes. The cardinality estimator then adaptively explores $k$-step neighboring buckets—groupings defined such that all hash codes at Hamming distance $k$ from the central bucket are probed as a unit. The process incrementally increases $k$ and incorporates progressive sampling within each bucket-group, terminating early if estimation bounds achieve target error tolerance.

Adaptive Probing and Progressive Sampling

The bucket probing algorithm is guided by a selectivity-based early stopping mechanism, supported by Chernoff bounds to guarantee error constraints with high confidence. Progressive sampling schedules within buckets allow the estimator to halt additional sampling once sufficient accuracy is statistically established. The framework fundamentally balances two objectives: minimizing distance computations (efficiency) and achieving tightly bounded cardinality error (accuracy).

Efficiency Optimizations: Product Quantization

To mitigate the library and latency costs of exhaustive Euclidean distance computations, the method optionally employs asymmetric distance computations via product quantization (PQ). The system partitions each vector into subvectors, clusters those to generate codebooks, and—upon each query—constructs a fast lookup table allowing rapid approximate distance evaluation using the (unquantized) query and quantized data vectors.

Figure 2: Offline construction time comparison indicates that the proposed estimator is substantially more efficient than learned-model baselines across all data sets.

Figure 3: Breakdown of offline construction phases reveals that PQ constitutes the bulk of any remaining overhead; hash index and neighbor table building remain negligible.

Dynamic Data Update Support

The method supports highly dynamic analytic workloads, where rapid insertions or large-scale batch updates to the vector collection are typical. The LSH index, PQ codebooks, and neighbor look-up structures are incrementally updatable with theoretically bounded impact on estimation accuracy. This distinguishes the framework from learned estimators, which generally require full retraining and cannot efficiently accommodate streaming updates.

Figure 4: Dynamic update efficiency shows linear scalability with data increment size, making the system suitable for continual ingestion scenarios.

Empirical Results

Accuracy and Efficiency

Comprehensive experimental evaluation demonstrates that the proposed method (Dynamic Prober, with and without PQ) achieves superior or competitive mean and high-percentile Q-Error across five large real-world datasets (image and text embeddings, with up to 2000000 objects and 1770 dimensions), even in adversarial cardinality ranges. Notably, accuracy degradation on data updates is negligible, in stark contrast to learned-model baselines, which suffer drastic error escalation if not retrained.

Online Latency

Estimation latency for the LSH-based methods is consistently two orders of magnitude lower than learned approaches utilizing neural inference, and competitive with brute-force uniform sampling, but with much tighter error bounds. Incorporation of PQ further improves inference latency for high-dimensional data, approaching $1.6\times$ speedup in the largest-dimension benchmarks.

Figure 5: Speedup from asymmetric PQ-based distance computation is especially pronounced as dimension increases.

Parameter Sensitivity and Practical Tuning

Error-latency tradeoff can be explicitly tuned by the $\epsilon$ parameter governing the progressive sampling stop condition. Experiments reveal diminishing returns for mean Q-error as $\epsilon$ decreases below dataset-specific turning points, providing practical guidance for system deployment.

Figure 6: Tuning $\epsilon$ exposes a clear accuracy-efficiency tradeoff, with optimal values found empirically per dataset.

Robustness to Updates

Unlike DNN-based estimators, whose performance rapidly degrades in the presence of bulk dataset updates unless expensive retraining is performed, the proposed architecture maintains near-identical Q-error post-update and does not exhibit pathologically high worst-case errors.

Figure 7: Q-error distributions remain stable after large-scale data updates, demonstrating the robustness of the update algorithms.

Theoretical and Practical Implications

This work establishes that non-learning, index-structured cardinality estimators can outperform neural and DNN-based alternatives both in efficiency and accuracy—contradicting prior claims of strict superiority for model-based CE4HD. The LSH-adaptive probing approach presents a paradigm shift for vector database systems and any analytic workload demanding sub-millisecond, high-confidence cardinality estimates over high-dimensional feature spaces.

On the theoretical front, the integration of progressive sampling with early stopping governed by selectivity decay and Chernoff bounds provides a rigorously analyzable framework for error guarantees. The extensibility to dynamic ingestion, and modular plug-in nature of PQ for high-dimensional efficiency, ensure relevance for both production databases and streaming analytic platforms.

Future Directions

Potential extensions include: (1) development of LSH-based cardinality estimators for non-Euclidean metrics and hybrid vector/text spaces, (2) integration with advanced query optimization pipelines in machine reasoning and LLM-driven retrieval augmentation, and (3) exploration of alternative, model-free trainingless cardinality frameworks that generalize across multiple similarity and aggregation predicates.

Conclusion

The presented adaptive LSH-bucket probing framework for high-dimensional cardinality estimation surpasses learned approaches in both accuracy and efficiency, especially in the presence of dataset dynamism. The theoretical grounding, modular optimization, and empirical robustness position the method as a practical solution for next-generation vector database systems, with broad implications for AI-driven data management and analytics (2604.04603).