Revisiting Filtered ANN Benchmarks: A Hardness-Controlled Benchmark Generator for Realistic Evaluation

Abstract: Filtered approximate nearest neighbor (FANN) search must satisfy both vector similarity and structured predicates, yet evaluations remain brittle because real hybrid workloads are rarely shareable and existing benchmarks rely on ad-hoc synthetic or semi-real constructions. We argue that realism hinges on execution-driven query difficulty: failures in early filtering trigger over-fetching of additional candidates, shaping latency, throughput, and recall. Building on this insight, we propose $α$-Hardness, a query-level hardness metric that models the conditional execution chain via the over-fetch factor and extends naturally to strategy-conditioned settings. Across diverse datasets and hybrid strategies, $α$-Hardness exhibits strong monotonic alignment with empirical performance, while common proxies such as selectivity or attribute-vector correlation are frequently unstable or strategy-inconsistent. We further introduce HCBGen, a hardness-controlled benchmark generator that uses $α$-Hardness as an explicit control signal to synthesize workloads under coarse bias modes or to match a target hardness profile. Our experiments show that widely used benchmarks occupy a narrow, relatively easy portion of the hardness spectrum, masking robustness gaps that emerge under harder queries. Finally, we demonstrate that matching hardness distributions enables privacy-preserving proxy workloads that closely reproduce performance trends, bridging research benchmarks and real evaluation.

• The paper presents an execution-grounded hardness model that quantifies FANN query difficulty via over-fetch factors.
• It introduces HCBGen, a benchmark generator that synthetically produces queries spanning easy to hard regimes for robust evaluation.
• Empirical results highlight that traditional benchmarks favor easy queries, potentially masking performance collapses in high-hardness scenarios.

Revisiting Filtered ANN Benchmarks: Execution-Grounded Hardness and Hardness-Controlled Benchmark Generation

Introduction and Motivation

Filtered approximate nearest neighbor (FANN) search underpins key retrieval workloads that require both vector similarity and structured predicate matching. The problem has gained practical significance with the proliferation of hybrid data in RAG, recommendation, search, and multi-modal analytics. However, robust and generalizable benchmarking of FANN indices remains an open problem. Current studies often rely on ad-hoc synthetic or semi-real benchmarks, which are constrained by the limited shareability of real-world hybrid workloads and consequentially suffer from uncontrolled, often "easy" query regimes. This paper introduces an execution-grounded framework for modeling query-level hardness in FANN and presents HCBGen, a benchmark generator that enables controlled, realistic, and privacy-preserving evaluation across the hardness spectrum (2606.14193).

Execution-Grounded Hardness Modeling

A central insight is that FANN query difficulty is not fully captured by popular proxies such as selectivity (fraction of candidates passing filters) or attribute–vector correlation. Instead, hardness is execution-dependent: it reflects the cost incurred as a system retrieves candidates and over-fetches to satisfy both similarity and structured constraints. The authors formalize query hardness as the effective over-fetch factor, denoted $H_\alpha(q|K)$, which quantifies the number of vector-ranked candidates required before $K$ filter-satisfying results are found.

The proposed estimator incorporates both global selectivity and local density of valid points around the query vector, yielding:

$$\widehat{\alpha}(q;K) = \frac{|V_0|}{|V_f|} \cdot \frac{d(q_v, K\text{-NN}|V_f)}{d(q_v, K\text{-NN}|V_0)} \cdot K$$

where $|V_0|$ and $|V_f|$ are the dataset sizes before and after filtering, and $d$ denotes $K$-NN distances. This definition retains strong monotonic alignment with empirical recall–QPS trade-offs.

The model is extended to account for strategy-conditioned hardness—reflecting the major execution families in FANN (vector-centric, e.g., post-filtering, and filter-centric, e.g., pre-filtering/UNG)—by inverting the hardness ordering for filter-centric strategies.

Figure 1: Hardness-driven query ordering shows monotonic alignment between estimated hardness and recall–QPS in (a); selectivity and correlation fail to consistently order query difficulty in (b) and (c).

Hardness Validation and Empirical Alignment

Across 29 datasets (synthetic and semi-real) and seven FANN algorithms, strategy-conditioned $H_\alpha$ demonstrates strong Spearman rank correlation (mostly $\rho \le -0.7$) with empirical performance, outperforming selectivity and correlation proxies which are unstable or consistently misaligned in cross-strategy scenarios.

Figure 2: Across 29 diverse benchmarks and strategies, the proposed hardness estimation aligns consistently with empirical recall–throughput performance, while selectivity and correlation often fail or reverse.

Further, the method generalizes naturally to complex predicates (range, polygon, regex, set overlap) by augmenting the scan cost to reflect per-query predicate complexity, preserving alignment under compute-bound regimes.

Hardness-Controlled Benchmark Generator (HCBGen)

HCBGen operationalizes the execution-driven hardness metric to enable three benchmark construction modes:

• Coarse-grained biasing: Systematic generation of high/low hardness regimes for stress tests or robustness analysis.
• Hardness-neutral evaluation: Reproducible fixed benchmarks independent of index design.
• Fine-grained profile matching: Synthesis of queries to faithfully match a given hardness distribution, providing a privacy-preserving proxy when real query logs are inaccessible.

Architecturally, HCBGen decouples label/predicate synthesis from the hardness estimation and employs an adaptive, execution-model-aware query selection loop.

Figure 3: The HCBGen framework separates candidate query generation from execution-aware hardness estimation, supporting both synthetic and real-data-driven construction modes.

Benchmark Sensitivity and Robustness Evaluation

Empirical distributions reveal that commonly used synthetic and semi-real hybrid workloads occupy a narrow, easy part of the hardness spectrum, with little coverage of difficult queries. By contrast, HCBGen-generated benchmarks (particularly in high-hardness mode) stretch into more challenging regimes, directly exposing the robustness boundaries of FANN algorithms.

Figure 4: Hardness-controlled benchmarks generated with HCBGen provide wider and more representative coverage of the query hardness spectrum than workloads from prior benchmarks or semi-real workloads.

Recall–QPS trade-offs for FANN strategies (NHQ, ACORN, UNG, RWalks, post-filtering) exposed large performance variability and inconsistent ranking when evaluated beyond the favorable regimes—demonstrating that earlier benchmarks can obscure performance collapses under "hard" queries.

Figure 5: Performance of FANN strategies exhibits marked sensitivity to the underlying hardness regime, with many methods degrading sharply or inverting performance ranking as query hardness increases.

Hardness-Based Privacy-Preserving Workload Approximation

A practical application of $H_\alpha$ is in privacy-sensitive settings, where real workloads cannot be shared; HCBGen can synthesize proxy workloads matching only the target $K$0 distribution. Experiments on four semi-real datasets show that performance trends (recall–QPS curves) are preserved by such proxy queries, even when base data is entirely synthetic.

Figure 6: Synthetically generated workloads can closely approximate the hardness distributions of original, unavailable query sets, supporting privacy-preserving benchmarking.

Figure 7: Matched-hardness proxy workloads reproduce recall–QPS behavior across FANN strategies with high fidelity.

Implications and Future Directions

By formalizing hybrid query hardness as an execution-grounded chain and providing a practical, adaptable benchmark generator, this work addresses critical limitations in reproducibility and robustness assessment for FANN research. Its findings show that:

• Most published benchmarks bias toward easy queries, possibly overstating robustness.
• Hardness-aware benchmarking enables more reliable cross-strategy and cross-paper comparison.
• Privacy-preserving evaluation based on hardness distribution facilitates community sharing without exposing sensitive or proprietary workload details.

Potential extensions include integration with more complex, semantic predicates, adaptation to dynamic or evolving datasets, and broadening to new index architectures or pruning strategies. The methodology provides a principled foundation for systematic, meaningful stress testing as FANN search is integrated into ever more mission-critical applications.

Conclusion

This work presents the first execution-informed definition and estimator of FANN query hardness, demonstrating its empirical validity and proposing the HCBGen benchmark generator for controlled, robust FANN benchmarking. The results indicate that current benchmarks are insufficiently challenging, and that hardness-controlled evaluation is essential for accurately assessing and comparing hybrid vector search algorithms (2606.14193).