FANNBench: Unified Filtering ANN Benchmark

• FANNBench is a unified framework that standardizes filtering approximate nearest neighbor search by harmonizing diverse index structures, filtering semantics, and evaluation settings.
• It distinguishes filtering strategies such as pre-filtering, post-filtering, and joint-filtering, enabling fair comparisons of algorithmic components across varied selectivities and datasets.
• Component-level analyses in FANNBench reveal critical trade-offs in entry-point selection, pruning techniques, and edge filtering overhead, offering actionable insights for optimizing search performance.

Searching arXiv for the cited FANNBench-related papers to ground the article in current sources. FANNBench is a unified benchmark and analysis framework for Filtering Approximate Nearest Neighbor (Filtering ANN) search, defined as ANN search over vectors with additional attribute constraints. It was introduced to standardize a fragmented literature in which different papers used different index structures, filtering semantics, query types, hyperparameters, and implementation choices, thereby making methods difficult to compare fairly. In that role, FANNBench provides a common interface, common task definitions, common datasets, and common evaluation settings, while also analyzing why some methods perform differently. The name also appears in the vector-search accelerator literature as the benchmark and evaluation context used for FANNS, an end-to-end framework for co-designing IVF-PQ algorithms and FPGA hardware; in practice, however, FANNBench most commonly denotes the Filtering ANN benchmark framework associated with the public codebase at https://github.com/lmccccc/FANNBench (Li et al., 22 Aug 2025, Jiang et al., 2023).

1. Definition and problem setting

FANNBench is centered on Filtering ANN search, where the result must satisfy both a vector-similarity objective and an attribute predicate. The benchmark formalizes the exact search task as

$$NN(q|r) = \arg\min_{o \in D,\, r(o.a)} \|q - o.v\|.$$

Here, $D$ is the dataset of vector-attribute objects, $o.v$ is the vector embedding, $o.a$ is the attribute value or values, and $r(o.a)$ denotes satisfaction of the filter predicate. Approximate filtering ANN algorithms are used because exact search is usually too expensive.

The framework was created because Filtering ANN lies at the intersection of vector similarity search and database filtering. The motivating claim is that the literature had become fragmented: different systems were evaluated under incompatible assumptions, and nominal performance differences often reflected inconsistent evaluation pipelines rather than algorithmic differences. FANNBench therefore aims both to standardize the task and to expose the algorithmic components that drive observed behavior.

Within the broader vector-search literature, the term “FANNBench” also appears in work on FPGA acceleration for ANN search. There, the emphasis is not attribute filtering but benchmarkable evaluation of IVF-PQ and OPQ+IVF-PQ workloads under recall targets, throughput measurements, online latency distributions, and multi-accelerator scale-out. That usage is materially different from the Filtering ANN framework, but it helps situate FANNBench within a wider benchmarking tradition for vector retrieval systems (Li et al., 22 Aug 2025, Jiang et al., 2023).

2. Unified interface and search model

A central contribution of FANNBench is a unified Filtering ANN search interface. Its purpose is to ensure that differences in reported performance reflect algorithmic behavior rather than incompatible implementations. The interface captures four major dimensions of Filtering ANN.

First, it distinguishes attribute type: numerical, categorical, or arbitrary filtering. Second, it distinguishes filtering strategy: pre-filtering, post-filtering, or joint-filtering. Third, it records the base ANN index, including graph-based, IVF-based, and quantization-based methods. Fourth, it includes the attribute index or subset organization, such as segmented edges, segmented subgraphs, partitioning, labeled edges, or subset masks.

The benchmark uses these dimensions to compare methods under the same datasets, selectivity levels, recall target, similarity metric, threading policy, and construction and search settings where possible. For example, graph-based methods are evaluated with fixed $M=40$ and ef_construction=1000, while search parameters such as ef_search or nprobe are tuned to reach 90% recall. Query execution is standardized to one thread, whereas index construction uses 128 threads.

The framework’s treatment of selectivity is foundational. Selectivity is defined as the fraction of items that satisfy the filter: high selectivity means many items pass, and low selectivity means few items pass. This provides a common axis for comparing methods across filtering strategies. Pre-filtering is characterized as filtering first and then searching only the matching subset; post-filtering as running ANN search first and then discarding non-matching candidates; and joint-filtering as integrating filtering directly into ANN traversal. The benchmark treats these not merely as implementation variants, but as distinct execution regimes with different robustness profiles (Li et al., 22 Aug 2025).

3. Taxonomy and benchmark coverage

FANNBench proposes a two-axis taxonomy, organized by attribute type and filtering strategy. By attribute type, it distinguishes range filtering for numerical attributes, label filtering for categorical attributes, and general filtering for arbitrary attributes or conditions. The corresponding predicate forms are given as $l \le O_i.a \le u$ for range filtering, $f \in O_i.A$ for label filtering, and unrestricted filtering expressions for general filtering.

The benchmark evaluates 10 algorithms / 12 concrete methods, grouped by task:

• Range filtering: SeRF; DSG; $\beta$-WST, evaluated as WST-Vamana and WST-opt; iRangeGraph; and UNIFY, evaluated as UNIFY-CBO and UNIFY-joint.
• Label filtering: Filtered-DiskANN, evaluated as FDiskANN-VG and FDiskANN-SVG; and NHQ, evaluated as NHQ-NSW and NHQ-KGraph.
• Arbitrary filtering: Faiss-HNSW, Faiss-IVFPQ, Milvus-HNSW, Milvus-IVFPQ, and ACORN.

These are explicitly positioned as the most representative, open-source, and relevant methods for their respective filtering scenarios. The taxonomy is not only descriptive; it is used to frame tradeoffs. Pre-filtering is described as effective when the searchable subset is much smaller, but problematic for graph search if pruning destroys connectivity. Post-filtering is described as effective when selectivity is high, but expensive at low selectivity because many invalid candidates are explored and discarded. Joint-filtering is characterized as especially useful at moderate selectivity, while also being the most sophisticated and hardest to design (Li et al., 22 Aug 2025).

4. Experimental protocol, datasets, and reproducibility

FANNBench evaluates four datasets, each with 10K queries:

Dataset	Labels	Size
SIFT	Synthetic	10M
SpaceV	Synthetic	10M
Redcaps	Real/Synthetic	1M
Youtube-RGB	Real/Synthetic	1M

The associated dimensionalities are 128 for SIFT, 100 for SpaceV, 512 for Redcaps, and 1024 for Youtube-RGB. The benchmark evaluates selectivity levels of 0.1%, 1%, 10%, 50%, and in some settings 100%. For range queries, selectivity is controlled by adjusting the attribute interval $[l,u]$. For label queries, it is controlled by assigning categorical labels with fixed probabilities. For arbitrary filtering, the benchmark uses two-predicate hybrid filtering combining categorical and numerical predicates, with selectivities combined multiplicatively under uniform assumptions; the paper gives the example of a categorical label assigned to 60% of vectors and a range query selectivity of 83%, yielding combined selectivity of approximately 50%.

The experimental environment is specified as Ubuntu 24.04 LTS, Intel Xeon Platinum 8358 @ 2.60GHz, 128 physical cores, and 2 TB memory. Index construction uses 128 threads and query evaluation uses 1 thread. All datasets use L2 / Euclidean distance. The principal metric is QPS (Queries Per Second), supplemented by comparisons per query, especially for graph-based methods, where fewer comparisons indicate more efficient traversal. The usual evaluation target is 90% recall@10, and search hyperparameters are tuned to reach that target.

FANNBench also records reproducibility-oriented implementation choices. All methods use in-memory indexes; DiskANN is treated as having an in-memory search component; SeRF and DSG use parallelism; ACORN’s filtering storage is optimized; and Faiss’s is_member() is improved. Each experiment is averaged over three runs, and the paper notes that full configurations are provided in the appendix or supplement.

Two dataset-oriented difficulty measures are also defined. Dis_Ratio is the ratio of average top-10 ground-truth distance to average pairwise dataset distance, where higher values indicate a broader search region. JS_Div is the Jensen-Shannon divergence between the full dataset distribution and the queried subset. The reported observation is that lower selectivity makes queries harder (Li et al., 22 Aug 2025).

5. Component-level analysis

A distinctive feature of FANNBench is its component analysis of pruning, entry-point selection, and edge filtering cost. This moves the benchmark beyond aggregate QPS and recall reporting toward mechanism-level explanation.

In pruning analysis, the benchmark compares Relative Neighbor Graph (RNG) pruning, two-hop pruning, and label-covered pruning. RNG pruning removes an edge $D$0 if there exists a $D$1 that is closer to $D$2 than $D$3 is, and also closer to $D$4 than $D$5 is. FANNBench argues that, in Filtering ANN, RNG pruning ignores attribute diversity and can be harmful at low selectivity. ACORN’s two-hop pruning is described as preserving more potentially useful connections and aligning better with two-hop query expansion. Filtered-DiskANN’s label-covered pruning retains an edge unless both geometric redundancy and label or attribute coverage justify pruning.

The benchmark’s findings on pruning are explicit: RNG pruning breaks down at low selectivity; for ACORN, two-hop pruning is best or comparable at low selectivity; for SeRF, KGraph-style pruning helps somewhat at moderate low selectivity but still fails at extreme selectivity; and simpler connectivity rules can outperform more aggressive geometric pruning in filtered search.

Entry-point selection is treated as equally important. FANNBench distinguishes unrestricted or default entry points, used by methods such as Faiss-HNSW with filtering checks, ACORN, NHQ, and UNIFY post-filtering modes, from specialized entry points used by SeRF, DSG, iRangeGraph, and some UNIFY modes. SeRF and DSG in particular skip hierarchical HNSW navigation and choose multiple bottom-layer entry points within the valid range, often evenly spaced. The benchmark reports a strong practical result: more entry points improve performance across all selectivities. Increasing bottom-layer entry points from 3 to 30 or 300 reduces comparisons and improves QPS without recall loss. By contrast, UNIFY’s hierarchical structure helps mainly at high selectivity, while at low selectivity the hierarchy adds limited value.

FANNBench also isolates edge filtering overhead. A central example is that DSG can have fewer comparisons than SeRF in some settings yet lower QPS. The reported cause is not traversal quality but overhead in edge filtering during construction: DSG spends a significant fraction of time finding valid edges, this overhead grows with ef_search, and SeRF spends almost no time on that step. The implication is that construction-time filtering logic can become a first-order performance factor even when traversal efficiency appears favorable (Li et al., 22 Aug 2025).

6. Empirical findings, method-selection guidance, and open problems

FANNBench’s empirical conclusions are organized by filtering type. For range filtering, the main finding is that segmented subgraph methods are generally the strongest, with iRangeGraph, WST-opt or WST-Vamana, and UNIFY variants highlighted. SeRF and DSG are reported as strong at moderate and high selectivity but failing at 0.1% selectivity. BST-based segmentation excels at very low selectivity, especially iRangeGraph and WST-Vamana. Segmented edges and segmented subgraphs are both effective, but subgraph-based methods are described as more robust. A key limitation is that edge-based approaches depend on successful graph construction, which is fragile at low selectivity.

For label filtering, the benchmark reports that FDiskANN-SVG is robust in many cases, but that performance depends on graph quality. NHQ is unstable at low selectivity, while NHQ-KGraph often outperforms NHQ-NSW because it keeps more connections. The broad conclusion is that label filtering is still underdeveloped, and no evaluated method is robust across all selectivities.

For arbitrary filtering, FANNBench finds that Faiss-HNSW and ACORN are competitive at high selectivity, Milvus-HNSW is strong at low selectivity, Milvus gains robustness through partitioning, and Faiss-IVFPQ is reliable at low selectivity but often slower than Faiss-HNSW. The benchmark also reports that multi-predicate filtering under uniform distributions behaves similarly to single-predicate filtering at the same overall selectivity. The stated limitation is that arbitrary filtering remains hard, and ACORN is the only purpose-built method evaluated, but is not robust across all selectivities.

The benchmark distills these findings into method-selection guidance. For numerical or range filtering, iRangeGraph is presented as offering the best query performance but slower construction; SeRF as offering faster construction and lower memory but worse low-selectivity behavior; WST-opt as performing well but being too memory-heavy to recommend broadly; and UNIFY / iRangeGraph as strong general choices. For label filtering, the recommendation is FDiskANN-SVG if label cardinality is 1 and NHQ-KGraph when cardinality exceeds 1. For arbitrary filtering, the guidance is ACORN for flexible complex filtering, Faiss-HNSW for high selectivity, and Milvus-HNSW for very low selectivity.

The benchmark also identifies open problems: arbitrary filtering remains open; hyperparameter tuning is still too manual; better integration with vector databases is needed; dynamic updates are underexplored, with DSG singled out as supporting dynamic insertions well; and attribute distribution analysis remains limited, since many experiments assume uniform attribute distributions rather than skewed or correlated real-world patterns (Li et al., 22 Aug 2025).

7. Reassessment of filtered ANN benchmarking

Subsequent work on filtered ANN benchmarking argues that fair evaluation remains difficult even when interfaces, datasets, and metrics are standardized. The 2026 paper "Revisiting Filtered ANN Benchmarks: A Hardness-Controlled Benchmark Generator for Realistic Evaluation" introduces $D$6-Hardness as a query-level hardness metric and HCBGen as a hardness-controlled benchmark generator. Its core argument is that realism depends on execution-driven query difficulty: in post-filtering, failures in early filtering force over-fetching of additional candidates, which in turn shapes latency, throughput, and recall. The paper defines $D$7 as the number of vector-ranked candidates that must be examined so that the scan phase outputs $D$8 results satisfying the filter, and uses this to model hardness as an execution cost over the fetch and scan chain (Lim et al., 12 Jun 2026).

That work reports that, across 29 datasets and seven processing strategies, $D$9-Hardness is strongly monotonic with actual recall–QPS behavior, typically with Spearman correlation $o.v$0, whereas selectivity and attribute–vector correlation are often unstable or strategy-inconsistent. It further argues that common benchmarks occupy a narrow and relatively easy hardness range, with little mass in hard regions, and that generated workloads matched to a target hardness distribution can reproduce recall–QPS trends while preserving privacy through distributional signatures rather than raw query logs.

This suggests a complementary relationship between FANNBench and later hardness-controlled benchmarking. FANNBench standardized the comparison surface for Filtering ANN algorithms: common interface, common task definitions, common datasets, and component-level analysis. The later hardness-based framework argues that standardized evaluation still needs explicit control over query difficulty and strategy dependence. Taken together, these works define two layers of filtered ANN evaluation: algorithmic comparability through benchmark unification, and workload realism through hardness-aware benchmark generation (Lim et al., 12 Jun 2026).