MCI: A Maximal Clique Index for Efficient Arbitrary-Filtered Approximate Nearest Neighbor Search

Abstract: Approximate Nearest Neighbor Search with arbitrary filtering predicates (AFANNS) is essential for modern data applications, yet existing methods often incur substantial storage and computational costs. In this work, we introduce the Maximal Clique Index (\mci), a novel graph-based index designed for robust and efficient AFANNS. The core idea of \mci is to approximate a dense Nearest Neighbor Graph (NNG) through a compact, clique-based representation. We propose two key techniques: (1) Maximal Clique Cover (\mcc), which exploits the geometric transitivity of high-dimensional spaces to encode dense neighborhoods as maximal cliques, achieving an index with high compression and connectivity; and (2) Local Neighborhood Graph Geometric Densification, a strategy that constructs an index approximating a large NNG from a sparse initial NNG, recovers global connectivity by progressively increasing distance thresholds to locally densify the structure. The index is built in a lock-free parallel manner for scalability and queried via a carefully-designed multi-seed strategy to handle fragmented predicate-induced subgraphs. Extensive experiments on 10 datasets show that \mci significantly outperforms state-of-the-art methods by up to one order of magnitude in QPS at high recall while using substantially smaller space, and remains competitive even on range/keyword filtering tasks, demonstrating robust general-purpose performance.

• The paper presents a novel clique-based index (MCI) that outperforms traditional ANNS systems in high-recall filtering scenarios.
• It leverages geometric transitivity and maximal clique cover to densify local neighborhoods with minimal computational overhead.
• Experimental results demonstrate up to one order of magnitude higher QPS and 10× smaller index sizes compared to state-of-the-art methods.

Maximal Clique Index (MCI) for Arbitrary-Filtered Approximate Nearest Neighbor Search

---

Introduction and Motivation

Arbitrary-Filtered Approximate Nearest Neighbor Search (AFANNS) is a central primitive for applications such as vector search, semantic retrieval, recommendation, and Retrieval-Augmented Generation (RAG). In this setting, queries combine high-dimensional vector similarity with ad-hoc, unpredictable filtering predicates over data features. Existing ANNS indices, including graph-based (e.g., HNSW, NSG), quantization-based (IVFPQ), and hybrid architectures, are fundamentally ill-suited for AFANNS due to their limited connectivity after predicate-induced filtering and high memory or computation overhead when adapting to these scenarios.

The "MCI: A Maximal Clique Index for Efficient Arbitrary-Filtered Approximate Nearest Neighbor Search" (2604.22171) introduces a new approach: replace extended explicit neighbor storage with a compact, clique-based graph structure that leverages the intrinsic geometry of high-dimensional vector spaces. This enables robust high-recall search for arbitrary predicates, outperforming contemporary systems in both efficiency and space.

---

Core Technical Contributions

Clique-Based Indexing and Geometric Transitivity

Conventional proximity graph indices cannot guarantee connectivity in the predicate-induced subgraph when the expected number of valid neighbors per node falls below one. ACORN compensates via large expanded (2-hop) adjacency structures, incurring substantial space costs. MCI, by contrast, formalizes a geometric insight: in high-dimensional settings, the local neighborhoods of $k$-NN graphs form dense, highly transitive cliques. The Maximal Clique Cover (MCC) is leveraged to encode these neighborhoods implicitly:

• Maximal Clique Cover (MCC): Nodes are covered by a minimal set of maximal cliques, with a minimum clique size $\tau$, ensuring robust high-degree neighborhoods even after aggressive filtering.
• Geometric Densification: Starting from a sparse $k'$-NNG, the local graphs are iteratively densified by progressively relaxing the connectivity threshold (controlled by $\alpha$), efficiently expanding coverage with only modest computation.

The theoretical underpinnings are established by proving (via the Distance Concentration Phenomenon) that, under typical high-dimensional distributions, neighborhoods of the $k$-NNG are highly likely to form cliques as density increases. Thus, the large $k$ required for AFANNS can be efficiently approximated from a smaller $k'$ through clique mining.

Figure 1: Illustration of $k$-NNG, ACORN index with two-hop pruning, and MCI's clique-based representation.

Index Construction Algorithm

MCI construction proceeds as follows:

1. Sparse $k'$-NNG Construction: The initial $k'$-NNG on vectors is built (using NN-Descent).
2. Clique Mining (Greedy Algorithm): For each node, mine maximal cliques of size $\tau$0 in local neighborhoods built via densification ($\tau$1-expansion).
3. Dual Coverage: Each node is covered not only by its own local cliques but also those of its neighbors, amplifying effective out-degree and ensuring sufficient post-filtering connectivity.
4. Handling Outliers and Hubs: Isolated nodes and super-center nodes are addressed with early-termination and clique-pruning, ensuring algorithmic efficiency.

Figure 2: Illustration of the MCI construction algorithm for $\tau$2.

Query Processing with Multi-Seed Beam Search

Predicate-induced subgraphs in AFANNS can be highly fragmented. MCI employs a robust beam search, initialized from $\tau$3 randomly selected valid seeds (scaling $\tau$4 by selectivity), to ensure traversal across disconnected regions. Node expansions proceed via the union of the maximal cliques containing each node, for aggressive neighbor enumeration.

Complexity and Scalability

• Space: Worst-case $\tau$5, but empirical index sizes are much smaller (often $\tau$6 entries).
• Construction Time: $\tau$7, with $\tau$8 for $\tau$9-NNG build and $k'$0 the (logarithmic) number of densification rounds. Lock-free parallelization achieves high scalability.
• Query: $k'$1, where $k'$2 is query selectivity and $k'$3 the dataset size.

---

Experimental Results

The evaluation covers ten diverse datasets and compares MCI to ACORN, HNSW (Faiss and Milvus), IVFPQ, and specialized range/keyword-filtered search methods.

Recall/Throughput Trade-off

MCI achieves up to one order of magnitude higher QPS at high recall compared to state-of-the-art baselines under mixed selectivity regimes. For example, at recall@10 $k'$4 on deep1M, MCI yields $k'$5 QPS versus $k'$6 for the best baseline. Notably, ACORN is unable to reach $k'$7 recall on challenging datasets with low selectivity; MCI remains robust and efficient.

Figure 3: Recall@10 vs QPS of different methods under mixed selectivity.

Index Compactness

MCI delivers up to $k'$8 lower index size compared to ACORN, and matches/surpasses IVFPQ on storage for most datasets, a remarkable result for a graph-based scheme.

Figure 4: Index size of different methods.

Index Build Speed and Parallelization

Construction times are $k'$9 that of ACORN (including $\alpha$0-NNG build and clique mining), and the lock-free approach enables near-linear speedup with increasing thread count.

Figure 5: Index construction time of different methods.

Selectivity Robustness

Performance remains high as selectivity $\alpha$1 ranges from $\alpha$2 (broad filters) to $\alpha$3 (extremely narrow filters). MCI's recall remains stable, while competitors often degrade sharply or fail.

Figure 6: Recall@10 vs QPS of different methods under various selectivity.

Specialized Filtering Tasks

Even against domain-tuned baselines (e.g., iRange, SuperPF, UNG), MCI remains competitive for range and keyword filtering—despite its predicate-agnostic design.

Figure 7: Results of different methods on range filtering.

Figure 8: Results of different methods for keywords filtering.

Sensitivity Analyses

Varying core construction parameters ($\alpha$4, $\alpha$5), seed count ($\alpha$6), and clique expansion parameters ($\alpha$7), MCI shows strong robustness: performance is largely indifferent to $\alpha$8, modestly improved with larger $\alpha$9, and insensitive to $k$0 step size. The effective average out-degree grows substantially larger than $k$1, providing resilience under filtering (Table~\ref{tab:outdegree}).

---

Theoretical and Practical Implications

MCI demonstrates that geometric clique structure, when exploited appropriately, can yield connectivity and coverage in high-dimensional data that is otherwise unobtainable without incurring prohibitive memory costs. This insight overturns the assumption that dense indices must be expensive by encoding local transitive structure in a graph representation that is both space-efficient and robust under filtering.

Implications:

• For practical systems, MCI enables efficient, high-recall vector search with arbitrary predicates—critical for real-world RAG, recommendation, and search workloads.
• The index is compatible with dynamic feature and predicate types; it does not require assumptions on metadata distribution.
• The approach provides a new paradigm for graph-based vector search, with direct implications for scalability and adaptability in modern vector databases.

---

Future Directions

Open avenues for further research include:

• Dynamic Updates: Efficient insertion and deletion without full reindexing, leveraging localized clique mining and maintenance routines.
• Hybridization: Integrating MCI with quantization or hierarchical approaches for further space/time trade-offs.
• Distributed Architectures: Scaling MCI to billion-scale datasets with distributed memory and computation models.

---

Conclusion

Maximal Clique Index (MCI) introduces a clique-based, density-amplifying index structure that sets a new bar for AFANNS systems. MCI matches or outperforms state-of-the-art graph and quantization-based methods on both general and predicate-specific tasks, especially excelling in the challenging low-selectivity, high-recall regime. This work redefines robust connectivity for high-dimensional, filtered ANNS and establishes clique mining as a practical foundation for future vector search systems.