ANNS-AMP: Accelerating Approximate Nearest Neighbor Search via Adaptive Mixed-Precision Computing

Abstract: Approximate nearest neighbor search(ANNS) is a critical kernel in modern applications such as LLM and recommendation systems.However,its efficiency is fundamentally limited by the need to compute distances between a query and a massive number of high-dimensional vectors,most of which are non-neighbors.Existing approaches reduce redundancy via index optimization or early termination,but remain constrained by fixed-precision computation,leading to unnecessary arithmetic and memory bandwidth overhead.This paper presents ANNS-AMP,an adaptive mixed-precision framework and accelerator that adapts the precision of distance computation to the characteristics of queries and data distribution.The key insight is that different regions of the vector space require different levels of precision to preserve top-k accuracy.ANNS-AMP leverages the clustered structure of PQ-based indices and introduces a lightweight predictor to determine cluster-level precision at runtime based on features such as scale,radius,and query distance.To efficiently realize variable-precision execution,we design a bit-serial accelerator with a bit-interleaved data layout,enabling throughput to scale with reduced precision while mitigating memory bandwidth bottlenecks and load imbalance through a greedy scheduling strategy.Moreover,the runtime predictor can also reuse the bit-serial computing array for efficient runtime prediction and can be fitted to the ANNS pipeline without performance penalty.According to our experiments on representative datasets,ANNS-AMP achieves 163.76x,10.57x,and 2.06x performance speedups on average,and reduces average energy consumption by 1100.00x,39.41x,and 6.66x compared to CPU,GPU,and customized ANNS accelerator baselines,respectively,while maintaining accuracy loss below 2.7%.These results demonstrate that adaptive mixed-precision computing is a promising direction for efficient large-scale ANNS.

• The paper presents a novel hardware-software co-designed system that adaptively assigns bit-width per subspace using an SVR model for precision prediction.
• It employs a bit-serial accelerator and bit-interleaved memory layout to minimize bandwidth and energy, demonstrating over 160× speedup and 1100× energy savings on SIFT100M.
• Empirical results confirm that ANNS-AMP maintains under 2.7% accuracy loss while leveraging up to 93.75% low-precision operations, outperforming CPU, GPU, and competing accelerators.

ANNS-AMP: Acceleration of Approximate Nearest Neighbor Search via Adaptive Mixed Precision

Motivation and Problem Statement

Approximate Nearest Neighbor Search (ANNS) is a core computational building block in high-dimensional data retrieval, exemplified by applications in LLM inference and recommender systems. The overwhelming majority of ANNS cost arises from distance calculations between queries and vast vector corpora, despite most candidates being non-neighbors. While indexing, pruning, and early termination improve efficiency, nearly all current accelerators and libraries employ fixed-precision arithmetic, leading to over-provisioned computation and memory traffic. ANNS-AMP departs from this paradigm by introducing adaptive mixed-precision computation driven by online precision prediction, thereby economizing both arithmetic and bandwidth in a highly dynamic, accuracy-aware manner.

Figure 1: Illustration of ANNS with cluster-based PQ index (L2 distance).

The proposed approach recognizes that the required precision for reliable nearest neighbor identification fundamentally depends on localized query-subspace interactions: certain clusters and subspaces per query admit aggressive quantization without accuracy penalty, while others are precision-critical.

ANNS-AMP Framework and Algorithm

The ANNS-AMP framework is instantiated as a hardware-software co-designed system around cluster-based PQ indices but is agnostic to variant PQ schemes. It introduces a two-stage spatial partitioning of the vector space, performing both dimension-wise slicing and vector-wise clustering, followed by feature extraction per subspace for online decision-making.

Key extracted features for each subspace include the centroid-to-query distance, subspace radii, and subspace occupancy, which together control the error sensitivity under varying precision configurations. These features are inputs to an offline-trained SVR model (with RBF kernel, empirically justified as linear kernels failed to capture feature–precision dependencies) which predicts the minimal sufficient bit-width per subspace to maintain top-k accuracy.

Figure 2: The overview of ANNS-AMP framework.

Figure 3: Illustration of the relation among a query, the sub-space it locates, and the neighbor sub-space.

Subspaces are constructed so that high-dimensional intersections map to separations in lower-dimensional projections, preserving the separability required for precision adaptation. Empirically, the fraction of subspaces benefiting from low-precision operations increases with finer slicing but plateaus beyond a certain granularity.

Figure 4: Intersecting high-dimensional sub-spaces may be separate in low-dimensional projection space.

This is quantified over realistic datasets such as SIFT100M, where the neighbor sub-space distance statistics demonstrate that query-aligned clusters become well separated after sufficient partitioning.

Figure 5: Average ratio between neighbor sub-space distances and the sum of their radius in CL and LC on SIFT100M. $N$ input vectors are split into $\sqrt{N}$ sub-spaces.

Precision prediction labels are generated by measuring, within each subspace, the distance gaps required to distinguish true top-k neighbors from non-neighbors, facilitating supervised regression model training.

Figure 6: Illustration of label generation. (a) The sub-space contains global neighbor(s). (b) The sub-space contains none of global neighbors.

Accelerator Microarchitecture

ANNS-AMP implements an explicit bit-serial accelerator architecture, where every computational path—query, cluster centroid, codebook operations—supports variable bit-width arithmetic scheduled at runtime per subspace. This choice reflects two design goals: (1) throughput scales almost inversely with active bit-width, and (2) fine-grained workload balancing is supported via dynamic scheduling.

The architecture consists of:

• Distance Calculation Module (DCM): Bit-serial subtractor, multiplier, and adder pipelines for L1/L2 distance computations.
• Distance Reduction Module (DRM): Parallel adder trees, managed at the subspace level.
• Top-k Sorting Module (TSM): On-chip priority queues for selection.
• Precision Prediction Module (PPM): Online SVR inference leveraging bit-serial compute resources.
• Data Fetching Module (DFM): Optimized for bit-interleaved memory layout, which coalesces memory traffic by precision assignment.
• Load Scheduling Module (LSM): Greedy task redistribution based on analytical modeling for mitigating load imbalance induced by non-uniform precision demands.

  Figure 7: The overall hardware architecture of ANNS-AMP.

Efficient mixed-precision operation relies on a bit-interleaved layout, where bits are stored across memory buffers so that only the required bit planes are fetched during low-precision phases, minimizing bandwidth consumption and memory energy.

Figure 8: Illustration of the bit-interleaved memory layout.

Area and power breakdowns indicate minimal footprint and power consumption relative to comparator ASIC solutions, with the compute-dominant modules being highly area-efficient due to the bit-serial design.

Figure 9: The area and power breakdown graphics.

Experimental Results

Comprehensive evaluation was conducted on SIFT100M and DEEP100M (quantized to 8 bits). Hardware was implemented in a cycle-accurate simulator with physical RTL synthesis (TSMC 14nm). ANNS-AMP was compared against Faiss on CPU/GPU and the ANNAx12 accelerator.

Key results include:

• Throughput: ANNS-AMP is $163.76\times$ faster than CPU, $10.57\times$ over GPU, and $2.06\times$ over ANNAx12.
• Energy: Average energy savings are $1100\times$ (CPU), $39.4\times$ (GPU), $6.66\times$ (ASIC), under equivalent accuracy (<2.7% loss).
• Low-precision ratio: Up to $87.49\%$ of cluster location and $93.75\%$ of LUT construction distance computations are performed with low precision.

  Figure 10: Performance speedup over CPU, GPU, ANNAx12.

  

  Figure 11: Energy reduction over CPU, GPU, ANNAx12.

Precision/accuracy trade-off: Across a range of index configurations, accuracy loss remains below 2.7% with the vast majority of operations at reduced bit-width.

Figure 12: Accuracy loss with mixed precision ANNS among various parameter configurations on SIFT100M.

Figure 13: Benefits from mixed precision ANNS among various sub-space settings in cluster locating phase on SIFT100M.

The bit-interleaved memory layout yields bandwidth efficiency improvements of $\sqrt{N}$0 over conventional layouts.

Figure 14: # of memory accesses under different data layouts and datasets.

Load scheduling offers up to $\sqrt{N}$1 speedup when relaxed accuracy constraints result in higher precision variance among subspaces.

Figure 15: Speedup of Load Scheduling Module (LSM) on the look-up table construction phase with loose accuracy constraints.

ANNS-AMP scales with bandwidth and offers substantial future headroom for throughput scaling as memory interfaces improve.

Figure 16: Relation between the throughput and bandwidth of ANNS-AMP.

Comparison and Implications

ANNS-AMP subsumes and outperforms prior ASIC accelerators, as well as recent near-memory and in-memory ANNS proposals, thanks to its granularity of precision adaptation and highly efficient hardware-software co-optimization. Notably, unlike classical top-k-based early termination and index-tuning methods, ANNS-AMP's model is orthogonal to early-termination heuristics and directly minimizes over-provisioned computation.

The strong empirical improvements in both throughput and energy efficiency suggest that adaptive precision should be treated as a first-class consideration in future ANNS design and accelerator stack. The results also indicate that for large-batch LLM applications, subspace-level dynamic quantization and runtime arithmetic adaptation can yield performance that outpaces even the most recent CPU/GPU and PIM designs.

Theoretically, the demonstrated decoupling of computation and bandwidth bottlenecks via per-subspace adaptation points to broader applicability in other high-dimensional retrieval primitives, including maximum inner product search, RAG pipelines, and high-cardinality grouping.

Conclusion

ANNS-AMP makes a compelling case for the adoption of adaptive mixed-precision strategies in cluster-based approximate nearest neighbor search over large, high-dimensional datasets. By jointly designing a regression-based precision predictor and a bit-serial hardware architecture with mem/scheduling optimizations, ANNS-AMP advances the state-of-the-art in both throughput and energy efficiency while minimizing accuracy degradation. Its methodology is immediately applicable to future data retrieval accelerators, and the outlined design principles could inform the evolution of vector search infrastructure underlying large-scale AI systems.