Fused Int8 ANN Search

• Fused Int8 ANN search is a technique that quantizes embeddings to 8-bit integers and fuses multiple ANN steps into one GPU-resident kernel.
• It minimizes memory bandwidth and latency by integrating candidate gathering, distance computation, and top-k selection, achieving up to 9× speedup over float-based methods.
• Efficient implementation leverages dp4a instructions, coalesced memory access, and IVF clustering with bloom filtering to optimize large-scale vector retrieval.

Fused Int8 ANN search refers to a class of approximate nearest neighbor (ANN) algorithms and system kernels that integrate (fuse) compatible steps of the search process into a single computational unit, while operating entirely on quantized 8-bit integer (Int8) representations of high-dimensional vectors. This strategy is driven by the need for efficient, low-latency, and high-throughput large-scale retrieval workloads on hardware accelerators, notably modern GPUs. By fusing candidate-gathering, distance computation, and top-k selection, and leveraging vectorized integer arithmetic, fused Int8 ANN search dramatically reduces both compute and memory footprint compared to conventional (e.g., float32, multi-stage) ANN pipelines.

1. Int8 Quantization and Embedding Representation

Fused Int8 ANN search begins by quantizing all floating-point embeddings—both index (item) vectors and cluster centroids—to 8-bit integers. The quantization process scans all embedding values $E_{ij}$ to determine their global minimum $x_{\min}$ and maximum $x_{\max}$, then calculates a scale factor $S = 255/(x_{\max}-x_{\min})$ and a zero-point $z = -128 - \mathrm{round}(x_{\min}\times S)$. Each embedding entry is quantized as: $$q_{ij} = \mathrm{clamp}\bigl(\mathrm{round}(E_{ij}\times S)+z,\ -128,\ 127\bigr)$$ Resulting quantized tensors—$\mathbf{C}_q\in\mathbb{Z}_8^{K\times d}$ for $K$ centroids, $\mathbf{X}_q\in\mathbb{Z}_8^{N\times d}$ for $N$ items—are stored on GPU, along with $S$ and $z$ for dequantization if needed. Query vectors are quantized identically at inference time. This quantization enables all-core integer matrix-multiplication instructions (e.g., NVIDIA’s dp4a) and halves the embedding memory footprint, enabling scaling to billions of items with competitive recall (Xue et al., 18 Nov 2025).

2. Algorithmic Fusion: Single-Kernel ANN Search

Traditional IVF-ANN search decomposes into (i) centroid scoring, (ii) candidate gathering, (iii) distance computation, and (iv) top-k selection, often split across multiple kernels or device-host roundtrips. The fused Int8 ANN paradigm combines candidate gathering, similarity computation, and top-k selection into a single, device-resident CUDA kernel per query or query batch:

• Query-centroid dot-products are computed with quantized arithmetic to score clusters.
• Top-$P$ clusters are identified in registers.
• For each selected cluster, contiguous blocks of quantized embeddings are streamed (coalesced loads), and quantized dot-products are computed in-tile via dp4a instructions.
• A register-resident top-k min-heap is maintained per thread block; updates are performed as candidate scores are generated.
• By avoiding explicit gathering of large candidate lists (no full “idxs” tensor), memory bandwidth is reduced and kernel launch/host coordination overhead collapses from three calls to one (Xue et al., 18 Nov 2025).

Pseudocode excerpt:

// Fused Int8 ANN search kernel for a query for cluster in topP_clusters: for item_id in cluster: EmbTile = load_int8_tensor(X_q[item_id:item_id+TILE, :]); scores = dp4a(EmbTile, q_q); update_heap(heap, scores, base_id=item_id);

3. Data Layout and Index Co-Design

The index structure for fused Int8 ANN search typically employs IVF clustering—KMeans++ partitions $N$ items into $K$ clusters. Quantized item embeddings are stored as a $N \times d$ tensor, physically sorted by cluster id, with cluster_offsets[K+1] marking boundaries for contiguous, coalesced GPU reads. Each query only probes a subset of clusters (top-$P$), minimizing memory access and enabling high throughput. Bloom filter bitmasks often provide feature-level filtering; these can be evaluated inline within the ANN kernel via blockwise masking, further reducing candidate loads—empirically, 256 cluster probes on 80M items touch only 2.8% of items, yielding up to $30\times$ reduction in memory reads (Xue et al., 18 Nov 2025).

4. GPU Implementation Details and Hardware Efficiency

Fused Int8 kernels are designed for high throughput on modern GPUs by exploiting hardware-specific instructions and maximizing memory access efficiency:

• Each kernel maps a query or batch to a thread block; warps handle contiguous blocks (TILE) of embeddings in lock-step.
• Coalesced, 128-byte-wide loads maximize streaming throughput from HBM to registers/shared memory.
• Quantized dot-products use dp4a (4x int8 $\times$ int8 $\to$ int32 per cycle).
• Top-k heaps reside in registers; across-warps merges utilize warp shuffles and reductions.
• By quantizing embeddings and keeping all steps device-resident, index and buffer footprints are minimized; dense, vectorized computation approaches the memory bandwidth limit.
• Fused design eliminates the need for large intermediate buffers, which for classic ANN can require 35 MB at 32 probes for 20M items—reduced to 18 MB in the fused approach (Xue et al., 18 Nov 2025).

5. Performance Characteristics and Comparative Evaluation

Fused Int8 ANN search demonstrates major speed, throughput, and cost-efficiency advantages over float32-based and classic unfused ANN systems:

• Standalone search (20M items, $d=128$, batch=16, top-k=2048, recall=0.90): Faiss GPU (float) latency $\approx$ 8 ms vs. SilverTorch fused Int8 $\approx$ 0.9 ms—$9\times$ lower latency.
• Latency speedup versus Faiss GPU ranges $2.2\times$–$14.7\times$ across recall targets; up to $50\times$ vs. CPU-based HNSW or Faiss.
• End-to-end retrieval (80M items, 24 probes, top-k=1024, 2 GPUs): SilverTorch QPS=1210 ($23.7\times$ that of CPU service), $P99 = 14$ ms (5.6$\times$ lower latency than CPU).
• Cost efficiency: $0.0077$ per 1k requests for SilverTorch versus $0.158$ for baseline CPU ($20.9\times$ cost improvement), with maintained or very slightly reduced recall ($\approx$0.98 maximum recall) (Xue et al., 18 Nov 2025).

System	QPS	\$ per 1k req	Latency
Baseline–CPU	51	0.158	$P99 =160$ms
Faiss GPU (float, forward)	186	0.045	$P99 \approx 60$ms
SilverTorch Int8 fused	1210	0.0077	$P99 =14$ms

This demonstrates that fused Int8 ANN kernels deliver order-of-magnitude throughput and cost advantages at the same or slightly reduced accuracy relative to established methods.

6. Design Trade-offs and Limitations

The adoption of Int8 quantization for ANN search introduces a modest limitation on maximum attainable recall (e.g., recall plateaus at $\approx$0.98). However, in production retrieval systems (e.g., as in SilverTorch), downstream re-ranking layers (such as OverArch neural scoring) mitigate minor recall loss at the ANN stage. Bloom filter integration (for pre-filtering) achieves substantial reductions in memory reads and intermediate buffer requirements; false positives (e.g., 0.07% at 1024 bits/item) are pruned downstream. The approach is highly tuned for modern CUDA (NVIDIA) GPU architectures and exploits device-specific instructions; porting to other platforms may necessitate substantial retuning (Xue et al., 18 Nov 2025).

A plausible implication is that this paradigm—fusing quantized, memory-local steps of ANN search into single GPU kernels—can generalize to other vector-retrieval domains, including semantic search and image retrieval, wherever high-dimensional embedding search is a bottleneck.

7. Broader Applicability and Extensions

Fused Int8 ANN search is directly applicable to large-scale recommendation serving (as in SilverTorch), supporting multi-task retrieval, learned similarity metrics, and tight integration with GPU-resident model architectures. The fused, tensor-native design enables straightforward extension to multi-task aggregation (e.g., Value Model) and complex neural scoring (e.g., OverArch), all within unified frameworks such as PyTorch. This design is currently deployed at scale, serving recommendations for billions of users across hundreds of models (Xue et al., 18 Nov 2025). The ANN search kernel’s generality suggests applicability to other large-scale vector-retrieval tasks where throughput, latency, and cost-efficiency are paramount.

• SilverTorch: "SilverTorch: A Unified Model-based System to Democratize Large-Scale Recommendation on GPUs" (Xue et al., 18 Nov 2025)
• General quantized NAS approaches for efficient inference: "SpaceEvo: Hardware-Friendly Search Space Design for Efficient INT8 Inference" (Zhang et al., 2023)