GPUSparse: GPU-Accelerated Learned Sparse Retrieval with Parallel Inverted Indices

Abstract: Learned sparse retrieval models such as SPLADE achieve retrieval quality competitive with dense models while preserving the interpretability and exact-match advantages of sparse representations. However, inference-time scoring still relies on CPU-bound inverted index traversal algorithms (WAND, Block-Max WAND), creating a fundamental bottleneck for real-time serving at scale. We present GPUSparse, a system for GPU-accelerated exact learned sparse retrieval that introduces: (1) a GPU-parallel inverted index with block-aligned, warp-coalesced posting lists; (2) a batched scatter-add scoring algorithm that processes hundreds of queries simultaneously; and (3) fused Triton kernels with an analysis of the tradeoff between work-efficiency and hardware utilization. On MS MARCO passage ranking (8.8M passages) with real SPLADE embeddings, GPUSparse matches CPU exact scoring to three decimals (MRR@10=0.383, equal to Pyserini SPLADE at this precision; Recall@1000>=0.999 vs. dense matmul, the residual from floating-point tie-breaking) while providing a 235x speedup over Pyserini CPU at 8.8M documents (1.27ms vs. 298ms per query). Compared to Seismic (the fastest CPU sparse retrieval system), which trades 25% recall for speed (R@1000=0.738 vs. 0.983 exact), GPUSparse achieves exact scoring at 787 QPS throughput (batch 500) on the full 8.8M collection, with 1.3ms per query. Our document-parallel kernel reaches 62.6% of H100 peak HBM bandwidth, revealing a fundamental work-efficiency vs. bandwidth-efficiency tradeoff in GPU sparse retrieval. The reformulation of sparse scoring as scatter-add over an inverted index is shared with SPARe's iterative mode; our contribution is its fused-kernel realization, which we measure to be 23-270x faster than a faithful SPARe iterative reimplementation.

• The paper introduces a GPU-native scatter-add method to perform exact learned sparse retrieval with up to 235× speedup over CPU baselines.
• The method reformulates retrieval scoring as parallel scatter-add operations, optimizing memory access with warp-coalesced arrays and fused Triton kernels.
• Empirical evaluation on MS MARCO shows high recall (R@1000=0.983) and low latency, underscoring its potential for large-scale, real-time applications.

GPU-Accelerated Exact Learned Sparse Retrieval with GPUSparse

Introduction

GPUSparse presents a GPU-native architecture for exact learned sparse retrieval, focusing on SPLADE embeddings and related sparse lexical models. The primary innovation is the elimination of traditional CPU-bound inverted index bottlenecks, common in scoring algorithms such as WAND and Block-Max WAND (BMW), by fully leveraging GPU parallelism through a novel scatter-add approach over an optimized inverted index. The system specifically targets high-throughput, real-time, and large-scale serving scenarios while maintaining exact retrieval semantics.

Learned Sparse Retrieval and Bottlenecks

Contemporary sparse retrieval models (e.g., SPLADE, LACONIC, uniCOIL) utilize transformer-based architectures to produce high-dimensional sparse vectors over large vocabularies. These approaches combine competitive dense-level retrieval performance with the interpretability of lexical methods. However, at inference, such learned sparse models remain predominantly CPU-bound due to the dependence on inverted index traversal algorithms that are inherently sequential and resistant to GPU parallelization because of pivot-selection and pruning logic.

The current state-of-the-art methods like Seismic introduce geometric blocking and aggressive term pruning to approximate retrieval for speed, sacrificing recall (empirically, R@1000=0.738 at 8.8M docs with SPLADE). In contrast, dense retrieval on GPUs simply executes large-scale matrix multiplication, fully exploiting the hardware’s bandwidth and compute.

GPUSparse System Properties

GPUSparse bridges the gap between the sparse and dense retrieval paradigms by introducing:

1. A GPU-Parallel Inverted Index Structure: Posting lists are stored in flat, block-aligned, warp-coalesced arrays, eschewing variable-length encoding and sequential delta-decoding. This enables efficient and coalesced memory access patterns for large batches.
2. Batched Scatter-Add Scoring Algorithm: Scoring is reformulated as a set of independent scatter-add operations, one per query-term pair, which can be executed in embarrassingly parallel fashion on the GPU. This replaces sequential skipping logic with massive parallelization.
3. Fused Triton Kernel Implementation: The core operation is realized as a single Triton GPU kernel that fuses posting list traversal, score computation, and atomic scatter-add, minimizing kernel launch overheads and intermediate memory usage.
4. Comprehensive Analysis of Hardware Tradeoff: The system characterizes the tradeoff between work efficiency (sparse, targeted computation) and hardware utilization (bandwidth efficiency).

Algorithmic Reformulation

Given a batch of $B$ queries, each with up to $M$ nonzero terms, the scatter-add approach for scoring computes the inner product between each query and candidate document over their shared nonzero vocabulary dimensions. Instead of the document-level skipping of WAND/BMW, the kernel processes every posting for each nonzero term in the query, realizing a direct and exact computation with minimal branching.

This approach guarantees Recall@$k \geq 0.999$ versus reference dense matmul, with any residual attributable to floating-point tie-breaking rather than substantive scoring errors. No approximation or top-$k$ heap coordination is performed—the ranking is exact.

Kernel Design and Efficiency Analysis

Performance on GPU is determined by both work and bandwidth efficiency. The scatter-add kernel, though bandwidth-inefficient (12.5 GB/s on H100, 0.37% of peak), is highly work-efficient, reading and writing only necessary postings that intersect with active query terms, and achieves dramatic wall-clock speedups—up to 235$\times$ over CPU exact scoring (Pyserini SPLADE at 8.8M documents).

A document-parallel kernel variant saturates 62.6% of peak bandwidth (2097 GB/s), but incurs a prohibitive work overhead by processing every document-term pair, resulting in slower wall-time performance except for extremely small collections. The fundamental design tension is between memory transfer volume (doc-parallel) and processing only active intersections (scatter-add), and for real-world SPLADE sparsity profiles, the scatter-add approach dominates for all practical collection sizes.

Empirical Evaluation

GPUSparse is evaluated on the MS MARCO passage ranking benchmark using SPLADE embeddings, covering scales from 100K to 8.8M documents. Key results include:

• Retrieval Quality: MRR@10 = 0.383, nDCG@10 = 0.449, Recall@1000 = 0.983 on 8.8M documents, exactly matching CPU SPLADE implementations.
• Batch Throughput: 787 QPS at 8.8M scale with 1.27ms per-query latency (batch size 500).
• Comparison to Baselines:
  • 235$\times$ speedup over Pyserini SPLADE CPU exact
  • 6.3$\times$ speedup over cuSPARSE batched SpMV
  • 8.0$\times$ speedup over dense matrix multiplication (torch.mm) at 100K scale
  • Seismic achieves lower latency (206μs/query) but at the cost of 25% recall reduction versus exact
• Memory Utilization: At full MS MARCO scale (8.8M docs), the posting index consumes 8.5 GB GPU memory, and the complete system (including accumulation buffer) fits comfortably on a commodity H100 GPU.
• Cross-Domain Robustness: On BEIR datasets (e.g., SciFact, NFCorpus, TREC-COVID), the system maintains high-fidelity and low latency.

Implications and Future Directions

GPUSparse demonstrates that with a suitable reformulation, learned sparse retrieval can be executed exactly and efficiently on modern GPUs, negating the traditional requirement for CPU-bound scoring. This opens up the use of interpretable, exact sparse retrieval models for both research and production at web scale, provided that batch serving is feasible and GPU resources are available.

Potential avenues for theoretical and practical improvement include:

• Hybrid kernels that capture both work and bandwidth efficiency through intra- and inter-warp reductions.
• Compressed or quantized posting lists for scaling beyond single GPU limits.
• Integration with adaptive batching for variable and streaming workloads.
• Efficient multi-GPU merging and distributed top-$k$ computation for 100M+ collection regimes.

Conclusion

GPUSparse establishes a new paradigm for learned sparse retrieval on GPU, yielding exact results at orders-of-magnitude higher throughput compared to CPU baselines, without resorting to approximate methods or sacrificing recall. The architecture is relevant for settings where high throughput, batch processing, and exact correctness are required, particularly when GPU resources are already provisioned (e.g., for end-to-end deep inference pipelines). The fundamental insights into work versus bandwidth efficiency have broader implications for the design of sparse algorithms on modern manycore hardware.