---
title: 'GPUSparse: GPU Sparse Retrieval'
url: https://www.emergentmind.com/topics/gpusparse
type: topic
---

# GPUSparse: GPU Sparse Retrieval

GPUSparse is a GPU-centric system for exact learned sparse retrieval that targets the inference-time bottleneck of SPLADE-style ranking by replacing CPU-bound inverted-index traversal with GPU-parallel scoring over a GPU-resident inverted index. It introduces a GPU-parallel inverted index with block-aligned, warp-coalesced posting lists, a batched scatter-add scoring algorithm that processes hundreds of queries simultaneously, and fused Triton kernels accompanied by an analysis of the tradeoff between work-efficiency and hardware utilization. On MS MARCO passage ranking with 8.8M passages and real SPLADE embeddings, it matches CPU exact scoring to three decimals while reducing per-query latency from 298 ms in Pyserini SPLADE to 1.27 ms and reaching 787 QPS at batch 500 on an H100 GPU [2606.26441].

## 1. Retrieval model and problem setting

GPUSparse is designed for learned sparse retrieval models such as SPLADE, which represent queries and documents as sparse weighted term vectors over a vocabulary and score them by inner product:
$$
\text{score}(q,d)=\mathbf{s}(q)^\top \mathbf{s}(d).
$$
The system addresses the specific mismatch between this scoring rule and the dominant CPU execution model for sparse retrieval. Traditional exact retrieval systems rely on inverted indices with traversal algorithms such as WAND and Block-Max WAND, which maintain posting-list iterators, choose pivots, perform conditional skipping, and depend on fine-grained branching. The paper characterizes these procedures as inherently sequential and therefore poorly matched to GPUs, which prefer regular memory access, massive parallelism, and limited warp divergence [2606.26441].

The motivation is not to alter the retrieval objective, but to preserve exact learned sparse retrieval while removing the CPU traversal bottleneck. This is important because learned sparse retrieval models preserve the interpretability and exact-match advantages of sparse representations, yet their inference-time scoring had remained tied to CPU-oriented indexing algorithms. GPUSparse therefore treats sparse retrieval as a GPU systems problem rather than as a direct port of CPU WAND/BMW logic. The paper also contrasts this with dense retrieval, where GPU matrix multiplication is already natural, and argues that learned sparse retrieval should not remain CPU-bound solely because its classic implementation is organized as inverted-list traversal [2606.26441].

## 2. GPU-resident inverted index and memory layout

The core data structure is a GPU-resident inverted index whose layout is explicitly tailored to SIMT execution. Instead of a CPU-oriented compressed posting-list representation, GPUSparse stores flattened `doc_ids` and `scores` arrays together with `offsets[t]`, `lengths[t]`, `padded_lengths[t]`, and `max_scores[t]` for each term. Posting lists are padded to a multiple of the warp size:
$$
\text{padded\_length}(t)=\left\lceil \frac{|\text{PL}(t)|}{W}\right\rceil \times W,\qquad W=32.
$$
Padding uses dummy entries with `doc_id = -1` and `score = 0`, which are masked out during scoring [2606.26441].

This organization is intended to make GPU loads efficient. The posting lists are block-aligned, neighboring threads access neighboring memory locations, and the flat arrays avoid pointer chasing and irregular indirection. The design deliberately omits delta coding and related CPU-friendly compression mechanisms because they introduce sequential decompression dependencies. The paper gives the resulting memory estimate as
$$
\text{Memory} \approx N \cdot \bar{k} \cdot (4+4)\cdot (1+\epsilon_{\text{pad}}),
$$
where $N$ is the number of documents, $\bar{k}$ is the average number of nonzero terms per document, `4+4` denotes 4 bytes for the document ID and 4 bytes for the score, and $\epsilon_{\text{pad}}$ is the padding overhead. For SPLADE on MS MARCO, the index remains compact enough to fit large collections on one GPU; the full 8.8M-passage index occupies 8.5 GB [2606.26441].

## 3. Scatter-add scoring and fused Triton execution

The central algorithmic reformulation in GPUSparse is to treat exact sparse retrieval as scatter-add over posting lists rather than as top-down traversal with pruning. For a query with nonzero terms $\{(t_i,w_i)\}$, the score decomposes as
$$
\text{score}(q,d)=\sum_{i=1}^{|q|} w_i \cdot s_d(t_i),
$$
which can be executed as independent updates over the posting list of each query term:
$$
scores[d] \mathrel{+}= w_i \cdot PL(t_i)[d] \quad \forall d \in PL(t_i).
$$
For each query term, GPUSparse loads the corresponding posting list, multiplies each posting weight by the query term weight, and atomically adds the contribution into the document accumulator. Because no documents are skipped and no lists are pruned, this is exact scoring rather than approximation [2606.26441].

The execution model is batched. For batch size $B$ and up to $M$ query terms, GPUSparse launches a two-dimensional grid indexed by query and term position; each Triton program processes one `(query, term)` pair and atomically updates a `[B, N]` score buffer. The implementation is fused to avoid Python-level looping, repeated kernel launches, and intermediate allocations. The paper reports a default chunk size of `BLOCK_PL=128`. It also notes that the scatter-add formulation is shared with SPARe’s iterative mode, while the contribution of GPUSparse is the fused-kernel realization, measured as 23–270x faster than a faithful SPARe iterative reimplementation [2606.26441].

## 4. Work-efficiency and bandwidth-efficiency

A distinctive feature of GPUSparse is its explicit analysis of two competing GPU execution strategies. The first is the term-parallel scatter-add kernel, which processes only query-matching postings and has work complexity of roughly
$$
O(B \cdot \bar{q} \cdot \bar{L}),
$$
where $\bar{q}$ is the average number of query terms and $\bar{L}$ is the average posting-list length. This kernel is work-efficient, but its atomic writes are scattered and its bandwidth utilization is low. The second is a document-parallel CSR gather kernel in which each program handles one `(query, doc)` pair, iterates over all document terms, and looks up query weights, with complexity roughly
$$
O(B \cdot N \cdot \bar{k}).
$$
This version has more regular memory access and much higher hardware utilization, but it performs far more total work [2606.26441].

The paper’s measurements show that the document-parallel kernel reaches 2,097 GB/s, or 62.6% of H100 peak HBM bandwidth, whereas the scatter-add kernel reaches only 12.5 GB/s, or 0.37% of peak. Yet the higher-bandwidth kernel is slower in wall-clock time. At 100K documents, the scatter-add kernel reads 0.09 GB per batch and runs in 7.3 ms, while the document-parallel kernel reads 76.3 GB per batch and runs in 36.4 ms. A common misconception is that the better GPU sparse retrieval kernel must be the one that attains higher HBM utilization; GPUSparse argues the opposite for this workload, namely that work-efficiency dominates bandwidth-efficiency when sparse scoring can avoid touching the overwhelming majority of document-term entries [2606.26441].

## 5. Evaluation, accuracy, and comparison with prior retrieval systems

The main evaluation uses MS MARCO passage ranking with the full 8.8M-passage collection, the official dev-small set of 6,980 queries, and real embeddings from `SPLADE-cocondenser-ensembledistil`. The reported representation statistics are an average document sparsity of 127.2 nonzeros, an average query sparsity of 49.9 nonzeros, a vocabulary of 30,522, and score values approximately in $[0, 3.5]$. The hardware is an NVIDIA H100 80GB SXM5 GPU with CUDA 12.4; the CPU baseline uses an Intel Xeon Gold 6448Y [2606.26441].

At full scale, GPUSparse matches the CPU exact reference to three decimals in MRR@10 and nDCG@10 while dramatically reducing latency. The paper also reports Recall@1000 $\ge 0.999$ against dense matmul, with the residual attributed to floating-point tie-breaking. In the direct system comparison against Pyserini SPLADE, both systems report Recall@1000 = 0.983 on the 8.8M collection.

| System | Quality | Latency / throughput |
|---|---|---|
| GPUSparse | MRR@10 0.383; nDCG@10 0.449; Recall@1000 0.983 | 1.27 ms/query; 787 QPS at batch 500 |
| Pyserini SPLADE | MRR@10 0.383; nDCG@10 0.449; Recall@1000 0.983 | 298 ms/query |
| Seismic | MRR@10 0.326; Recall@1000 0.738 | 206 µs/query for top-1000 |

These numbers define the paper’s main empirical claims. Relative to Pyserini SPLADE, GPUSparse provides a 235x speedup at 8.8M documents, from 298 ms to 1.27 ms per query. Relative to Seismic, described as the fastest CPU sparse retrieval system in the paper, GPUSparse achieves exact scoring at 787 QPS throughput on the full 8.8M collection, while Seismic trades 25% recall for speed, with Recall@1000 = 0.738 versus 0.983 exact. At smaller scales, GPUSparse also outperforms dense and library baselines: at 100K passages it achieves 15 µs/query, compared with 117 µs/query for dense matmul, and it is reported as 8.0x faster than dense matmul and 6.3x faster than cuSPARSE SpMV in the same batch setting. Dense exact scoring also becomes memory-prohibitive at larger scales; the paper notes that 1M documents $\times$ 30K vocabulary corresponds to approximately 114 GB in dense form, exceeding H100 memory, whereas the sparse index scales to the full collection within 8.5 GB. For the end-to-end pipeline including SPLADE encoding, scoring, and top-$k$, the paper reports 0.43 ms/query at batch 32 and 0.36 ms/query at batch 128 on 1M documents, with throughput of 2,758 QPS at batch 128 [2606.26441].

## 6. Significance and relation to GPU sparse systems

The principal significance of GPUSparse is that it demonstrates exact learned sparse retrieval need not remain a CPU-bound service primitive. The system does not introduce an approximate shortcut to make sparse retrieval GPU-friendly; instead, it reorganizes the exact computation around a GPU-resident inverted index, batched scatter-add, and fused kernels. This makes the paper notable in two separate senses: it preserves the ranking semantics of exact SPLADE scoring, and it shows that the main obstacle was algorithmic formulation rather than an intrinsic incompatibility between sparse retrieval and GPU hardware [2606.26441].

A plausible interpretation is that GPUSparse belongs to a broader class of GPU sparse systems that obtain performance by reformulating irregular sparse workloads around GPU-friendly primitives rather than by directly transplanting CPU execution logic. GraphBLAST does this for graph traversal through sparsity-aware GraphBLAS primitives and backend selection [1908.01407]. tSparse does it for spGEMM by repurposing Tensor Cores through bitmap-based tiling [2009.14600]. gSoFa does it for nonsymmetric symbolic LU factorization by turning symbolic fill computation into fine-grained SIMT graph traversal [2007.00840]. PSCToolkit does it for large-scale SPD linear systems through GPU-aware sparse formats, Krylov methods, and AMG preconditioners across up to 8192 GPUs [2406.19754]. In that context, GPUSparse extends the same design pattern to learned sparse retrieval: exactness is retained, but the computation is re-expressed in a form that matches warp-coalesced memory access, massive batching, and fused SIMT execution.

The paper also narrows a common distinction between “exact but slow” and “fast but approximate” sparse retrieval. Its comparison with Pyserini and Seismic indicates that this distinction was partly an artifact of CPU traversal algorithms. GPUSparse does not eliminate all tradeoffs—the paper is explicit about the tension between work-efficiency and hardware utilization—but it shows that exact learned sparse retrieval can be served at millisecond latency and high batch throughput on modern GPUs.

Source: https://www.emergentmind.com/topics/gpusparse