SPLADE-Code: Sparse Retrieval for Code

• The paper’s main contribution is showing that learned expansion tokens bridge lexical and semantic gaps in code search.
• SPLADE-Code employs a bi-encoder to map queries and code into sparse vectors, enabling efficient dot product scoring with an inverted index.
• Benchmark results demonstrate state-of-the-art performance and sub-millisecond latency, validating sparse models for large-scale code retrieval.

SPLADE-Code is a family of learned sparse retrieval models specialized for code retrieval. Introduced in "On the Challenges and Opportunities of Learned Sparse Retrieval for Code" (Lupart et al., 23 Mar 2026), it adapts the SPLADE paradigm to large code search settings by representing queries and code documents as sparse vectors over a vocabulary, scoring them with a dot product, and serving retrieval through an inverted index rather than approximate nearest-neighbor search. The model family spans 600M to 8B parameters, is trained in a lightweight one-stage training pipeline, and is reported to achieve state-of-the-art performance among retrievers under 1B parameters (75.4 on MTEB Code) and competitive results at larger scales (79.0 with 8B). A central claim of the work is that, despite longstanding skepticism about sparse retrieval for code, learned expansion tokens can bridge lexical and semantic matching while preserving the efficiency and interpretability associated with sparse indexing (Lupart et al., 23 Mar 2026).

1. Research setting and motivation

SPLADE-Code emerges from a specific systems problem: retrieval over large codebases has become a key component of modern LLM-based software engineering systems, yet most existing approaches rely on dense embedding models. In that setting, learned sparse retrieval had remained comparatively underexplored for code, even though sparse methods offer the practical advantages of classical inverted indexing, transparent term activation, and straightforward scaling (Lupart et al., 23 Mar 2026).

The paper identifies several reasons why code retrieval is unusually difficult for sparse methods. These include subword fragmentation, because identifiers and mixed alphanumeric code tokens are often split into awkward subwords; semantic gaps between natural-language queries and code, because intent words may not literally occur in code snippets; multilingual and codebase diversity, because evaluation spans many programming languages and task types; and long documents and sparse-index growth, because long snippets or files can make representations denser and therefore slower to search (Lupart et al., 23 Mar 2026). The authors also emphasize a more operational issue: sparse methods are only attractive if sparsity remains controlled, since long posting lists and dense sparse vectors undermine latency advantages.

Against that background, SPLADE-Code is positioned not as a minor variant of an existing code retriever, but as a proof that learned sparse retrieval can work for code search despite these obstacles. A common misconception is that sparse code retrieval can only reproduce lexical matching over code tokens. The paper argues the opposite: code retrieval requires a representation that retains lexical precision while also supporting semantic abstraction, especially for text-to-code and code-to-code tasks (Lupart et al., 23 Mar 2026).

2. Sparse formulation and retrieval mechanism

SPLADE-Code is a bi-encoder learned sparse retrieval model based on SPLADE. Queries and documents are mapped to high-dimensional sparse vectors over the model vocabulary. In the paper’s formulation, a sparse representation is

$$\mathbf{u} = (u_1,\dots,u_N) \in \mathbb{R}^N,\qquad \|\mathbf{u}\|_0 \ll N$$

and query-document similarity is computed by a dot product, $\mathbf{u} \cdot \mathbf{v}$ (Lupart et al., 23 Mar 2026).

The SPLADE-style encoding step uses token-level scores aggregated by max pooling with log-saturation:

$$\mathbf{u}_t = \max_{i \in \{1,\ldots,n\}} \log(1+\mathrm{ReLU}(\mathbf{z}_{i,t}))$$

This is the standard SPLADE pattern: each vocabulary term receives an activation, and the model may assign nonzero weight not only to tokens present in the input but also to expansion tokens inferred from context (Lupart et al., 23 Mar 2026).

That expansion mechanism is especially important for code. The paper’s qualitative examples describe a recursive Fibonacci function activating terms such as Fibonacci, fib, recursive, and Python, while an SQL snippet activates oldest and MAX(age) rather than only generic SQL syntax. In this formulation, sparse retrieval is not limited to literal overlap. Instead, it preserves an indexable lexical surface while allowing the model to activate semantically relevant vocabulary dimensions that bridge gaps between natural language and code, between two code snippets, or across programming languages (Lupart et al., 23 Mar 2026).

This design places SPLADE-Code within the broader learned sparse retrieval lineage. Earlier SPLADE work on text retrieval had already shown that sparse lexical representations can carry both exact-match and expansion-based signals (Mackenzie et al., 2023). A plausible implication is that SPLADE-Code inherits the same representational flexibility, but under the stricter constraints imposed by code tokenization and cross-modal query-code mismatch.

3. Model family and training pipeline

The released SPLADE-Code family is built on Qwen3-0.6B, Qwen3-1.7B, and Qwen3-8B, yielding models from 600M to 8B parameters (Lupart et al., 23 Mar 2026). Training uses the CoIR benchmark training split, which contains 2.2M training samples from 10 datasets. The training recipe is described as a single-stage pipeline, and the paper explicitly notes that this setup is lighter than several dense code embedding baselines, which often use multi-stage training (Lupart et al., 23 Mar 2026).

The reported configuration includes LoRA adaptation with rank 64, batch size 256, 7 or 8 negatives per query depending on model size, maximum input length 512, learning rate 1e-4, KLD distillation loss with temperature 300, and a FLOPs penalty to encourage sparsity. Negatives are mined using Qwen3-Embedding-0.6B and reranked by Qwen3-Reranker-4B, with negatives selected from ranks 50–100 (Lupart et al., 23 Mar 2026).

A technically significant modification is the treatment of the backbone attention pattern. The paper states that decoder-only causal attention is suboptimal for lexical matching, so the models are adapted with bidirectional attention and a masked-token-style adaptation on MS MARCO passages. Final checkpoints are combined using weighted spherical merging (Lupart et al., 23 Mar 2026). This combination of sparse retrieval objectives with decoder-style backbone adaptation distinguishes SPLADE-Code from standard text-only SPLADE variants based on masked-language-model heads.

The paper’s emphasis on a lightweight pipeline matters because it frames SPLADE-Code as a practical alternative rather than an architecture that depends on elaborate multi-stage distillation. That positioning parallels, in a different domain, the argument made by "Mistral-SPLADE: LLMs for better Learned Sparse Retrieval" (Doshi et al., 2024), where stronger backbones are used to simplify other parts of the sparse-retrieval recipe.

4. Benchmarks, effectiveness, and scaling behavior

SPLADE-Code is evaluated on CoIR, MTEB Code, CodeRAG-Bench, and CPRet, using nDCG@10 as the standard metric (Lupart et al., 23 Mar 2026). In the controlled comparison, the paper reports the following results:

• SPLADE-Code-0.6B: CoIR 72.6, MTEB Code 73.5, CodeRAG 63.7, CPRet 56.4
• SPLADE-Code-1.7B: CoIR 74.3, MTEB Code 75.1, CodeRAG 64.9, CPRet 62.5
• SPLADE-Code-8B: CoIR 76.7, MTEB Code 77.8, CodeRAG 68.5, CPRet 76.8 (Lupart et al., 23 Mar 2026)

For comparison, Dense-8B is reported at CoIR 76.0, MTEB Code 75.6, CodeRAG 65.8, and CPRet 76.6, so the paper characterizes SPLADE-Code-8B as better on CoIR, MTEB Code, and CodeRAG, and essentially tied or slightly better on CPRet (Lupart et al., 23 Mar 2026).

On MTEB Code, the merged models are highlighted separately: SPLADE-Code-0.6B reaches 75.4, and SPLADE-Code-8B reaches 79.0. The paper claims that SPLADE-Code-0.6B is state-of-the-art among systems under 1B parameters, while SPLADE-Code-8B is highly competitive at larger scale (Lupart et al., 23 Mar 2026). It also notes best scores on some individual datasets, including CodeSearchNet, CodeTrans-Contest, and StackOverflow-QA.

The out-of-domain results are particularly important to the paper’s argument. On CodeRAG-Bench, SPLADE-Code-0.6B improves over C2-LLM-0.5B by +5.8 nDCG@10, and SPLADE-Code-8B by +6.1. On CPRet, the gains are +7.5 and +4.4, respectively (Lupart et al., 23 Mar 2026). The authors also note that on tasks such as DS-1000, ODEX, and text-to-code retrieval, BM25 is often near-zero or very low, while SPLADE-Code improves by roughly 30 nDCG points. This supports the claim that code retrieval requires more than literal lexical overlap.

5. Expansion tokens, ablations, and interpretability

A central empirical result is that expansion tokens are crucial. The paper compares SPLADE-Code against SPLADE-lexical, a variant that uses token weighting without learned expansion, and reports that the lexical-only system performs badly, close to BM25 and far below SPLADE-Code (Lupart et al., 23 Mar 2026). This directly addresses the misconception that sparse code retrieval succeeds primarily through identifier matching. In the authors’ account, the method works because expansion terms connect natural-language intent words to code tokens, link semantically similar snippets across languages, and encode algorithmic concepts in sparse vocabulary dimensions.

The paper gives a rough estimate that among the top-25 activated terms, about 35% come from the input and 65% are expansion terms (Lupart et al., 23 Mar 2026). That proportion is presented as evidence that semantic enrichment, rather than direct copying, dominates the sparse representation.

Ablations over backbones further indicate that the method is not tightly coupled to a single LLM family. The paper reports experiments with Qwen3-8B, Llama3-8B, Qwen2.5Coder-0.5B / 1.5B / 7B, and sparse autoencoder variants from SPLARE, and states that performance is fairly stable across backbones (Lupart et al., 23 Mar 2026). This is interpreted as robustness to the projection vocabulary.

The training-pipeline ablations are also informative. The paper tests base training, an instruction-tuned version, English intermediate finetuning, and a contrastive variant. The findings are that instruction tuning barely changes performance, intermediate English finetuning hurts, and contrastive-only alternatives are also worse (Lupart et al., 23 Mar 2026). The authors interpret this as evidence that learned sparse retrieval already maintains a workable balance between lexical matching and semantic abstraction.

Interpretability is treated as a substantive property rather than a rhetorical advantage. The examples of activations such as pivot, sort, array, and swap for Java quicksort, or oldest and MAX(age) for SQL, are intended to show that SPLADE-Code remains sparse and inspectable even when it is semantically expansive (Lupart et al., 23 Mar 2026).

6. Efficiency, pruning, and relation to the wider SPLADE systems literature

Efficiency is one of the paper’s strongest practical claims. SPLADE-Code evaluates sparse retrieval with the Seismic inverted-index library and dense retrieval with HNSW via FAISS, with all latency measurements run on a single AMD EPYC 7313 CPU and top-1000 retrieval in all cases (Lupart et al., 23 Mar 2026). To control sparsity, it uses pruning settings such as (500,1000) for final retrieval and (10,100) for the first stage of a two-step setup.

The headline result is that SPLADE-Code can achieve below 1 ms per query on the CodeSearchNet 1M-passage collection, with only a marginal ranking loss (Lupart et al., 23 Mar 2026). By contrast, BM25 is reported at about 49–50 ms/query on the same setting, which the paper attributes to the long average query and document lengths found in code retrieval. The appendix notes that query encoding time is not included in the latency figure and that, for two-step SPLADE, the extra cost is approximated by computing 1000 dot products, estimated at about 50 microseconds for vectors of size 100K with only 400 non-zero dimensions (Lupart et al., 23 Mar 2026).

These results place SPLADE-Code within a broader engineering trajectory for learned sparse retrieval. On web titles at scales up to 9 billion titles, other work has shown that SPLADE-style models are consistently more effective than BM25 but require pruning strategies such as document-centric pruning, top-$k$ query term selection, and Boolean thresholding to achieve a practical effectiveness-efficiency balance (Won et al., 27 Nov 2025). Production-oriented work on DF-FLOPS similarly argues that latency bottlenecks arise from high-document-frequency terms and reports that directly penalizing those terms can make SPLADE-Doc around 10x faster while keeping latencies on par with BM25 (Porco et al., 21 May 2025). At the systems level, GPUSparse demonstrates that exact SPLADE scoring need not remain CPU-bound, reporting 235x speedup over Pyserini CPU at 8.8M documents through GPU-resident inverted indices and fused scatter-add kernels (Sharma, 24 Jun 2026).

A plausible implication is that SPLADE-Code’s sub-millisecond result is not an isolated code-search artifact but part of a more general maturation of learned sparse retrieval into a deployable systems paradigm. Within that paradigm, SPLADE-Code contributes domain specialization for code, while related SPLADE work contributes collection-level pruning, term-frequency-aware regularization, and alternative serving backends.

In summary, SPLADE-Code establishes learned sparse retrieval as a serious option for code search. Its contribution lies not only in benchmark effectiveness, but in showing that sparse lexical representations with learned expansion tokens can handle the semantic gaps, multilingual diversity, and efficiency constraints that characterize modern code retrieval workloads (Lupart et al., 23 Mar 2026).