SPLARE: Sparse Latent Retrieval

• SPLARE is a learned sparse retrieval model that uses autoencoders to replace fixed token vocabularies with latent, semantically rich features, enhancing multilingual retrieval.
• It employs a GPT-style LLM backbone with a sparse autoencoder projection to produce high-dimensional, interpretable sparse vectors compatible with inverted-index search.
• Empirical evaluations demonstrate SPLARE’s competitive nDCG performance and balanced efficiency over benchmarks like MS MARCO, MIRACL, and XTREME-UP in multilingual settings.

Searching arXiv for SPLARE and closely related papers to ground the article in the current literature. arXiv search query: "SPLARE sparse autoencoders retrieval multilingual" SPLARE, short for “SParse LAtent REtrieval,” is a learned sparse retrieval method that replaces the vocabulary-based sparse space used by earlier learned sparse retrieval models with a latent feature space learned by sparse autoencoders over LLM activations. Queries and documents are encoded as high-dimensional, highly sparse vectors whose dimensions correspond to latent features rather than lexical items, allowing inverted-index retrieval while aiming for better semantic structure, stronger multilingual transfer, and greater interpretability than token-vocabulary approaches such as SPLADE (Formal et al., 27 Feb 2026). In multilingual retrieval settings, SPLARE is described as producing “generalizable sparse latent representations” and a shared, language-agnostic “Bag-of-Concepts” space, and it has been evaluated both on broad retrieval benchmarks and in the WSDM Cup 2026 multilingual document retrieval task (Formal et al., 24 Feb 2026).

1. Definition and conceptual position

SPLARE is a learned sparse retrieval (LSR) model. Its central departure from prior LSR systems is that the sparse dimensions are not tied to the fixed token vocabulary of a pretrained LLM. Instead, SPLARE uses sparse autoencoders (SAEs) to define a latent sparse space in which each dimension is a learned feature. This substitutes sparse bag-of-words with sparse bag-of-features, or, in the multilingual formulation, a language-agnostic Bag-of-Concepts representation (Formal et al., 27 Feb 2026).

This design is motivated by limitations in three established retrieval families. Classical sparse retrieval such as BM25 is efficient and naturally compatible with inverted indexes, but it is purely lexical and language-specific, with no mechanism for semantic sharing across languages (Formal et al., 24 Feb 2026). Vocabulary-based LSR methods such as SPLADE retain sparse scoring and indexing, but remain constrained by the underlying model vocabulary; in multilingual settings this creates very large vocabularies, tokenization redundancy, and imperfect alignment across languages (Formal et al., 27 Feb 2026). Dense LLM-based embedding models such as Qwen3-8B-Embed and Gemini Embeddings map texts into a shared continuous space and show strong multilingual generalization, but they depend on dense indexes and approximate nearest-neighbor search rather than classical inverted-index retrieval (Formal et al., 24 Feb 2026).

SPLARE therefore occupies an intermediate position. It preserves the sparse retrieval paradigm—high-dimensional sparse representations, sparse dot products, and inverted indexes—while abandoning the rigid token-vocabulary constraint. The claimed consequence is a retrieval space that is more semantically structured, more expressive, and more naturally multilingual than vocabulary-space LSR, while remaining compatible with efficient sparse search infrastructure (Formal et al., 27 Feb 2026).

2. Architecture and representation

SPLARE uses a GPT-style LLM backbone with bidirectional attention and inserts an SAE-based projection in place of the language-model head. In the main formulation, the backbone is Llama-3.1-8B paired with Llama Scope SAEs; additional experiments use Gemma-2-2B with Gemma Scope SAEs (Formal et al., 27 Feb 2026). The SAE is trained on residual stream activations, pre-trained independently, and then frozen during retriever training.

Given a tokenized sequence $t=(t_1,\dots,t_n)$, the LLM is run to a chosen transformer layer $l$, producing hidden states $h_i\in\mathbb{R}^d$. The SAE encoder maps each $h_i$ to a sparse latent activation vector:

$$w_i = f(W_{\text{enc}} h_i + b_{\text{enc}})\in\mathbb{R}^{|\mathcal W|}, \quad i=1,\dots,n.$$

These token-level sparse activations are then aggregated with SPLADE-style max pooling over the latent space:

$$u_j = \max_{i=1,\dots,n}\log\bigl(1+\operatorname{ReLU}(w_{ij})\bigr), \quad j\in\mathcal W.$$

The result $u$ is the SPLARE representation of the sequence: a high-dimensional sparse vector over latent features rather than over vocabulary items (Formal et al., 27 Feb 2026).

The representation space is intentionally wide. Reported SAE widths include 32k and 131k for Llama Scope and 16k, 32k, up to approximately 1M for Gemma Scope (Formal et al., 27 Feb 2026). The main multilingual model, SPLARE-7B, uses Llama-3.1-8B truncated at layer 26 with an SAE width $|\mathcal W|=131k$, yielding an effective size of approximately 7B parameters. A lighter SPLARE-2B truncates the same backbone at layer 6 and provides a lower-cost point on the efficiency–effectiveness curve (Formal et al., 27 Feb 2026).

A key empirical claim is that latent dimensions are often interpretable and language-agnostic. In the multilingual WSDM setting, an English query about “Iranian female athlete refugees” is described as activating features such as “references to athletes and athleticism,” “references to asylum seekers,” and “references to refugees and related humanitarian crises,” while a relevant non-English document can activate the same features (Formal et al., 24 Feb 2026). In this sense, the sparse dimensions function as shared concept units rather than as language-specific lexical indicators.

3. Training objectives and retrieval mechanism

SPLARE training is stage-wise. First, the SAE is trained separately on LLM residual activations using reconstruction plus sparsity regularization, for example

$$\mathcal L_{\text{SAE}} = \|x-\hat x\|^2 + \lambda \|z\|_1.$$

Second, the pre-trained SAE is plugged into the retriever, frozen, and the LLM is fine-tuned with LoRA adapters while bidirectional attention is enabled (Formal et al., 27 Feb 2026).

The retrieval objective in the main SPLARE paper is KL-divergence distillation from a cross-encoder teacher. For a query $q$ and candidate set $l$0, teacher scores define a probability distribution

$l$1

while the student SPLARE scores define

$l$2

The distillation loss is

$l$3

This is combined with FLOPS regularization to maintain sparsity at the sequence level:

$l$4

The reported $l$5 and $l$6 are both fixed at 0.0001 (Formal et al., 27 Feb 2026).

At retrieval time, relevance is computed with a sparse dot product,

$l$7

which is directly analogous to sparse scoring in classical IR and SPLADE. Because both vectors are sparse, indexing and search can be implemented with inverted indexes keyed by latent feature IDs (Formal et al., 27 Feb 2026). The WSDM multilingual paper uses Seismic for this purpose, storing documents in inverted lists over non-zero latent features and traversing the lists associated with active query features at query time (Formal et al., 24 Feb 2026).

Inference commonly applies Top-K truncation: the default reported setting is $l$8 active query features and $l$9 active document features. This truncation is performed only at inference and serves as the main efficiency–effectiveness control (Formal et al., 27 Feb 2026). In the WSDM Cup runs, however, the authors explicitly choose non-pooled representations to maximize effectiveness, with average non-zero counts $h_i\in\mathbb{R}^d$0 and $h_i\in\mathbb{R}^d$1 (Formal et al., 24 Feb 2026).

4. Multilingual retrieval and empirical performance

A major focus of SPLARE is multilingual and out-of-domain retrieval. In the main paper, multilingual SPLARE is trained on a corpus that includes English datasets such as MS MARCO, NQ, and HotPotQA, Chinese retrieval datasets, and multilingual benchmarks including MIRACL and Mr.TyDi; only instances labeled by the BGE-M3 reranker are retained, resulting in approximately 1.3M queries with hard negatives (Formal et al., 27 Feb 2026). The WSDM multilingual system uses a closely related multilingual training mixture: English MS MARCO and NQ, “an extensive collection of Chinese datasets,” and the training splits of MIRACL and Mr. TyDi (Formal et al., 24 Feb 2026).

On MMTEB retrieval, the reported results position SPLARE as the strongest LSR model in the multilingual setting. For English MTEB v2 retrieval, multilingual-trained SPLARE achieves 59.3 nDCG@10 with Top-K $h_i\in\mathbb{R}^d$2, compared with 58.9 for multilingual SPLADE-Llama; the no-pool variant reaches 61.4 (Formal et al., 27 Feb 2026). On the MTEB Multilingual split v2, SPLARE records 62.3 versus 61.7 for SPLADE, and on multilingual-only datasets within that split it reaches 72.6 versus 71.7 (Formal et al., 27 Feb 2026).

On MIRACL, reported as an 18-language benchmark with nDCG@10, SPLARE-7B averages 71.7, compared with 69.9 for SPLADE-Llama, and is described as state of the art or near state of the art for English, Finnish, Hindi, Japanese, Russian, Swahili, German, and Yoruba (Formal et al., 27 Feb 2026). On XTREME-UP, which targets under-represented Indic languages and is evaluated with MRR@10, multilingual SPLARE reaches 58.6 versus 56.2 for multilingual SPLADE-Llama; English-only SPLARE reaches 42.6 versus 30.5 for English-only SPLADE (Formal et al., 27 Feb 2026). The paper characterizes this as evidence that language-agnostic SAE features improve cross-lingual and low-resource generalization.

The WSDM Cup 2026 task provides a more operational testbed. The task takes English queries and retrieves from a collection of approximately 10M documents in Chinese, Persian, and Russian, with nDCG@20 on a development set as the metric (Formal et al., 24 Feb 2026). The system operates directly on the original multilingual collection rather than on machine-translated English versions.

Run	Configuration	nDCG@20
RUN1	SPLARE-7B, no pooling	0.498
RUN2	SPLARE-7B + SPLADE-8B via RRF	0.517
RUN3	RUN1 + Qwen3-Reranker-4B	0.545
RUN4	RUN2 + Qwen3-Reranker-4B	0.548
RUN5	RRF of RUN2 and RUN4	0.559

These runs show several distinct effects (Formal et al., 24 Feb 2026). First, non-pooled SPLARE-7B improves over a Top-K $h_i\in\mathbb{R}^d$3 version from 0.471 to 0.498 nDCG@20, at the cost of denser vectors. Second, SPLARE and SPLADE provide complementary sparse signals: their reciprocal rank fusion with $h_i\in\mathbb{R}^d$4 produces 0.517. Third, reranking with Qwen3-Reranker-4B yields substantial gains, with a reranking depth of 100 documents selected for both RUN3 and RUN4. Finally, combining the first-stage fused run and the reranked run with RRF $h_i\in\mathbb{R}^d$5 yields the best result, 0.559 nDCG@20 (Formal et al., 24 Feb 2026).

In comparisons against dense baselines from NeuCLIRBench, RUN1 outperforms Qwen3-8B-Embed 0.498 to 0.423, a gain of +0.075 nDCG@20, and exceeds the reported “Fusion” baseline at 0.468. With reranking, RUN3 and RUN4 also surpass the dense baseline with Qwen3-Reranker-4B, reported at 0.487 (Formal et al., 24 Feb 2026). The WSDM report interprets this as evidence that the main advantage comes from the first-stage quality of SPLARE-based retrieval rather than solely from reranking.

5. Interpretability, efficiency, and deployment characteristics

Interpretability is presented as one of SPLARE’s distinguishing properties. Because the sparse dimensions correspond to SAE features rather than tokens, they can often be described in semantic terms. The main paper reports Neuronpedia-based analyses in which top SPLARE features for cross-lingual query–document pairs are phrases such as “elements related to historical or cultural contexts,” “mentions of India and its relation to various contexts,” and “references to military casualties and losses,” whereas SPLADE’s top dimensions are token strings such as “Indian,” “Indians,” “casualties,” and “Raj” (Formal et al., 27 Feb 2026). In the WSDM paper, the same interpretability argument is framed as evidence that SPLARE operates in a Bag-of-Concepts space rather than a token space (Formal et al., 24 Feb 2026).

From an indexing perspective, SPLARE remains a sparse retriever. It can be deployed with classical inverted files or with optimized learned sparse retrieval engines such as Seismic (Formal et al., 27 Feb 2026). This is operationally significant because it preserves exact or approximate sparse search workflows rather than requiring dense ANN infrastructure. The reported engine-only retrieval speed on MS MARCO with Top-K $h_i\in\mathbb{R}^d$6 is approximately 5 ms/query, single-threaded, over 8.8M documents (Formal et al., 27 Feb 2026).

A further efficiency claim is robustness to aggressive pruning. With document Top-K $h_i\in\mathbb{R}^d$7, SPLARE is reported to lose only approximately 2% nDCG on MS MARCO, whereas SPLADE loses more than 6% (Formal et al., 27 Feb 2026). The explanation given is that SPLARE’s latent dimensions are used more uniformly and are less dominated by a small set of high-document-frequency features. This suggests a more balanced sparse space and a more favorable effectiveness–latency trade-off under static pruning or truncation.

The WSDM multilingual deployment emphasizes the opposite choice: maximizing effectiveness rather than minimizing latency. The system uses document inputs formed by concatenating title and content, retrieval context lengths up to 1024 tokens, reranking inputs up to 2048 tokens, Seismic with default hyperparameters, and non-pooled representations for both SPLARE and SPLADE (Formal et al., 24 Feb 2026). The paper explicitly notes that efficiency could be improved through Top-K pooling or through two-step SPLADE-style cascades adapted to SPLARE, but those optimizations were not the focus of the shared-task submission.

6. Limitations, misconceptions, and research directions

Several limitations are stated directly. SPLARE depends on high-quality pre-trained SAEs, which are expensive to train and available only for a limited set of major LLMs (Formal et al., 27 Feb 2026). Performance also depends on the chosen SAE suite and transformer layer. The method preserves SAE interpretability by freezing the SAE during retriever training, but this may limit the retrieval optimum because no joint SAE–retriever training is performed (Formal et al., 27 Feb 2026).

Domain dependence is another boundary condition. SPLARE tends to underperform SPLADE on code retrieval: in the reported MTEB Code split, SPLARE obtains 63.0 versus 64.3 for SPLADE (Formal et al., 27 Feb 2026). The proposed explanation is that SAEs trained on natural-language residual streams do not specialize to code semantics. By contrast, SPLARE is reported to be on par or slightly better in medical, law, and ChemTEB evaluations, with 67.7 in medical, 60.8 in law, and 78.1 in ChemTEB (Formal et al., 27 Feb 2026).

The multilingual WSDM study introduces additional practical limitations. It does not optimize latency, uses relatively large non-pooled document vectors with average $h_i\in\mathbb{R}^d$8, does not provide language-by-language performance breakdowns for Chinese, Persian, and Russian, and relies on a substantial multilingual training mixture that includes English, Chinese, and multilingual IR datasets (Formal et al., 24 Feb 2026). A plausible implication is that low-resource deployment without comparable multilingual supervision may be more difficult, although the paper does not quantify that regime directly.

A recurrent misconception in retrieval discourse is that sparse retrieval is inherently English-centric or structurally inferior to dense multilingual embeddings. The WSDM paper argues that this conclusion may hold for vocabulary-based sparse models because their sparse spaces inherit the inefficiencies and alignment problems of multilingual token vocabularies, but that it does not follow for latent sparse retrieval built on multilingual SAEs (Formal et al., 24 Feb 2026). The empirical record presented for MIRACL, XTREME-UP, MTEB Multilingual, and the WSDM Cup supports the narrower claim that redesigning the sparse space around latent concepts materially changes the multilingual behavior of learned sparse retrieval (Formal et al., 27 Feb 2026).

The stated research directions follow directly from these observations. They include domain-specific SAEs for code, scientific, or legal domains; improved SAE architectures such as JumpReLU and Top-K variants; wider latent spaces scaling to millions of features; joint pretraining of LLM and SAE with retrieval objectives; multimodal extensions using multimodal SAEs; and hybrid systems that combine SPLARE with dense retrievers or multi-vector models (Formal et al., 27 Feb 2026). In the multilingual sparse retrieval setting specifically, future work also includes integrating Top-K pooling, static pruning, and two-step cascades into SPLARE’s latent space, and analyzing per-language behavior in typologically diverse collections (Formal et al., 24 Feb 2026).