MixLoRA-DSI: Continual Generative Retrieval

• The paper introduces MixLoRA-DSI as a rehearsal-free continual learning framework for generative retrieval that incrementally integrates new documents without full retraining.
• It combines an expandable mixture of LoRA experts with a top-k cosine router and Residual Quantization-based docids to achieve stability and lower training costs.
• Experiments on NQ320k and MS MARCO demonstrate a favorable stability-plasticity tradeoff with minimal parameter overhead and enhanced efficiency in dynamic corpora.

Searching arXiv for the main paper and closely related continual generative retrieval work to ground the article and verify citations. MixLoRA-DSI is a rehearsal-free continual-learning framework for generative retrieval, or Differentiable Search Index (DSI), over dynamic corpora. It is introduced for the setting in which a seq2seq Transformer, here T5-based, must continually incorporate newly added documents without full retraining and without replaying prior raw documents. The framework combines an expandable mixture of Low-Rank Adaptation experts, a layer-wise out-of-distribution-driven expansion policy, and Residual Quantization-based docid strategies intended to stabilize continual indexing. In the reported experiments on NQ320k and MS MARCO Passage, the method is presented as achieving a favorable stability-plasticity tradeoff with minimal parameter overhead and substantially lower training costs than full-model update baselines (Huynh et al., 14 Jul 2025).

1. Problem formulation and retrieval setting

The method is defined in the context of generative retrieval, also called Differentiable Search Index, where a seq2seq Transformer is trained to map each document $d_i$ to a discrete document identifier $id_i$, and then generates docids autoregressively from queries at retrieval time. Formally, a DSI model $f_\theta$ is trained during indexing so that each document $d_i \in D$ is associated with a unique identifier $id_i \in I_D$. At query time, constrained beam search generates top-$k$ docids, which are then mapped back to documents (Huynh et al., 14 Jul 2025).

The continual-indexing regime assumes an evolving corpus

$\{D_0, D_1, \dots, D_T\},$

where $D_0$ is an initial large corpus with annotated query-docid pairs and $D_1,\dots,D_T$ are newly added documents that are generally without annotations. If $\theta_{t-1}$ denotes the model after indexing through $id_i$0, the system must update to $id_i$1 to incorporate $id_i$2 and then be evaluated on all corpora indexed so far. The paper characterizes this as difficult because the model must retain old retrieval ability, learn new documents efficiently, and operate in a task-agnostic setting in which corpus boundaries are blurry and newly added documents may be semantically close to earlier ones (Huynh et al., 14 Jul 2025).

A central motivation is the cost structure of full retraining. The backbone used is T5-base with approximately $id_i$3M parameters. In this setting, indexing is itself model training rather than mere inference, since corpus information is encoded into model weights. Repeated full-model fine-tuning across $id_i$4 is therefore computationally costly, and rehearsal-based continual learning is additionally problematic because it requires access to old documents, incurring storage cost and privacy concerns. MixLoRA-DSI is proposed as a rehearsal-free, task-agnostic, parameter-efficient alternative based on incremental expansion rather than repeated end-to-end retraining (Huynh et al., 14 Jul 2025).

2. Architectural design: Mixture-of-LoRA experts in DSI

The architectural core of MixLoRA-DSI is the insertion of a mixture of LoRA experts into selected decoder feed-forward network blocks of T5-base. The paper places MixLoRA in the first five decoder FFN layers, motivated by the fact that the decoder is responsible for autoregressive docid generation (Huynh et al., 14 Jul 2025).

The standard FFN is written as

$id_i$5

MixLoRA augments the frozen pretrained FFN with LoRA experts. The LoRA configuration reported is rank $id_i$6, dropout $id_i$7, and scaling factor $id_i$8. Instead of using a single adapter, each adapted layer contains a mixture of LoRA experts, with a router selecting top-$id_i$9 experts per token; the default is top-2 routing (Huynh et al., 14 Jul 2025).

For each FFN layer, LoRA experts are introduced separately for the input and output projections,

$f_\theta$0

The adapted hidden computation is

$f_\theta$1

and the full MixLoRA output is

$f_\theta$2

An important implementation detail is that the same gate values are shared across both MLP layers, so no extra router is needed for the output projection (Huynh et al., 14 Jul 2025).

This design isolates adaptation into small trainable modules while freezing most backbone weights. A plausible implication is that the method inherits the usual PEFT advantages of bounded trainable state and modular capacity growth, but in this work those properties are explicitly tied to continual generative retrieval rather than generic downstream fine-tuning.

3. Routing, recency bias, and the cosine-classifier reformulation

In standard mixture-of-experts routing, a token $f_\theta$3 is routed through logits produced by a router matrix $f_\theta$4,

$f_\theta$5

with softmax gate values

$f_\theta$6

Top-$f_\theta$7 experts are then selected and combined (Huynh et al., 14 Jul 2025).

The paper argues that this conventional router exhibits recency bias in continual expansion. Newly added experts tend to dominate, older experts are underused, and the router’s unnormalized logits for new experts can overpower older ones. In response, MixLoRA-DSI replaces the usual router with a top-$f_\theta$8 cosine classifier trained using an auxiliary loss (Huynh et al., 14 Jul 2025): $f_\theta$9

Here, $d_i \in D$0 are hidden representations of docids from the previous decoder block, $d_i \in D$1 is the newly added router weight vector, and all router vectors are $d_i \in D$2-normalized. The first term aligns the new router weight with token or docid representations, while the second keeps the new router distinct from prior router weights. The stated effect is improved expert-token alignment, more balanced routing, and preservation of expert diversity (Huynh et al., 14 Jul 2025).

The router reformulation is significant because the framework’s expansion mechanism depends on the router latent space. If routing collapses onto recently added experts, then the expansion policy would tend to confound genuine novelty with routing pathology. The paper’s router analysis reports that the improved router reduces recency bias and distributes tokens more evenly across experts (Huynh et al., 14 Jul 2025).

4. OOD-driven layer-wise expansion and sublinear parameter growth

The key novelty of MixLoRA-DSI is the treatment of expert expansion as an out-of-distribution detection problem in router latent space. The paper motivates this by linking expert knowledge storage to router-based expert selection: if incoming documents are sufficiently novel in latent space, the currently available experts may be inadequate. Instead of adding a new expert for every new corpus, the method expands only when there is strong evidence of novelty (Huynh et al., 14 Jul 2025).

Novelty is measured using an energy score,

$d_i \in D$3

where $d_i \in D$4 is a token representation, $d_i \in D$5 is the router weight vector for expert $d_i \in D$6, and $d_i \in D$7 is the temperature, set to $d_i \in D$8 in practice. Higher energy indicates more OOD-like input (Huynh et al., 14 Jul 2025).

Before training on a new corpus $d_i \in D$9, the model scans pseudo-queries or document representations and computes energy scores layer by layer. For each MixLoRA layer, it maintains an EMA threshold $id_i \in I_D$0 from prior indexed corpora. A token $id_i \in I_D$1 is marked OOD when its energy exceeds the threshold; a query is OOD if it contains at least one OOD token; and if the number of OOD queries at a layer exceeds a predefined threshold $id_i \in I_D$2, that layer is expanded. Expansion adds one new LoRA expert and a corresponding new router weight vector $id_i \in I_D$3, yielding

$id_i \in I_D$4

Expansion is therefore layer-wise rather than globally synchronized: some layers may expand while others remain fixed (Huynh et al., 14 Jul 2025).

The paper uses the term sublinear parameter growth to distinguish this selective policy from naive expert allocation. Adding one expert per corpus per layer would induce linear growth in trainable parameters with the number of corpora or updates. MixLoRA-DSI instead grows only when OOD evidence crosses thresholds, so parameter growth can be slower than the number of corpus updates. The empirical illustration on NQ320k reports $id_i \in I_D$5M trainable parameters for the dynamic version, $id_i \in I_D$6M for the OOD-disabled ablation, and $id_i \in I_D$7M for the non-expanded variant. The paper explicitly states that on NQ320k, the dynamic version uses only $id_i \in I_D$8 of the trainable parameters of the OOD-disabled variant while maintaining over $id_i \in I_D$9 of its AP and FWT (Huynh et al., 14 Jul 2025).

This suggests that the method’s notion of scalability is not merely PEFT-based compression, but adaptive capacity management conditioned on observed corpus novelty. The distinction matters because continual indexing over dynamic corpora involves both stability and capacity allocation.

5. Residual-quantized docids, masking, and continual-learning constraints

A major practical component of MixLoRA-DSI is the use of Residual Quantization-based docids rather than atomic docids. Each document embedding $k$0 is approximated by a sum of codebook centroids,

$k$1

and the RQ codebooks are concatenated to the DSI vocabulary,

$k$2

The experimental setting uses $k$3 codebooks and $k$4 centroids per codebook (Huynh et al., 14 Jul 2025).

Because each token in the docid sequence belongs to a specific codebook segment, the model applies a mask

$k$5

and masked logits

$k$6

The stated purpose is to prevent unnecessary competition among unrelated tokens and make the structured docid space easier to learn (Huynh et al., 14 Jul 2025).

For continual indexing, the paper also introduces a slow-learner strategy and KL regularization. Gradients on the RQ token embeddings are scaled down while the original $k$7 remains frozen. The KL term aligns current and previous model predictions: $k$8 These mechanisms are part of how rehearsal is avoided: the framework does not store and replay previous raw documents, but instead relies on parameter isolation through LoRA experts, router-based expansion, KL regularization for RQ embeddings using the previous model’s outputs, and extensive freezing of parameters (Huynh et al., 14 Jul 2025).

The full training objective is

$k$9

During pretraining on $\{D_0, D_1, \dots, D_T\},$0, $\{D_0, D_1, \dots, D_T\},$1 is not used. During continual indexing, if expansion occurs, only new router weights, new LoRA experts, and extended RQ vocabulary weights are optimized; otherwise, only the extended RQ vocabulary weights are updated. All other parameters are frozen. The paper characterizes this as a freeze-and-expand PEFT setup (Huynh et al., 14 Jul 2025).

6. Experimental protocol, empirical results, and ablations

The reported evaluation uses two datasets: NQ320k, with $\{D_0, D_1, \dots, D_T\},$2k query-document pairs and $\{D_0, D_1, \dots, D_T\},$3k documents, and MS MARCO Passage, with $\{D_0, D_1, \dots, D_T\},$4M passages and $\{D_0, D_1, \dots, D_T\},$5k queries, evaluated on the $\{D_0, D_1, \dots, D_T\},$6k-query development set. Each dataset is split into an initial $\{D_0, D_1, \dots, D_T\},$7 containing $\{D_0, D_1, \dots, D_T\},$8 of the documents and four incremental $\{D_0, D_1, \dots, D_T\},$9 splits $D_0$0–$D_0$1. Test queries are partitioned accordingly (Huynh et al., 14 Jul 2025).

The evaluation metrics are R@10 and M@10, together with continual-learning metrics: $D_0$2

$D_0$3

$D_0$4

Baselines include BM25, DPR, BASE, DSI++, CLEVER with $D_0$5 and $D_0$6, PromptDSI, CorpusBrain++, Naive Expansion, and MixLoRA-DSI variants $D_0$7, $D_0$8, and $D_0$9 (Huynh et al., 14 Jul 2025).

The final reported results after indexing $D_1,\dots,D_T$0 are summarized below.

Model	Params	Final metrics after $D_1,\dots,D_T$1
MixLoRA-DSI on NQ320k	0.9M	AP$D_1,\dots,D_T$2: 68.1 / 55.2; BWT$D_1,\dots,D_T$3: 13.0 / 18.1; FWT$D_1,\dots,D_T$4: 78.0 / 70.0
MixLoRA-DSI$D_1,\dots,D_T$5 on NQ320k	1.6M	AP$D_1,\dots,D_T$6: 68.8 / 56.2; BWT$D_1,\dots,D_T$7: 12.4 / 18.3; FWT$D_1,\dots,D_T$8: 78.3 / 71.6
MixLoRA-DSI$D_1,\dots,D_T$9 on NQ320k	0.6M	AP$\theta_{t-1}$0: 66.5 / 53.0; BWT$\theta_{t-1}$1: 13.8 / 17.2; FWT$\theta_{t-1}$2: 76.7 / 66.3
CLEVER(1024) on NQ320k	235.4M	AP$\theta_{t-1}$3: 66.1 / 47.4; BWT$\theta_{t-1}$4: 28.2 / 42.5; FWT$\theta_{t-1}$5: 90.2 / 84.4
MixLoRA-DSI on MS MARCO	0.6M	AP$\theta_{t-1}$6: 36.4 / 20.6; BWT$\theta_{t-1}$7: 16.5 / 20.4; FWT$\theta_{t-1}$8: 53.5 / 41.7
MixLoRA-DSI$\theta_{t-1}$9 on MS MARCO	1.6M	AP$id_i$00: 35.0 / 20.0; BWT$id_i$01: 16.6 / 19.8; FWT$id_i$02: 51.9 / 40.3
CLEVER(1024) on MS MARCO	235.4M	AP$id_i$03: 38.3 / 23.2; BWT$id_i$04: 33.1 / 36.0; FWT$id_i$05: 72.4 / 60.6

On NQ320k, the paper interprets these figures as showing that MixLoRA-DSI achieves a better stability-plasticity tradeoff than full-model continual-learning baselines, uses about $id_i$06M trainable parameters, and substantially reduces forgetting relative to CLEVER. On MS MARCO Passage, the paper states that MixLoRA-DSI remains competitive and much more parameter-efficient, while noting that CLEVER can still achieve higher AP and FWT in some settings at a major cost in memory and forgetting behavior (Huynh et al., 14 Jul 2025).

The resource measurements specify training GPU memory at batch size $id_i$07: DSI++ $id_i$08 GiB, CLEVER(512) $id_i$09 GiB, CLEVER(1024) $id_i$10 GiB, PromptDSI $id_i$11 GiB, CorpusBrain++ $id_i$12 GiB, and MixLoRA-DSI $id_i$13 GiB. Inference memory is reported as $id_i$14 GiB for MixLoRA-DSI, $id_i$15 GiB for CorpusBrain++, and $id_i$16 GiB for CLEVER. Storage is reported as $id_i$17 GiB for MixLoRA-DSI, $id_i$18 GiB for CLEVER, and $id_i$19 GiB for PromptDSI, CorpusBrain++, and DSI++. The paper’s claim of minimal parameter overhead and substantially lower training costs is therefore tied to concrete trainable-parameter counts in the sub-million to low-million range, modest training memory, lower storage than EWC-based methods, competitive or lower inference memory than some PEFT alternatives, and the absence of replay storage (Huynh et al., 14 Jul 2025).

The ablation results on NQ320k identify distinct contributions from the components. The RQ-based docid mask yields a large AP gain but little forgetting reduction. The RQ-based docid continual-learning strategies improve retrieval and significantly reduce forgetting. The improved router further boosts performance. Pretraining on $id_i$20 produces the largest improvement in retrieval. OOD-driven expansion preserves performance while preventing unnecessary growth. The paper summarizes the component effects as: mask, $id_i$21 R@10 AP improvement; CL strategies, $id_i$22 R@10 AP and $id_i$23 R@10 BWT; pretraining, $id_i$24 R@10 AP; and OOD expansion, limited growth without hurting performance (Huynh et al., 14 Jul 2025).

7. Interpretation, limitations, and practical significance

MixLoRA-DSI is positioned as a rehearsal-free continual-learning method for generative retrieval that replaces full-model updates with a mixture of LoRA experts, uses a top-$id_i$25 cosine router and auxiliary diversity-alignment loss, detects novelty via layer-wise energy-based OOD scoring, expands only when OOD evidence exceeds a threshold, and integrates RQ-based docid masking with KL regularization. The central empirical claim is a strong stability-plasticity tradeoff with far lower parameter and memory overhead than full retraining and reduced forgetting relative to prior continual-learning generative retrieval methods (Huynh et al., 14 Jul 2025).

The paper also states several limitations. It does not integrate recent rank-optimization techniques, partly because state-of-the-art RQ-based docid methods require multi-step training that could worsen forgetting. Experiments are limited to T5-base because of compute constraints. No extensive hyperparameter tuning is reported, although the results are described as stable. A further limitation is a freeze-and-expand tradeoff: CorpusBrain++ indicates that continually fine-tuning fixed adapters can also be robust, which leaves open whether dynamic expansion is always preferable under strict parameter budgets. The paper additionally notes that, on large-scale retrieval such as MS MARCO, generative retrieval methods still lag BM25 and DPR in some settings, and that zero-shot retrieval is not applicable because GR requires indexing or model updates (Huynh et al., 14 Jul 2025).

In practical terms, the method suggests a workflow for dynamic corpus indexing in which one starts from a pretrained GR model, represents document IDs with scalable RQ codebooks, uses PEFT to minimize update cost, and expands capacity only when the incoming corpus is genuinely novel in latent space. This suggests applicability in environments where old data cannot be stored or replayed, updates are frequent, and compute budget is constrained. Within that scope, MixLoRA-DSI can be understood as a selective-capacity continual indexing scheme rather than a general replacement for full retraining across all retrieval regimes (Huynh et al., 14 Jul 2025).