SCoRD: Semantic-Assisted Continual Retriever-Reranker Distillation for LLM-Based Recommendation

Abstract: Recommendation systems increasingly adopt a two-stage pipeline, where an ID-based retriever retrieves candidates and an LLM-based reranker refines their rankings. To improve retrieval quality, reranker-to-retriever distillation is commonly used to transfer the reranker's knowledge to the retriever. For practical deployment, however, this pipeline must continually adapt to evolving interests and incoming interactions. A naive solution is to repeatedly update the LLM reranker and distill its latest knowledge, but this incurs prohibitive costs. Updating the retriever alone is cheaper, but its limited capacity makes adaptation from sparse data difficult. We propose SCoRD, a continual knowledge distillation framework for LLM-based reranking pipelines under a non-stationary data stream. SCoRD introduces a semantic reasoning assistant that distills the LLM's ability to infer underlying user intents into reusable intent-level guidance. It selectively distills reranker knowledge to the retriever on low-confidence sequences, guides retriever-only updates without repeated LLM inference, and feeds retriever-derived representations and intent-drift signals back to the reranker. Experiments on real-world datasets show that SCoRD enables effective and efficient retriever-reranker co-adaptation.

• The paper introduces SCoRD, a continual co-adaptation framework that distills LLM intent reasoning into a reusable semantic assistant while coordinating asynchronous retriever and reranker updates.
• SCoRD selectively distills hard sequences, uses collaborative pseudo-labels and intent-drift-aware negative sampling during retriever-only updates, and transfers updated retriever knowledge back to the reranker.
• The method outperforms prior baselines across Amazon Books, Yelp, and Movies datasets, including Book retriever NDCG@5 of 0.5327 versus 0.3787 for CCD, while reducing semantic generation time by roughly 10× compared with CoT-Rec.

Motivation and problem setting

Modern LLM-based recommendation systems typically deploy a two-stage retrieve-then-rerank pipeline: a lightweight ID-based retriever (e.g., SASRec) narrows the item space to top-$N$ candidates, and an LLM-based reranker refines their ordering. Reranker-to-retriever knowledge distillation (KD) is widely used to improve retrieval quality, but existing KD methods assume static settings. In production, data arrives as a non-stationary stream of new users, items, and interactions, requiring both modules to adapt continually. The paper identifies a fundamental cost asymmetry: the LLM reranker requires 3.5–9.4× longer training time and 72.4–144.4× longer inference time than the ID-based retriever across datasets, making repeated reranker updates and distillation infeasible.

This creates a dilemma that the authors formalize as an asynchronous update cycle: the retriever updates at short intervals $\tau_S$ (e.g., daily), while the reranker updates at long intervals $\tau_T$ (e.g., weekly). Distilling from a stale reranker gives outdated supervision; updating the retriever alone leaves its limited capacity to learn from sparse incremental data without semantic guidance. The closest prior work, continual collaborative distillation (CCD), decouples teacher-student update cycles but targets ID-based teacher-student pairs with moderate capacity gaps; it does not address the substantial asymmetry between an LLM reranker and an ID-based retriever.

The paper formulates three design questions: how to use scarce LLM supervision effectively, how to reuse LLM semantic reasoning during retriever-only updates without repeated inference, and how retriever-side knowledge can support infrequent reranker updates. SCoRD is proposed as the first framework addressing co-adaptation of an ID-based retriever and an LLM reranker over a data stream.

The semantic reasoning assistant

The core architectural contribution is a Semantic Reasoning Assistant ($\mathcal{A}$) that converts the LLM's intent-inference capability into reusable intent-level guidance. Rather than distilling free-form generation—which would require either small models lacking generation ability or large models undermining efficiency—SCoRD redesigns LLM reasoning knowledge as memory-based selection over a dynamic intent memory $\mathcal{G}$, a discrete set of intent units (e.g., "budget-friendly electronics") that expands along the stream.

Memory construction proceeds in three steps per user sequence: intent inference, where the LLM selects suitable intents from the current memory or proposes new ones; intent verification, where the LLM must identify a held-out item among $N{+}1$ candidates given the sequence and inferred intents, with up to three revision trials before the intent is discarded; and memory update, which unions verified intents into $\mathcal{G}$. This verification step is a notable safeguard against hallucinated intents, though it means users whose intents cannot be verified are deferred to future blocks rather than handled immediately.

The assistant itself is a lightweight query-key-value module: the retriever's sequence representation $\mathbf{h}_u$ serves as the query against learnable intent embeddings, producing multi-label intent relevance scores. Relevant intents' values are aggregated and injected via residual addition and LayerNorm to produce a semantic-guided representation $\tilde{\mathbf{h}}_u$, trained jointly with the retriever through reasoning-distillation (binary cross-entropy against LLM-verified intents) and semantic-guided next-item prediction losses.

Three-stage continual co-adaptation

Stage 1 (reranker-to-retriever KD) concentrates expensive LLM supervision on hard sequences only. Confidence is estimated by sequence-item alignment—the average sigmoid score between $\tilde{\mathbf{h}}_u$ and in-sequence item embeddings—and the bottom $\tau_S$0 (set to $\tau_S$1) of sequences are selected for distillation. For these, the intent memory is updated via the same infer-verify-update procedure, and listwise ranking distillation is applied using the semantic-guided scores. Hyperparameter analysis shows performance improves sharply from $\tau_S$2 to $\tau_S$3 and saturates thereafter, implying that targeting roughly 5–20% of sequences suffices—a strong practical result for cost control.

Stage 2 (retriever-only updates) introduces two techniques. First, collaborative semantic assistant adaptation addresses the absence of verified intent labels on new sequences by constructing pseudo-labels from behaviorally similar users: the collaborative label averages the assistant's own prediction with similarity-weighted predictions of top-$\tau_S$4 neighbors within the mini-batch, reinforcing intents consistently shared across similar sequences. Second, intent-drift-aware negative sampling captures fading interests. Intent histograms aggregated over each block yield a faded-intent score comparing consecutive blocks; because this prior is estimated from sparse observations, it is smoothed with evidence from the retriever's top-$\tau_S$5 ($\tau_S$6) plausible unobserved items via a Dirichlet-Multinomial-inspired conjugate update, and negatives are sampled proportionally to alignment with this posterior. A representation-shift regularization term guards against catastrophic forgetting.

Stage 3 (retriever-to-reranker KD) closes the loop when $\tau_S$7 is due. Following E4SRec, retriever item embeddings are projected into the LLM input space so the reranker consumes up-to-date collaborative representations directly in its prompt. An auxiliary drift-aware cross-entropy loss contrasts the ground-truth next item against intent-drift negatives under a restricted softmax, providing focused supervision against faded-intent items.

Experimental results

Experiments use Amazon Books, Yelp, and Amazon Movies & TV, chronologically split into a base block $\tau_S$8 (60% of interactions) and four incremental blocks $\tau_S$9–$\tau_T$0 (10% each), evaluated with HR and NDCG at cutoffs {5, 10, 20} after blocks $\tau_T$1–$\tau_T$2, averaged over five seeds. All methods share the same backbone configuration: SASRec retriever and E4SRec-style Llama-3.2-3B-Instruct reranker with LoRA. Baselines span reference bounds (Full-Batch, Fine-Tune), CL methods (PISA, Reloop2), continual KD (CCD), and LLM-based KD (LLMD4Rec, CoT-Rec).

SCoRD outperforms all baselines across datasets and metrics, with statistical significance ($\tau_T$3 paired t-test) in most configurations. Representative gains include Book retriever N@5 of 0.5327 versus 0.3787 for the best baseline (CCD) after $\tau_T$4, and Movies reranker N@5 of 0.5658 versus 0.5068 (LLMD4Rec). Notably, SCoRD sometimes exceeds even the Full-Batch upper bound—for example, Book retriever N@5 of 0.5327 versus Full-Batch's 0.1853 after $\tau_T$5—which the authors attribute to mutual knowledge transfer unavailable to independently retrained modules. Gains are consistently larger on fine-grained metrics (N@5, H@5) than H@20; the authors conjecture that continuous semantic assistance mainly enhances fine-grained ranking and leave adaptive control of assistant guidance for broader candidate coverage as open work.

On challenging user groups, SCoRD substantially outperforms CCD and PISA on both new users—where intent-level guidance provides semantic context absent any history—and preference-shift users (top 25% by embedding cosine distance), validating the intent-drift mechanism. On stability-plasticity metrics, SCoRD simultaneously improves retained average (stability), learning average (plasticity), and their harmonic mean over both baselines on all three datasets, indicating no stability-plasticity trade-off is incurred.

Efficiency analysis shows SCoRD reduces semantic generation time by roughly 10× compared to CoT-Rec (e.g., 1.4h vs. 15.4h on Books) while achieving higher accuracy; training time falls between LLMD4Rec (longest, due to full-sequence distillation) and CoT-Rec (shortest); inference time is comparable across methods, confirming negligible deployment overhead from intent prediction.

Ablations confirm each component's contribution: removing the assistant or replacing intent-drift negatives with random negatives causes the largest drops (e.g., Book N@5 falling from 0.5088 to 0.3549 with random negatives), while $\tau_T$6 and $\tau_T$7 contribute smaller gains. Sensitivity analysis shows modest sensitivity overall, though excessive drift-loss weight ($\tau_T$8) degrades reranker accuracy.

Limitations and open questions

Several limitations are acknowledged or evident. The framework depends on the quality of LLM-inferred intents; verification mitigates hallucination but defers unverifiable users, and the memory grows monotonically (140→192 intents on Books), raising unanswered questions about scalability and pruning for very long streams. The faded-intent signal conflates true interest decay with sparse observation, addressed only heuristically via conjugate smoothing. Evaluation uses simulated block splits rather than production traffic, and the reported advantage over the Full-Batch reference—an unusual outcome—warrants further scrutiny regarding whether distillation benefits generalize beyond these backbones. Finally, the observed saturation of gains at coarse cutoffs (H@20) suggests assistant guidance may narrow candidate diversity, an effect the paper explicitly leaves unaddressed.

Conclusion

SCoRD formulates and addresses continual co-adaptation of ID-based retrievers and LLM rerankers, a previously unexplored intersection of knowledge distillation, continual learning, and LLM-based recommendation. Its central idea—distilling LLM intent reasoning into a lightweight, memory-based selection module—enables selective distillation on low-confidence sequences, LLM-free semantic guidance during frequent retriever updates, and retriever-informed reranker adaptation, yielding consistent accuracy gains at competitive training cost and unchanged inference latency.