RCEM: Embedder Equipped with Query Rewriting Skill for Robust Conversational Search in Distributional Shift

Abstract: Conversational search has become increasingly important in retrieval-augmented generation (RAG) systems, where users interact with AI assistants through multi-turn conversations containing context-dependent queries. We propose RCEM, a conversational dense retrieval model that distills the query reformulation capability of LLMs into the embedding model, enabling context-aware retrieval without explicit query rewriting during inference. Unlike prior conversational dense retrieval approaches that learn direct conversation-to-document matching, RCEM aligns conversational-query embeddings with rewritten-query embeddings, improving robustness under distributional shift. RCEM does not require conversational query-to-document relevance mappings for training, which are often expensive and difficult to obtain with high quality. Extensive experiments on QReCC, TopiOCQA, and TREC CAsT demonstrate that RCEM consistently outperforms strong conversational retrieval baselines, achieving particularly large gains under distributional shift, including up to 20% improvement in Recall@10. RCEM further extends the base embedding model with conversational query rewriting capability while preserving its original retrieval functionality, allowing both standalone and conversational queries to be encoded by a single model and searched against existing document indexes without rebuilding the retrieval database.

• The paper introduces RCEM, an embedding model that distills LLM-generated conversational query rewriting into a single retrieval pass, improving QReCC performance by 3.0% in Recall@10, 2.7% in MRR, and 2.5% in NDCG@3.
• RCEM combines point and pairwise structure learning to align conversational queries with rewritten-query embeddings while preserving the original document embedding space, enabling searches over existing indexes without re-encoding documents.
• The method generalizes substantially better than contrastive conversational retrieval under distributional shift, achieving up to 20% higher Recall@10 in cross-dataset tests, but remains dependent on the quality of its LLM-generated rewrites.

RCEM addresses a central tension in conversational search for retrieval-augmented generation (RAG): LLM-based conversational query rewriting (CQR) is effective but adds a costly autoregressive generation step at inference time, while conversational dense retrieval (CDR) methods that embed conversations directly typically require conversation-to-document relevance supervision and can distort the embedding space. The paper proposes RCEM, an embedding model trained to map a context-dependent conversational query directly into the embedding of its rewritten standalone query, thereby distilling rewriting capability into the embedder while preserving compatibility with pre-existing document indexes (2606.01697).

Method

RCEM builds on a frozen base embedder $G(x)$ (Qwen3-0.6B by default), adding LoRA adapters and a two-layer MLP head with SELU activation to form $F_\theta$. A special token $[ST]$ is prepended to the conversational input $c_i = [{ST}, q_i; q_{<i}a_{<i}]$ to signal conversational mode. Training uses GPT-4.1 with a simple rewriting prompt to generate standalone rewrites $R_i = \mathrm{LLM}(q_i; q_{<i}, a_{<i})$; no conversation-to-ground-truth-passage mappings are needed.

The training objective, termed Structure Learning, combines two terms:

$$\mathcal{L}(\theta) = \mathcal{L}_{\text{point}}(\theta) + \lambda \mathcal{L}_{\text{pair}}(\theta)$$

where $\mathcal{L}_{\text{point}}$ aligns each input's embedding from $F_\theta$ with its target embedding from $G$ — specifically $F_\theta(c_i) \rightarrow G(R_i)$ for conversational inputs, and identity alignments $F_\theta$0 and $F_\theta$1 for standalone queries and documents. $F_\theta$2 additionally preserves pairwise distance structure between embeddings across the two models. With $F_\theta$3, this dual objective serves as a regularizer: it both teaches the model to resolve conversational context implicitly and anchors it to the original embedding space.

This design yields three properties the paper emphasizes. First, inference requires only one forward pass through the embedder — no separate LLM rewriting stage. Second, training labels are cheap LLM-generated rewrites rather than expensive relevance annotations. Third, because the original embedding space is preserved, both standalone and conversational queries can be encoded by the same model and searched against an existing index without document-side re-encoding.

Experimental results

Experiments use QReCC (~14K conversations, 80K QA pairs, 54M passages), TopiOCQA (~3.9K topic-switching conversations, 25M passages), and TREC CAsT 2019/2020 for zero-shot evaluation, reporting NDCG@3, MRR, and Recall@10 via pytrec_eval. The primary baseline is contextualized query embeddings for conversational search [emnlp2021], trained under identical hardware, base model, adaptation architecture, and epoch budget so that only the objective differs.

In-domain performance: On QReCC, RCEM outperforms the contrastive baseline by 3.0% Recall@10, 2.7% MRR, and 2.5% NDCG@3. Notably, the authors observe that contrastive training is highly sensitive to epoch count — small increases substantially degrade performance — whereas RCEM remains stable across epochs.

Distributional shift: This is where the results are most striking. When models are trained on one dataset and evaluated on another, RCEM consistently outperforms the baseline across all dataset combinations and metrics, with gaps frequently ranging from 10% to 20%. In the strongest case, when trained on QReCC and tested on TREC CAsT 2020, RCEM achieves approximately 20% higher Recall@10. The paper attributes the baseline's degradation to overfitting induced by the contrastive objective, which maps conversational queries directly to passages — a more complex target than rewritten-query alignment. Embedding-space preservation acts as an implicit regularizer against this overfitting.

Embedding-space preservation: Encoding queries with RCEM and retrieving against an index built with the original embedder's document embeddings yields performance comparable to the original embedder itself, whereas the contrastive baseline degrades on such an index due to space distortion. This confirms the practical claim that RCEM can be deployed without rebuilding retrieval databases.

Limitations

The paper identifies one principal bottleneck: RCEM's ceiling is bounded by the quality of the LLM-generated rewrites used as training targets. Although the embedder can occasionally exceed the explicit two-step pipeline — presumably because it learns a stronger latent representation than the surface rewritten text — performance remains dependent on rewriting quality. The authors suggest that incorporating stronger rewriting methods, such as test-time-adaptive reformulation frameworks, could further improve results, but they do not evaluate this. Two additional caveats bear noting: the distribution-shift experiments compare primarily against a single CDR baseline rather than the full set of modern conversational retrievers discussed in related work, and the claimed latency advantages of single-step inference are asserted rather than measured explicitly.

Conclusion

RCEM reframes conversational dense retrieval as embedding-space alignment toward LLM-rewritten queries rather than direct conversation-to-document matching. This substitution removes the need for costly relevance annotations, stabilizes training relative to contrastive objectives, preserves compatibility with existing indexes, and produces substantial robustness gains under distributional shift — up to 20% Recall@10 improvement in cross-dataset evaluation. The approach's dependence on teacher-rewriter quality remains the clearest open question for extending the method.