- The paper presents MSPA-CQR, a framework that aligns rewriting fidelity, retrieval efficacy, and response adequacy to enhance conversational query rewriting without requiring human annotations.
- It employs a two-stage process using large language models for candidate generation followed by prefix-guided multi-faceted direct preference optimization, achieving significant improvements in MRR and NDCG on benchmark datasets.
- Ablation and transfer studies validate that integrating multi-faceted self-consistency not only boosts retrieval performance but also ensures robust generalization across diverse retrieval models and settings.
Multi-Faceted Self-Consistent Preference Alignment for Conversational Query Rewriting
Introduction and Motivation
Conversational Query Rewriting (CQR) is integral to effective conversational search, serving to transform ambiguous, context-dependent user queries into self-contained forms amenable to reliable retrieval. While earlier CQR approaches have primarily concentrated on isolated query rewriting—often leveraging human-annotated rewrites as ground truth—they generally neglect feedback from subsequent retrieval and answer-generation steps. As a result, these approaches frequently prioritize human readability in rewrites over utility for retrieval performance. Only a subset of recent work has attempted to integrate retrieval feedback, but these methods typically demand annotated gold passages, limiting scalability and practical applicability.
This paper introduces a new paradigm: Multi-Faceted Self-Consistent Preference Aligned Conversational Query Rewriting (MSPA-CQR), which fundamentally departs from prior work by aligning query rewriting across three orthogonal and empirically distinct axes—rewriting fidelity, retrieval efficacy, and response adequacy—without explicit reliance on labor-intensive human annotations.
Methodology
Multi-Faceted Preference Data Construction
MSPA-CQR is designed as a two-stage process. The first stage samples multiple candidate rewrites per context-query pair using LLMs under few-shot prompting. For each sampled rewrite, the system:
- Retrieves passages via a standard IR system (BM25 for efficiency in the preference construction phase)
- Generates responses to the rewritten query using the LLM
- Scores each rewrite along three axes:
- Rewriting self-consistency: Semantic congruence among sampled rewrites measured by NLI-based similarity, penalized for brevity to enforce completeness.
- Retrieval self-consistency: The intersection of top retrieved passages among candidate rewrites.
- Response self-consistency: Semantic similarity among responses generated to candidate rewrites.
Top-scoring (chosen) and bottom-scoring (rejected) candidates for each preference axis are used to assemble triplets for preference alignment in stage two.
Prefix-Guided Multi-Faceted Preference Optimization
The second stage introduces a prefix-based extension to Direct Preference Optimization (DPO): Multi-Faceted Direct Preference Optimization (MDPO). Each training instance is tagged with a explicit preference indicator—[REWRITE], [RETRIEVAL], or [RESPONSE]—to steer the model toward optimizing query rewrites for the specified axis. The MDPO objective encourages the model to assign higher likelihood to chosen samples versus rejected ones, relative to a reference model, with the prefix indicating which reward function is operative.
Empirical analysis using Kendall’s Tau confirms low rank correlation between the three scoring axes, indicating that substantial, complementary divergences exist and thus warrant multi-faceted alignment.
During inference, for each user query, the system generates three rewrites corresponding to the respective preferences. Combining the results of the three queries (concatenation/fusion) produces the final candidate set for passage retrieval.
Experimental Results
Benchmarks and Evaluation
MSPA-CQR was rigorously evaluated on two canonical conversational search datasets: QReCC and TopiOCQA. Retrieval was assessed using both sparse (BM25) and dense (ANCE) systems, with standard metrics (MRR, NDCG@3, R@10, R@100).
MSPA-CQR outperforms recent SOTA baselines (AdaCQR, RETPO, CHIQ, etc.) across nearly all metrics and scenarios, including both in-domain and out-of-distribution settings. Significant improvements were observed in MRR and NDCG, demonstrating that MSPA-CQR consistently promotes more relevant passages to higher ranks—an explicit consequence of preference-guided alignment that drives rewrites to include information salient to the retriever and final answer.
Notable findings include:
- MSPA-CQR achieves statistically significant gains (p < 0.03) over retrieval-feedback-only baselines, confirming the utility of multi-faceted preference signals. For instance, on QReCC and TopiOCQA, MSPA-CQR consistently delivers higher recall and ranking metrics in both sparse and dense retrieval regimes.
- Zero-shot transfer experiments on CAsT-19/CAsT-20/CAsT-21 demonstrate robust generalization, surpassing baselines trained with similar or larger LLMs.
Analytical Studies
Ablation studies reveal:
- Removing any single preference (rewriting, retrieval, response) or their corresponding prefixes impairs retrieval efficacy, evidencing the complementarity of multi-axis alignment.
- NLI-based semantic scoring for consistency trumps simple majority voting, validating the importance of capturing deep semantic congruence over superficial agreement.
- Query expansion and high-quality LLMs (e.g., Qwen2.5-32B-Instruct, GPT-4o) for sampling candidate rewrites directly translate to higher retrieval performance and more diverse candidate sets.
Qualitative analysis of rewritten queries under different preferences confirms that the rewrite preference yields the longest, most specific queries; retrieval preference enforces conciseness; response preference encourages inclusion of answer-relevant details. The intersection of passages retrieved by the different preference-guided queries remains below 70%, affirming their substantive diversity and confirming that their fusion enhances coverage.
Case studies, including failure analyses, illustrate that certain relevant passages are accessible only via multi-preference fusion, while noisy candidates from individual axes can sometimes impede retrieval—a challenge for future reinforcement and aggregation mechanisms.
Theoretical and Practical Implications
The approach establishes a scalable, annotation-light pathway to robust CQR by leveraging LLM-based sampling and fully automated self-consistency scoring, sidestepping expensive human rewrites or relevance judgments. The distinct alignment axes, empirically validated, enable explicit modeling of the trade-offs between self-containedness, retrievability, and answer support in rewrites. Crucially, the prefix-guided architecture allows end-users or downstream agents to dynamically specify which aspect to prioritize depending on context or application need.
This work also demonstrates that direct preference optimization can be generalized to multiple reward axes via a simple, interpretable prefix mechanism, aligning with trends in controllable and modular LLM fine-tuning.
Limitations and Future Directions
MSPA-CQR opens several avenues for further refinement:
- Self-consistency scoring could be improved by integrating more sophisticated, perhaps task-specific, semantic evaluation techniques or leveraging retrieval-based answer verification.
- While prefix-based MDPO allows independent optimization, structured mutual promotion or constraint-based modeling between axes could further boost synergy.
- The approach currently utilizes DPO for preference optimization; integrating or benchmarking against emerging alignment techniques, especially those exploring preference aggregation or multi-objective RLHF, would be valuable.
From a systems perspective, reducing inference latency (e.g., by merging preference-guided queries into a single “mixed” query without information loss) remains an open efficiency vs. effectiveness optimization.
Conclusion
The MSPA-CQR framework advances conversational search by aligning query rewriting across the orthogonal dimensions of rewriting completeness, retrieval utility, and response adequacy through self-consistent, prefix-guided preference modeling and optimization. The empirical evidence substantiates the indispensable role of multi-faceted feedback in bridging the gap between human-readable rewrites and retrieval/answer performance, with tangible improvements over prior paradigms. This trajectory represents a significant step toward robust, adaptive, annotation-efficient conversational search systems capable of flexibly attending to diverse utility criteria (2604.06771).