Establish whether PCDR outperforms PCQR

Determine, with a statistically conclusive evaluation over substantially more personalized relevance judgments, whether Personalized Conversational Dense Retrieval (PCDR) outperforms Personalized Conversational Query Reformulation (PCQR), or whether PCQR outperforms PCDR.

Background

The appendix evaluates PCDR against PCQR systems using rewriters ranging from 0.6B-parameter Qwen3 models to GPT-4o and a human oracle rewrite. On 44 held-out turns, only the same-size 0.6B rewriter is significantly worse than PCDR after correction; the comparisons with larger rewriters and the human oracle are not statistically significant. Consequently, the available data do not establish which approach is superior, and the authors identify the need for many more personalized relevance judgments and a larger cross-year evaluation.

References

In other words, on this data we can neither conclude that PCDR beats PCQR nor the reverse: PCDR is statistically indistinguishable from strong rewriters and from the human oracle, while a rewriter of the same size as the PCDR encoder is clearly worse.

— RALI@iKAT 2026: Retrieval-aware User Profile Denoising via Personalized Conversational Dense Retrieval  (2610.17673 - Hui et al., 13 Oct 2026) in Appendix, Section “Is PCDR Better than PCQR? An Inconclusive Study,” subsection “Result”