Measure production drift of the LambdaRank selector

Measure drift in the LambdaRank selector and the cached explanation-selection framework through a production A/B test or shadow-traffic evaluation.

Background

The paper proposes retraining the small LambdaRank selector periodically using fresh BERTScore-F1 labels and notes that candidate pools must be refreshed when item metadata changes materially. However, the reported latency, cost, and performance results are based on offline experiments and synthetic load rather than live deployment. The authors therefore leave unresolved whether selector or system performance drifts under real production traffic and changing data distributions, motivating a production A/B or shadow-traffic evaluation.

References

The LambdaRank selector is small (approximately 1.7\,MB) and can be retrained weekly or monthly on fresh F1 labels at negligible compute cost; we have not measured drift in a production A/B test, which we identify as future work.

— Pairwise Ranking Outperforms Single-Action RL for Offline Explanation Selection: A Practical Lesson  (2608.18531 - Chowdhury et al., 19 Aug 2026) in Section 6, Discussion, third deployment-relevant observation