Determine the causes of ANCE and COCO-DR performance discrepancies on LoTTE

Determine the causes of the performance discrepancies between the reported and reproduced results for ANCE and COCO-DR on the LoTTE benchmark.

Background

The paper compares its reproduced retrieval results with scores reported in the original publications. Although most discrepancies are attributed to implementation, checkpoint, preprocessing, or evaluation differences, the authors identify ANCE and COCO-DR on LoTTE as cases for which they cannot provide a satisfactory explanation. Resolving these discrepancies would improve understanding of reproducibility and implementation dependence in document-retrieval evaluation.

References

In both cases, despite our best efforts, we do not have a satisfactory explanation for the differences in performance.

— A Systematic Multi-Domain Evaluation of Document Retrievers  (2609.29455 - Velev et al., 24 Sep 2026) in Section 4.1, “Comparison to Published Results”