Dice Question Streamline Icon: https://streamlinehq.com

Reproducing CLEF TAR baseline queries

Determine the exact original Boolean queries and the methodology used to construct the baselines for the CLEF TAR diagnostic test accuracy collections, and provide PubMed API–compatible formulations of these queries to enable reproduction of baseline results.

Information Square Streamline Icon: https://streamlinehq.com

Background

The paper assesses reproducibility using CLEF TAR datasets but notes that queries provided in CLEF TAR are not in a PubMed-compatible format. Additionally, the prior work being reproduced did not describe how its original queries were created, preventing the authors from reconstructing baselines. Establishing the baseline queries and their construction process is necessary for reliable replication and comparison.

References

For the CLEF TAR dataset, we were not able to reproduce the baselines, as this dataset does not provide queries in the PubMed API compatible format, and the paper did not state how the original queries were created.

— A Reproducibility and Generalizability Study of Large Language Models for Query Generation (2411.14914 - Staudinger et al., 22 Nov 2024) in Section 3.2 Baselines