Dice Question Streamline Icon: https://streamlinehq.com

Identifying which Seed baseline query was used

Ascertain which of the two available Boolean queries in the Seed dataset—the “query” or the “edited-search”—was used as the baseline in prior evaluations, and standardize documentation to enable reproducibility of baseline comparisons.

Information Square Streamline Icon: https://streamlinehq.com

Background

The Seed dataset provides two candidate baseline queries per review topic, leading to ambiguity when reproducing earlier work. Clarifying which query was used as the baseline is necessary to ensure valid comparison and interpretation of reproduced results.

References

However, since the Seed dataset also contains a second Boolean query edited-search, we also show the results for this query (Baseline-edit in the result tables), as it is unclear which query has been used based on the documentation.

A Reproducibility and Generalizability Study of Large Language Models for Query Generation (2411.14914 - Staudinger et al., 22 Nov 2024) in Section 4.1 Reproducibility and Generalizability Study