Systematic selection of the top‑k plans for pairwise comparison in plan selection

Develop a systematic method for selecting the number k of top-ranked candidate plans to include in pairwise comparisons within the Plan Selection algorithm, balancing evaluation thoroughness with computational efficiency in DocETL’s optimizer.

Background

DocETL’s optimizer uses a two-stage evaluation of candidate plans: it first rates outputs to identify the top k plans, then applies pairwise comparisons among these k to select the best plan. The paper currently sets k=6 as a heuristic.

The authors explicitly state that creating a principled strategy for choosing k remains future work, indicating the need for a formal, data- or task-driven approach to determine this parameter.