Determine whether suite-based partitioning improves score quality beyond oracle alignment
Determine whether building execution-consistency scores for text-to-SQL candidate queries on the distilled multi-instance suite partition genuinely improves their ability to predict correctness, rather than merely making the scores appear better because the suite partition also supplies the evaluation labels.
References
A substantive explanation remains open: the suite partition may capture the instability of a candidate set better and predict held-out suite correctness more accurately, while weak labels penalise it by counting its newly exposed errors as correct.
— Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL
(2609.25938 - Liu et al., 22 Sep 2026) in Section 5.2, subsection “Consistency scores are aligned with the oracle that built them” (Section 5.2; immediately before Section 5.3)