Create and explore imaginative 3D worlds with our new product Dreamfield (opens in a new tab)

Determine whether suite-based partitioning improves score quality beyond oracle alignment

Determine whether building execution-consistency scores for text-to-SQL candidate queries on the distilled multi-instance suite partition genuinely improves their ability to predict correctness, rather than merely making the scores appear better because the suite partition also supplies the evaluation labels.

Background

The paper compares execution-consistency scores constructed from candidate-query partitions induced by either the benchmark’s shipped single database or its distilled multi-instance test suite. Each score is evaluated using correctness labels from the same or an alternative oracle. The results show that consistency scores look better under the labels of the oracle that built their clusters, suggesting a possible circularity in oracle-internal evaluation.

The paper reports that an independent expert audit produces mixed evidence: the suite-partition score’s advantage persists for Kwai-32B but is not detected for XiYan-32B. Thus, the unresolved issue is whether suite-based partitioning provides a genuine improvement in predictive quality or whether the apparent advantage is primarily an artifact of alignment between the partitioning oracle and the evaluation labels.

References

A substantive explanation remains open: the suite partition may capture the instability of a candidate set better and predict held-out suite correctness more accurately, while weak labels penalise it by counting its newly exposed errors as correct.

— Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL  (2609.25938 - Liu et al., 22 Sep 2026) in Section 5.2, subsection “Consistency scores are aligned with the oracle that built them” (Section 5.2; immediately before Section 5.3)