Determine whether template coverage causes the accuracy gain

Determine whether the complementary template coverage of self-play-generated SQL pairs, relative to human-written BIRD pairs, causes the observed Text-to-SQL accuracy gain.

Background

The paper compares self-play-generated SQL corpora with human-written BIRD gold pairs using the masked SQL templates employed for challenger deduplication. The two sources occupy almost disjoint and complementary template regions, which rules out simple reconstruction of the human corpus. However, the reported coverage analysis is descriptive and does not establish a causal relationship between template coverage and solver accuracy.

References

This rules out simple reconstruction, but does not establish that coverage causes the accuracy gain.

— SQL-Zero: Self-Evolving Text-to-SQL  (2609.04697 - Pedrozo et al., 4 Sep 2026) in Section 4, paragraph “The gain is not reconstruction of the gold corpus”

These are single-run observations and do not establish which of diversity, entropy, or the training budget causes the plateau.

— SQL-Zero: Self-Evolving Text-to-SQL  (2609.04697 - Pedrozo et al., 4 Sep 2026) in Section 5, Section “Limitations and Future Work”

It nonetheless leaves all three margins unresolved: self-play wins more discordant pairs in every iteration (137 versus 119, 152 versus 137, 155 versus 129), and no test reaches significance ($p=0.29,0.41,0.14$; Holm-adjusted \citep{holm} $p=0.58,0.58,0.41$ across the three iterations).

— SQL-Zero: Self-Evolving Text-to-SQL  (2609.04697 - Pedrozo et al., 4 Sep 2026) in Section 4, paragraph “Zero labels beat human labels on the same databases”