EvoSchema: Towards Text-to-SQL Robustness Against Schema Evolution

Abstract: Neural text-to-SQL models, which translate natural language questions (NLQs) into SQL queries given a database schema, have achieved remarkable performance. However, database schemas frequently evolve to meet new requirements. Such schema evolution often leads to performance degradation for models trained on static schemas. Existing work either mainly focuses on simply paraphrasing some syntactic or semantic mappings among NLQ, DB and SQL, or lacks a comprehensive and controllable way to investigate the model robustness issue under the schema evolution, which is insufficient when facing the increasingly complex and rich database schema changes in reality, especially in the LLM era. To address the challenges posed by schema evolution, we present EvoSchema, a comprehensive benchmark designed to assess and enhance the robustness of text-to-SQL systems under real-world schema changes. EvoSchema introduces a novel schema evolution taxonomy, encompassing ten perturbation types across columnlevel and table-level modifications, systematically simulating the dynamic nature of database schemas. Through EvoSchema, we conduct an in-depth evaluation spanning different open source and closed-source LLMs, revealing that table-level perturbations have a significantly greater impact on model performance compared to column-level changes. Furthermore, EvoSchema inspires the development of more resilient text-to-SQL systems, in terms of both model training and database design. The models trained on EvoSchema's diverse schema designs can force the model to distinguish the schema difference for the same questions to avoid learning spurious patterns, which demonstrate remarkable robustness compared to those trained on unperturbed data on average. This benchmark offers valuable insights into model behavior and a path forward for designing systems capable of thriving in dynamic, real-world environments.

• The paper introduces EvoSchema, a BIRD-based benchmark with ten column- and table-level schema changes, human-verified split and merge cases, and Table Match F1 and Column Match F1 metrics.
• The paper finds table-level changes are the main failure mode, with added tables reducing Table Match F1 from about 89–90 to 55–58 for fine-tuned open-source models, while GPT-3.5 and GPT-4 remain more stable.
• The paper shows mixed perturbation training can improve Table Match F1 by up to 33 points on added-table cases and execution accuracy by up to 24 points on split-column cases, though noisy synthetic data causes slight regressions and over-conservative abstention.

Motivation and problem formulation

Text-to-SQL models translate natural language questions (NLQs) into SQL queries conditioned on a database schema, typically serialized as DDL. While performance on static benchmarks such as Spider and BIRD is strong, real database schemas evolve continuously—tables are split or merged for normalization, columns are renamed to reflect new business terminology, and new tables or columns are added. These changes induce distribution shifts (nomenclature, data granularity, table/column relation, and schema complexity shifts) that degrade models trained on static schemas. Prior robustness benchmarks address this incompletely: Dr.Spider and structure-grounded perturbation work focus on syntactic paraphrasing or simple semantic mappings; ADVETA covers only a few perturbation types; MT-TEQL restricts schema changes to those that do not affect the gold SQL; FootballDB restructures a single database to reduce foreign-key references and JOIN paths. EvoSchema formulates the problem directly: given a fixed NLQ, how sensitive is a text-to-SQL model to each type of schema change, and can training on systematically evolved schemas improve robustness?

A taxonomy of ten schema evolution types

The paper defines a taxonomy of ten perturbations split evenly between column-level operations (add, remove, remove-column-in-SQL, rename, split, merge) and table-level operations (add, remove, rename, split, merge). The authors ground this taxonomy in four categories of distribution shift that schema evolution induces: nomenclature shifts (renaming), granularity shifts (adding/removing), and relation/complexity shifts (splitting/merging). Two perturbation types—removing a column or table referenced in the gold SQL—make the question unanswerable, so the expected model behavior is abstention with an explicit insufficiency message.

Dataset construction

EvoSchema is built from BIRD, chosen over Spider for its larger, more realistic databases (7.3 tables and 72.5 columns per database on average) and more complex SQL. For each <NLQ, relevant schema, SQL> triple, the NLQ is held fixed while the schema is perturbed and the gold SQL revised as needed. Generation uses a hybrid strategy: GPT-3.5 synthesizes contextually appropriate names for added/renamed columns and tables; GPT-4 handles the more complex table splitting and merging, including primary-key handling, column-name conflict resolution, and JOIN-path revision of the gold SQL. Heuristics enforce referential integrity (foreign keys updated alongside renamed or removed tables/columns). Five annotators with SQL expertise manually verified all 1.5K split-table and 1.1K merge-table evaluation examples (~175 hours total); the authors report that LLM-generated split/merge data contains roughly 30% low-quality examples, justifying human validation for these two types. Training data, by contrast, is generated fully automatically without verification—an assumption whose consequences surface in the error analysis below.

The resulting benchmark spans roughly 9K training and ~1.5K evaluation instances per perturbation type, with two fine-grained metrics introduced alongside execution accuracy: Table Match F1 and Column Match F1, which measure precision/recall over predicted tables and columns respectively.

Main findings

Three results stand out:

• Table-level perturbations hurt far more than column-level ones. Adding tables drops Table Match F1 from ~89–90 to ~55–58 for fine-tuned open-source models (Code Llama, Mistral, Llama 3, SQLCoder), and splitting tables drops it to ~66–73. Column-level changes except splitting columns cause only marginal degradation. Merging tables degrades Column Match F1 specifically (~63–67), indicating models select tables correctly but struggle when columns from multiple source tables coalesce into one.
• Closed-source models are comparatively stable. GPT-3.5 and GPT-4 show small deltas across perturbation types, which the authors attribute to broader pretraining exposure; notably, perturbation-trained open-source models surpass both GPT models on the perturbed evaluations.
• Perturbation-augmented training yields large gains where they matter most. Training on original plus perturbed data produces up to 33 points Table Match F1 gain on add-tables, up to 14 points on split-tables, and up to 24 points execution accuracy gain on split-columns—with macro-average improvements of roughly 4–6 F1 points across models. The cost is a slight drop on the original data and simple column-level perturbations; error analysis attributes this to spurious or missing WHERE-clause conditions and incorrect column selection, i.e., noise introduced by unverified synthetic training data.

Ablations sharpen these conclusions. Removing table-level perturbations from training causes a catastrophic −30 point drop on add-tables while barely affecting column-level performance; removing column-level perturbations degrades everything modestly, showing both are needed. Simply appending irrelevant tables at training time recovers much of the add-tables gap but hurts other types, confirming that the full perturbation mix—not merely distractor exposure—is what drives robustness. Models also perform better in intra-database train/test splits than cross-database, suggesting partial memorization of specific schema names. Finally, training on out-of-scope (unanswerable) perturbations improves abstention but induces over-conservatism: false-positive abstentions account for nearly the entire performance drop on answerable cases, and true-positive abstention reaches only ~84%.

Baseline comparison

On a downsampled test set (200 examples per type) evaluated by humans, the authors compare GPT-4 zero-shot, GPT-4 three-shot ICL, CHESS (GPT-4o schema selection + candidate generation with a non-perturbed Code Llama generator), and perturbation-fine-tuned Code Llama. In-context learning helps only on split-columns, where name/address/date splits generalize well from demonstrations. CHESS performs well when pruning is accurate but over-prunes on merge-type perturbations, dropping below even the unperturbed Code Llama baseline. Perturbation fine-tuning achieves the best macro-average, with McNemar's tests confirming statistical significance ($p<0.05$) against GPT-4 ICL, unperturbed fine-tuning, and CHESS.

Limitations and open questions

Several caveats bear directly on the reported numbers. Execution accuracy is computed on filtered subsets (e.g., 846 vs. 1506 instances for add-columns) because synthesized database values must be executable by the revised gold SQL, so EX figures rest on a smaller, possibly easier sample. Training data quality is unverified, and the observed regression on simple perturbations suggests that automated synthesis introduces label noise whose mitigation is left open. The out-of-scope experiments reveal a tension between abstention capability and over-conservatism that the paper identifies but does not resolve. Generalization beyond BIRD is demonstrated only in commented-out material (a Spider experiment), so cross-dataset claims should be treated cautiously. Whether the gains persist under compositional or sequential evolutions—multiple perturbations applied to the same schema over time—remains unexamined.

Conclusion

EvoSchema contributes a controlled, ten-type taxonomy of realistic schema evolution, a benchmark built on BIRD with human-verified complex transformations, and a training paradigm that fixes the NLQ while varying the schema to force genuine schema grounding rather than spurious table-joining patterns. The central empirical finding—that table-level restructuring, not column renaming, is the dominant failure mode, and that mixed perturbation training largely closes this gap—provides a concrete recipe for more resilient text-to-SQL systems, at the cost of minor regressions on unchanged schemas and unresolved abstention calibration.