Text-to-SQL Benchmark Datasets Overview

• Text-to-SQL benchmark datasets are structured corpora that pair natural language queries with executable SQL to evaluate semantic parsing across diverse schemas.
• They facilitate rigorous assessments by integrating manual annotations, synthetic examples, and query log data to address challenges like ambiguity and multi-turn interactions.
• These benchmarks drive improvements in model architectures, evaluation metrics, and multilingual as well as domain-specific performance in practical NL2SQL applications.

Text-to-SQL benchmark datasets are structured corpora that pair natural language questions with executable SQL queries over relational databases, facilitating rigorous evaluation, analysis, and comparison of algorithms for natural language to SQL (NL2SQL) semantic parsing. Such datasets have become central to the advancement of neural semantic parsing architectures, cross-lingual and cross-domain generalization, robustness assessment, and practical deployment of text-to-SQL systems. Recent years have seen a proliferation of benchmarks targeting complexities such as variable schema diversity, entity linking, ambiguity, session-level interactions, multilinguality, domain-specific reasoning, and real-world deployment scenarios.

1. Foundational Datasets: Scope, Construction, and Statistical Properties

Benchmarks in text-to-SQL research span a spectrum from synthetic single-table corpora to complex, multi-domain and domain-specialized resources:

• Classic Single-Table Benchmarks: Datasets like WikiSQL focus on mapping natural language questions to SQL over simple, single-table schemas; they feature abundant annotation but limited compositional diversity.
• Multi-Table and Cross-Domain Datasets: SPIDER and its derivatives (e.g., MultiSpider) drive evaluation for generalization across diverse schemas, SQL patterns, and domains, often containing thousands of databases and tens of thousands of queries.
• Session-Level and Conversational Corpora: SeSQL (Huang et al., 2022) and CHASE structure data into multi-turn sessions, enabling research into context-dependent parsing and in-session error accumulation.
• Domain-Specific and Real-World Corpora: BEAVER (Chen et al., 3 Sep 2024) (enterprise), BiomedSQL (Koretsky et al., 23 May 2025) (biomedical), EHRSQL (Lee et al., 2023) (healthcare), and BULL/FinSQL (Zhang et al., 19 Jan 2024) (finance) contain genuine business or scientific queries, often with large schemas (hundreds of tables/columns), long compositional depth, and high annotation quality via expert review or log mining.
• Low-Resource and Multilingual Datasets: MultiSpider (Dou et al., 2022) enables evaluation and training of systems in seven languages; Dialect2SQL (Chafik et al., 20 Jan 2025) targets Moroccan Darija as the first large-scale Arabic-dialect text-to-SQL corpus.

Typical dataset statistics are summarized below for representative resources:

Dataset	Domains / DBs	Examples	Tables/DB (avg)	Schema Language(s)
WikiSQL	1,000+	80,654	1	English
SPIDER	200+	10,181	~5	English
MultiSpider	166 (Spider)	9,691*7 langs	~5	EN/DE/FR/ES/JA/ZH/VI
SeSQL	201	27,012	N/A	Chinese
BEAVER	2 (DW, NW)	93	99 / 321	English (Enterprise)
Dialect2SQL	69	9,428	2–60 (avg 8)	Moroccan Darija, English
BiomedSQL	1 (BigQuery)	68,000	multi	English (biomedical)

*Table/column counts and statistics vary by corpus; refer to corpus documentation for full schema details.

2. Annotation Protocols, Realism, and Data Augmentation

Annotation strategies vary significantly:

• Manual Annotation and Review: High-quality datasets such as BEAVER (Chen et al., 3 Sep 2024), EHRSQL (Lee et al., 2023), and SeSQL (Huang et al., 2022) employ expert-driven annotation and iterative review, increasingly using consensus mechanisms to mitigate bias and error propagation.
• Derived From Query Logs: BEAVER and HI-SQL (Parab et al., 11 Jun 2025) extract question-SQL pairs from enterprise log data, emphasizing authentic distributional properties not captured by synthetic or MTurk-generated examples.
• Synthetic and Augmented Examples: SynSQL-2.5M, SynSQL-Think-916K, and similar synthetic corpora introduce chain-of-thought and merge-revision annotations to promote logical decomposition and revision, particularly for small LLM (SLM) tuning (Sheng et al., 30 Jul 2025).
• Schema and Expression Diversity: TableQA (Sun et al., 2020) injects “expression diversity” explicitly—abbreviations, aliases, and schema-omission phenomena—motivating entity linking without reliance on string overlap.

Many datasets are further augmented by:

• Multilingual Human Translation Pipelines: MultiSpider employs multi-round translation, schema localization, and NLI-based verification (SAVe framework), achieving >92% inter-annotator agreement (Dou et al., 2022).
• Answerability and Ambiguity Annotations: TriageSQL (Zhang et al., 2020) and TableQA incorporate explicit categories for unanswerable or ambiguous queries, enabling evaluation of triage and intent detection modules necessary for real-world deployments.

3. Evaluation Protocols and Metrics

A standard suite of evaluation metrics is employed across most datasets:

• Logic Form Accuracy: Measures exact match between generated and gold SQL (strict structural equivalence).
• Execution Accuracy (EX): Compares execution results of generated and reference SQL on the database, accommodating logical equivalence despite surface divergence.
• Component/Subtask Performance: Disaggregated metrics for subtasks (WHERE-value, WHERE-column, JOINs, aggregations, etc.), as in TableQA and Dr.Spider (Chang et al., 2023).
• Domain-Specificity: BiomedSQL (Koretsky et al., 23 May 2025) introduces response quality and safety rates, emphasizing domain-validity and abstention on uncertain cases.
• Robustness and Perturbation: Dr.Spider introduces task-specific robustness via 17 perturbation types; AmbiQT (Bhaskar et al., 2023) targets ambiguity coverage with “BothInTopK” metrics for divergent SQL interpretations.

For example, TableQA reports state-of-the-art model (SQLova) logic form accuracy dropping from 79.9% (WikiSQL) to 43.0% (TableQA), and condition value accuracy from 95.1% to 54.3% (Sun et al., 2020). In BiomedSQL, GPT-o3-mini achieves 59.0% EX, while a domain-aware agent reaches 62.6%, both far below expert performance (∼90%) (Koretsky et al., 23 May 2025).

4. Dataset-Specific Challenges and Research Opportunities

Modern benchmark datasets encode challenges crucial for advancing NL2SQL models:

• Schema and Query Complexity: Enterprise (BEAVER) and financial (BULL/FinSQL (Zhang et al., 19 Jan 2024)) benchmarks confront models with “wide” schemas, ambiguous abbreviations, and multiple table joins (BEAVER: 4.25 joins/query, 105 tables/DB avg) (Chen et al., 3 Sep 2024).
• Ambiguity and Reasoning: AmbiQT (Bhaskar et al., 2023) explicitly assigns multiple valid SQL per question; LogicCat (Liu et al., 24 May 2025) emphasizes chain-of-thought, physical, commonsense, mathematical, and hypothetical reasoning—state-of-the-art models reach just 14.96% EX here, boosted to 33.96% when provided with reasoning steps.
• Multilinguality: MultiSpider and Dialect2SQL address surface form variation, script, and morphology, with dedicated schema augmentation, back-translation, and entailment-based verification to improve model schema linking (Dou et al., 2022, Chafik et al., 20 Jan 2025).
• Practical Limitations: EHRSQL (Lee et al., 2023) and BiomedSQL (Koretsky et al., 23 May 2025) enforce answerability/uncertainty flags and demand scientific reasoning (e.g., p-value thresholds), reflecting deployment in safety-critical domains.

These datasets provide systematic ablations for studying performance on phenomena such as:

Phenomenon	Datasets	Research Implications
Entity linking, expression	TableQA, MultiSpider	Entity resolution, schema matching, synonym handling
Answerability/ambiguity	TableQA, TriageSQL, AmbiQT	Triage, intent detection, multi-output decoding, safe refusal
Session/context dependence	SeSQL, CHASE	Multi-turn context modeling, session memory, error accumulation
Multilinguality/dialect	MultiSpider, Dialect2SQL	Schema translation, cross-lingual transfer, morphological generalization
Domain/scientific reasoning	BiomedSQL, LogicCat, EHRSQL	Domain knowledge grounding, multi-step/multi-hop reasoning, logic augmentation

5. Dataset-Driven Advances in Model Architectures and Evaluation Strategies

Text-to-SQL benchmarks have directly influenced the development and evaluation of parsing architectures:

• Table-Aware Architectures: TableQA’s end-to-end BiLSTM-BERT mechanism encodes full table content for semantic attention over potential condition values (Sun et al., 2020).
• Schema Augmentation: MultiSpider’s SAVe system programmatically generates synonym/morphology schema alternatives, using NLI models for high-precision filtering and boosting multilingual performance (Dou et al., 2022).
• Prompt Engineering and Fine-Tuning: FinSQL (Zhang et al., 19 Jan 2024) applies hybrid prompt construction (chain-of-thought, synonym augmentation, schema skeletonization) and LoRA-based plug-in fine-tuning, with weights merging for few-shot domain transfer (up to 36.64% improvement).
• Robustness-by-Design: Dr.Spider (Chang et al., 2023) and HI-SQL (Parab et al., 11 Jun 2025) demonstrate augmentation, historical hint injection, and verification/retry loops to withstand perturbations, achieving execution accuracy improvements across challenging subsets.
• Chain-of-Thought and Revision: SLM-SQL (Sheng et al., 30 Jul 2025) shows that SLMs can approach competitive EX with synthetic “reasoning think-steps”, RL-based post-training, and self-consistency-based merge revision.
• Evaluation Frameworks: UNITE (Lan et al., 2023) and DB-GPT-Hub (Zhou et al., 17 Jun 2024) aggregate and standardize evaluation across dozens of benchmarks, advancing compositional evaluation (e.g., via SQL nesting depth) and facilitating large-scale LLM fine-tuning and PEFT methods (LoRA/QLoRA).

6. Current Limitations and Future Directions

Benchmark datasets continue to evolve toward more realistic, challenging, and representative testing:

• Real-World Deployment Feedback: FootballDB (Fürst et al., 13 Feb 2024) integrates live user data and multiple schema designs for the same domain (World Cup football), revealing tradeoffs in schema robustness, latency, and inference cost in practice.
• Enterprise Context: BEAVER (Chen et al., 3 Sep 2024) exposes limitations in LLM-driven retrieval and SQL synthesis in the face of highly complex, private, and unfamiliar schemas, with state-of-the-art LLMs failing on both table retrieval and SQL assembly even in retrieval-free settings.
• Automatic Metadata Extraction: New methodologies combine table profiling, query log analysis, and LLM-generated schema summaries to produce richer, more usable metadata for both training benchmarks and real deployments (Shkapenyuk et al., 26 May 2025).
• Reasoning and Ambiguity Handling: As illustrated by LogicCat (Liu et al., 24 May 2025) and AmbiQT (Bhaskar et al., 2023), explicit modeling of multi-step reasoning and ambiguity coverage—via logical plan-based decoding and chain-of-thought prompts—offers future avenues for robust, generalizable systems.
• Multilingual and Low-Resource Evaluation: Continued progress requires expanding DB coverage to underrepresented languages, dialects, and real-world business/industry schemas, with careful attention to schema translation fidelity and domain-specific semantic transfer.

A plausible implication is that benchmark development will increasingly prioritize coverage of edge-case phenomena, domain specialization, multi-turn and interactive capabilities, schema diversity, and integration of real-world user behavior in order to support robust, practically-deployable text-to-SQL systems.