SING-SQL: Synthetic Data for Text-to-SQL

• SING-SQL is a synthetic data generation framework that produces in-domain Text-to-SQL pairs for fixed enterprise database schemas.
• It employs hierarchical schema partitioning, decomposing schemas at table and column levels to ensure joinability and diverse query complexity.
• The framework integrates LLM-based validation, executability checks, and column balancing to train specialized SingSQL-LM models for schema-specialized SQL inference.

SING-SQL is a synthetic data generation framework for in-domain Text-to-SQL translation that targets the single-database setting typical of enterprise deployments. It is defined as a fully automated two-stage pipeline that takes a target relational database, partitions its schema into sub-schemas, synthesizes SQL queries across multiple complexity levels, back-translates them into natural-language questions, and filters the resulting pairs through a quality-aware pipeline that includes LLM-as-a-judge validation, executability checks, automatic repair, and column balancing. The same work introduces SingSQL-LM, a family of compact LLMs fine-tuned on the synthetic data for schema-specialized Text-to-SQL inference (Caferoğlu et al., 30 Sep 2025).

1. Scope, motivation, and problem setting

SING-SQL addresses an in-domain Text-to-SQL regime in which the database schema is fixed and often large, and the operational objective is high accuracy on that specific schema rather than cross-domain generalization. The framework is motivated by the observation that many organizations have one or a few large proprietary databases, lack query logs or human annotations, and need to evaluate Text-to-SQL systems on their own databases. In this setting, public cross-domain benchmarks are only a partial proxy for deployment requirements, because they do not guarantee full schema coverage and do not directly solve schema specialization (Caferoğlu et al., 30 Sep 2025).

The framework assumes access to the target database schema, including tables, columns, data types, and ideally foreign-key constraints. It does not require SQL logs or manual annotations. It does require the ability to execute SQL against the database for executability checks, and access to a capable external LLM for SQL generation, SQL-to-text generation, judgment, and repair. The paper uses Gemini-2.5-Flash for these roles. A central architectural assumption is that foreign-key metadata governs joinability at the table-partitioning stage; when such metadata is missing or incomplete, it must be manually added or recovered (Caferoğlu et al., 30 Sep 2025).

The output of the framework is a large synthetic dataset of question-SQL pairs tied to sub-schemas of the target database. This dataset is then used to train a specialized model family, SingSQL-LM, based on Qwen2.5-Coder-Instruct backbones with LoRA. This suggests a deployment pattern in which synthetic-data construction and schema-specialized model adaptation are treated as a single integrated pipeline rather than as separate preprocessing and training stages (Caferoğlu et al., 30 Sep 2025).

2. Hierarchical schema partitioning

A defining element of SING-SQL is hierarchical schema partitioning. The database is first decomposed into table-level sub-schemas and then into column-level sub-schemas. This decomposition controls prompt size, enforces joinability, and increases schema coverage without requiring exhaustive enumeration of all table-column combinations (Caferoğlu et al., 30 Sep 2025).

At the table level, the schema is treated as a graph in which tables are connected through foreign-key relationships. The framework generates combinations of tables subject to a hyperparameter specifying table counts for sub-schemas. Only combinations in which all tables are transitively joinable are retained. The paper describes this step through the procedure GenTableLevelSubSchemas(tc), which first obtains joinable tables, enumerates all database table combinations, and retains only those combinations for which all tables are transitively joinable (Caferoğlu et al., 30 Sep 2025).

At the column level, each table-level sub-schema is further partitioned by distinguishing connection columns from non-connection columns. Connection columns include primary keys and columns participating in foreign keys, and they are always retained in each column-level view. The remaining columns are randomly permuted and then traversed with a sliding window of size $w$ and stride $s$. For each windowed portion, the connection columns are prepended, and the Cartesian product across tables yields the final column-level sub-schemas. The paper gives this explicitly in Algorithm 3 through GenColumnLevelSubSchemas, where connCols are always preserved and nonConnCols are processed with a sliding window (Caferoğlu et al., 30 Sep 2025).

This partitioning design is a coverage-control mechanism rather than a semantic parser. Sliding windows avoid combinatorial explosion, while the compulsory inclusion of connection columns preserves joinability and executability. The random shuffling of non-connection columns adds diversity. A plausible implication is that the framework treats schema reduction as a structured sampling problem: it does not attempt to recover the single best prompt schema a priori, but instead constructs a large family of promptable sub-schemas that collectively approximate full coverage (Caferoğlu et al., 30 Sep 2025).

3. SQL-first synthesis and the quality-aware pipeline

SING-SQL generates SQL before natural-language questions. For each column-level sub-schema, the framework synthesizes SQL queries at four complexity levels—simple, moderate, challenging, and window—and then back-translates each SQL query into a natural-language question. The paper states that it generates $N$ SQLs per sub-schema per complexity level and uses $N = 3$ in its experiments. The SQL generation prompt instructs the LLM to consider only the target database, determine relevant tables and columns, apply logical filters, aggregations, and window functions, and construct valid SQLite SQL. The model is asked to respond in a <reasoning> ... </reasoning> and <answer> ... </answer> format (Caferoğlu et al., 30 Sep 2025).

The SQL-first design is justified by the claim that SQL-to-text generation is generally easier than text-to-SQL generation, because natural language is flexible and can more readily be aligned to a given SQL query. The resulting question-SQL pairs are then processed by a quality-aware pipeline with four core stages: LLM-as-a-judge validation, executability checking, automatic repair, and optional reasoning-trace generation (Caferoğlu et al., 30 Sep 2025).

The LLM-as-a-judge stage tests whether the generated question correctly describes what the SQL query does and whether the analytical intent is logical. The paper characterizes this as a binary keep-or-drop decision implemented by EvaluateSQL2TextItem(SQL2TextItem). Appendix examples indicate that the judge can catch semantic misinterpretations, such as treating a column like enroll12 as referring only to 12th grade instead of grades 1–12, and illogical aggregates such as averaging an ID column (Caferoğlu et al., 30 Sep 2025).

The executability check runs each SQL query against the actual database. If execution fails, the framework invokes FixSQL, which uses the LLM to repair the failing SQL. The repaired SQL is then re-tested, and only executable items are retained. The paper does not describe a multi-step repair loop; the process is original query followed by at most one repair attempt (Caferoğlu et al., 30 Sep 2025).

Reasoning traces are generated for examples that survive validation and execution. These traces follow a divide-and-conquer style inspired by CHASE-SQL and are later used in some fine-tuning and inference-context experiments. However, the paper reports that including reasoning traces at inference time usually hurts performance once schema context is available (Caferoğlu et al., 30 Sep 2025).

4. Column balancing and dataset construction

After the initial generation pass, SING-SQL performs column-frequency analysis and targeted re-generation. The framework counts how often each column appears in the synthetic SQL queries, identifies under-represented columns by comparing their frequencies against a threshold $f_{\text{min}}$, and constructs new focus sub-schemas containing those columns. It then runs the same GenT2S procedure on the focus sub-schemas, with prompts explicitly requiring the use of the focus columns (Caferoğlu et al., 30 Sep 2025).

The paper presents this workflow in Algorithm 1, where CountCols computes column frequencies, GetFocusCols identifies low-frequency columns, FindFocusSchemas retrieves relevant sub-schemas, and GenT2S is called again to create column-focused Text-to-SQL examples. The additional examples are concatenated with the original synthetic set and passed through a final filter (Caferoğlu et al., 30 Sep 2025).

This balancing step is a coverage mechanism rather than a uniformity constraint. The paper states that it does not enforce perfect uniformity but removes extreme sparsity and ensures every column is represented at least a minimum number of times. In the California Schools setup, the threshold is given as 400. A plausible implication is that SING-SQL treats column coverage as a first-class dataset design criterion, in contrast to benchmark-derived training sets where large portions of the schema may never appear in any gold query (Caferoğlu et al., 30 Sep 2025).

The reported synthetic dataset statistics make this design choice explicit. For California Schools, the synthetic train split contains 34,266 examples, with synthetic dev and test splits of 1,124 each, balanced across simple, moderate, challenging, and window categories, and with full column coverage: 0 unused columns in train, dev, and test. By contrast, the BIRD dev subset for the same database contains 89 questions and leaves 15 out of 89 columns, or 16.85%, unused (Caferoğlu et al., 30 Sep 2025).

5. SingSQL-LM and context management

SingSQL-LM is the model family trained on SING-SQL data. The backbones are Qwen2.5-Coder-Instruct models at 1.5B and 3B scales, fine-tuned using LoRA. The paper reports two LoRA configurations: R32 with rank 32, alpha 32, and learning rate $1.0 \times 10^{-4}$; and R64 with rank 64, alpha 64, and learning rate $7.5 \times 10^{-5}$. Training uses a cosine learning-rate schedule, 0.1 warmup, effective batch size 8, and 2 epochs, implemented with Unsloth (Caferoğlu et al., 30 Sep 2025).

The training objective is standard supervised fine-tuning over prompt-target pairs, with the target sequence containing SQL and, in some settings, reasoning traces. The paper does not introduce a specialized loss beyond standard autoregressive cross-entropy. More important than the loss itself is the study of context management: which information should be shown during training and which during inference (Caferoğlu et al., 30 Sep 2025).

Three training regimes are considered. T2S uses Text-to-SQL pairs without schema context. T2SWS uses pairs augmented with the filtered schema as part of the input. T2SWS, T2S mixes schema-aware and schema-free examples. At inference, the paper varies the presence of filtered schema, the number of few-shot examples, and whether few-shot reasoning traces are included. The central finding is that schema-free fine-tuning combined with schema-only inference is the strongest configuration (Caferoğlu et al., 30 Sep 2025).

For Qwen2.5-Coder-3B, the paper reports that T2S training with schema-only inference and no few-shots yields EX upper bound $= 60.67$ and Soft F1 upper bound $= 72.35$ under R32, and EX upper bound $= 64.04$ and Soft F1 upper bound $s$0 under R64. Adding six few-shot examples reduces performance markedly; for example, with T2S training and schema plus six few-shots but no reasoning, EX upper bound drops to 48.31 and Soft F1 upper bound to 62.83 under R32. The paper concludes that schema itself is the most reliable external context and that few-shot examples and reasoning traces often introduce noise or prompt-length overhead (Caferoğlu et al., 30 Sep 2025).

This result places SING-SQL in a distinct methodological position relative to many inference-heavy Text-to-SQL systems. A plausible implication is that the synthetic-data pipeline is intended not merely to support retrieval at inference time, but to distill schema-specific regularities into the model parameters so that inference can remain schema-conditioned but retrieval-light (Caferoğlu et al., 30 Sep 2025).

6. Evaluation, empirical results, and relation to neighboring work

The main evaluation is conducted on the California Schools database from BIRD. Metrics follow the official BIRD protocol: Execution Accuracy (EX), which is 1 when the predicted SQL result table exactly matches the gold result table, and Soft F1-score, which measures overlap over rows and columns and is described as more indicative of practical correctness. The paper evaluates both lower bounds, corresponding to the first candidate, and upper bounds, corresponding to the best of $s$1 generated candidates with $s$2 (Caferoğlu et al., 30 Sep 2025).

The strongest reported model is SingSQL-LM-3B-R64. On the BIRD California Schools subset, with 32 candidates, it reaches 82.87% Soft F1 and 73.03% EX upper bound. The paper states that this outperforms the best 3B-scale baseline by +16.21 in Soft F1 and +12.36 in EX. At the 1.5B scale, SingSQL-LM-1.5B-R64 improves over prior systems by +9.30 in Soft F1 and +4.49 in EX (Caferoğlu et al., 30 Sep 2025).

On synthetic evaluation sets, the gains are larger. With $s$3, SingSQL-LM-3B-R64 achieves 65.21% EX upper bound and 75.33% Soft F1 upper bound on SING-Dev, and 64.08% EX upper bound and 72.37% Soft F1 upper bound on SING-Test. The paper compares this to CHESS, which reaches 38.70% EX and 53.44% Soft F1 on SING-Dev, and 40.75% EX and 55.24% Soft F1 on SING-Test (Caferoğlu et al., 30 Sep 2025).

A compact summary of representative results is as follows:

Model	Setting	Reported result
SingSQL-LM-3B-R64	BIRD California Schools, 32 candidates	82.87% Soft F1, 73.03% EX upper bound
SingSQL-LM-1.5B-R64	BIRD California Schools, 32 candidates	+9.30 Soft F1 and +4.49 EX over prior 1–1.5B systems
SingSQL-LM-3B-R64	SING-Dev, 8 candidates	65.21% EX upper bound, 75.33% Soft F1 upper bound
SingSQL-LM-3B-R64	SING-Test, 8 candidates	64.08% EX upper bound, 72.37% Soft F1 upper bound

The paper also evaluates schema filtering quality. Using BM25-Top6 + LLM, it reports Table Recall $s$4, Column Recall $s$5, Column Precision $s$6, and Strict Schema Recall $s$7. Using Vec-Top6 + LLM, it reports Table Recall $s$8, Column Recall $s$9, Column Precision $N$0, and Strict Schema Recall $N$1. These results are used to support the claim that synthetic data plus an LLM is sufficient for high-quality schema linking in the in-domain setting (Caferoğlu et al., 30 Sep 2025).

Within the broader Text-to-SQL literature, SING-SQL occupies a different niche from direct weakly supervised sequence generators such as SeqGenSQL, which focused on direct NL-to-SQL generation on WikiSQL with T5, gated extraction, schema augmentation, and silver data (Li et al., 2020). It also differs from candidate-generation-and-selection frameworks such as XiYan-SQL, which emphasize multi-generator inference and execution-aware selection on large cross-domain benchmarks (Liu et al., 7 Jul 2025). By contrast, SING-SQL treats the generation of in-domain synthetic supervision itself as the primary scaling mechanism. This suggests a division of labor in recent Text-to-SQL research: some systems optimize the inference stack over existing labeled data, whereas SING-SQL optimizes the construction of specialized data for a target schema (Caferoğlu et al., 30 Sep 2025).

The paper notes several limitations. The method depends on foreign-key metadata; sliding-window column partitioning can separate semantically related columns; only one divide-and-conquer prompting strategy is used for reasoning; no RL post-training is applied; decoding hyperparameters are not systematically explored; and schema linking is not fully optimized. Future work proposed in the paper includes keeping semantically related column groups together, exploring multi-path reasoning or self-consistent reasoning, and adding RL with execution-based rewards (Caferoğlu et al., 30 Sep 2025).